Module datarush.analytics
Package com.pervasive.datarush.analytics.cluster.kmeans

Class KMeans

  • All Implemented Interfaces:
    LogicalOperator, Serializable
    public final class KMeans
extends IterativeOperator
implements Serializable
    Computes clustering model for the given input based on the k-Means algorithm. All included fields of the given input must be of type double, float, long or int. The k-Means algorithm choses k random data points as the initial cluster centers. Within each iteration, it assigns input data points to current clusters based on the distance to the cluster centers and recomputes the centers based on the new assignments. Computation stops when one of the following conditions is true:
    1. maxIterations is exceeded
    2. the cluster centers do not change significantly between two iterations (see #isSignificantlyDifferent())
    See Also:
    Serialized Form

    • Constructor Detail

      • KMeans

        public KMeans()
        Create a new KMeans operator, initializing settings to their default values.

    • Method Detail

      • getInput

        public RecordPort getInput()
        Returns the input port. This is the learning data from which the model is built.
        Returns:
        the input port

      • getModel

        public PMMLPort getModel()
        Returns the model port. This is the model which is built from the learning data.
        Returns:
        the model port

      • getMaxIterations

        public int getMaxIterations()
        Returns the maximum number of iterations. Defaults to 99.
        Returns:
        The maximum number of iterations.

      • setMaxIterations

        public void setMaxIterations​(int max)
        Sets the maximum number of iterations. Defaults to 99.
        Parameters:
        max - The maximum number of iterations.

      • getK

        public int getK()
        Returns the "k" value, where k is the number of centroids to compute. Defaults to 3.
        Returns:
        the "k" value.

      • setK

        public void setK​(int k)
        Sets the "k" value, where k is the number of centroids to compute. Defaults to 3.
        Parameters:
        k - the "k" value.

      • getIncludedColumns

        public List<String> getIncludedColumns()
        Returns the list of columns to include for k-Means. An empty list means all columns of type double.
        Returns:
        The list of columns to include

      • setIncludedColumns

        public void setIncludedColumns​(List<String> includedColumns)
        Sets the list of columns to include for k-Means. An empty list means all columns of type double.
        Parameters:
        includedColumns - The list of columns to include

      • getDistanceMeasure

        public DistanceMeasure getDistanceMeasure()
        Returns the distance measure used to measure the distance between two points when building the model.
        Returns:
        the distance measure

      • setDistanceMeasure

        public void setDistanceMeasure​(DistanceMeasure distanceMeasure)
        Sets the distance measure used to measure the distance between two points when building the model.
        Parameters:
        distanceMeasure - the distance measure

      • computeMetadata

        protected void computeMetadata​(IterativeMetadataContext ctx)
        Description copied from class: IterativeOperator
        Implementations must adhere to the following contracts

        General

        Regardless of input ports/output port types, all implementations must do the following:

        1. Validation. Validation of configuration should always be performed first.
        2. Declare operator parallelizability. Implementations must declare by calling IterativeMetadataContext.parallelize(ParallelismStrategy).
        3. Declare output port parallelizablility. Implementations must declare by calling IterativeMetadataContext.setOutputParallelizable(com.pervasive.datarush.ports.LogicalPort, boolean)
        4. Declare input port parallelizablility. Implementations must declare by calling IterativeMetadataContext.setIterationParallelizable(com.pervasive.datarush.ports.LogicalPort, boolean).
        NOTE: There is a convenience method for performing steps 2-4 for the case where all record ports are parallelizable and where we are to determine parallelism based on source:
        • MetadataUtil#negotiateParallelismBasedOnSourceAssumingParallelizableRecords

        Input record ports

        Implementations with input record ports must declare the following:
        1. Required data ordering:
          2. Implementations that have data ordering requirements must declare them by calling RecordPort#setRequiredDataOrdering, otherwise iteration will proceed on an input dataset whose order is undefined.
        2. Required data distribution (only applies to parallelizable input ports):
          3. Implementations that have data distribution requirements must declare them by calling RecordPort#setRequiredDataDistribution, otherwise iteration will proceed on an input dataset whose distribution is the unspecified partial distribution.
        Note that if the upstream operator's output distribution/ordering is compatible with those required, we avoid a re-sort/re-distribution which is generally a very large savings from a performance standpoint.

        Output record ports (static metadata)

        Implementations with output record ports must declare the following:
        1. Type: Implementations must declare their output type by calling RecordPort#setType.
        Implementations with output record ports may declare the following:
        1. Output data ordering: Implementations that can make guarantees as to their output ordering may do so by calling RecordPort#setOutputDataOrdering
        2. Output data distribution (only applies to parallelizable output ports): Implementations that can make guarantees as to their output distribution may do so by calling RecordPort#setOutputDataDistribution
        Note that both of these properties are optional; if unspecified, performance may suffer since the framework may unnecessarily re-sort/re-distributed the data.

        Input model ports

        In general, iterative operators will tend not to have model input ports, but if so, there is nothing special to declare for input model ports. Models are implicitly duplicated to all partitions when going from non-parallel to parallel ports.

        Output model ports (static metadata)

        SimpleModelPort's have no associated metadata and therefore there is never any output metadata to declare. PMMLPort's, on the other hand, do have associated metadata. For all PMMLPorts, implementations must declare the following:
        1. pmmlModelSpec: Implementations must declare the PMML model spec by calling PMMLPort.setPMMLModelSpec.

        Output ports with dynamic metadata

        If an output port has dynamic metadata, implementations can declare by calling IterativeMetadataContext.setOutputMetadataDynamic(com.pervasive.datarush.ports.LogicalPort, boolean). In the case that metadata is dynamic, calls to RecordPort#setType, RecordPort#setOutputDataOrdering, etc are not allowed and thus the sections above entitled "Output record ports (static metadata)" and "Output model ports (static metadata)" must be skipped. Note that, if possible, dynamic metadata should be avoided (see IterativeMetadataContext.setOutputMetadataDynamic(com.pervasive.datarush.ports.LogicalPort, boolean)).

        Specified by:
        computeMetadata in class IterativeOperator
        Parameters:
        ctx - the context

      • isSignificantlyDifferent

        public static boolean isSignificantlyDifferent​(Cluster first,
                                               Cluster second)
        Returns true if the centers of both clusters are significantly different compared to each other. Two centers are considered significantly different, if, for any i, the following is true:

        Math.abs(firts.getNumArray()[i] - second.getNumArray()[i]) > 1e-10*(first.getNumArray()[i] + other.getNumArray()[i])

        Parameters:
        first - the reference cluster
        second - the cluster to be compared to the reference
        Returns:
        true if the cluster centers are significantly different