Module datarush.library

Package com.pervasive.datarush.operators

Class CompositeOperator

java.lang.Object

com.pervasive.datarush.operators.AbstractLogicalOperator

com.pervasive.datarush.operators.CompositeOperator

All Implemented Interfaces:

LogicalOperator

Direct Known Subclasses:

AbstractReader, AbstractRecordCompositeOperator, AbstractRelationalJoin, AbstractWriter, AnalyzeDuplicateKeys, AnalyzeLinkKeys, AssertEqualHash, AssertEqualRecordType, AssertEqualTypes, AssertRowCount, BlockCartesian, BlockRecords, BlockSelf, ClusterDuplicates, ClusterLinks, DataQualityAnalyzer, DiscoverDomain, DiscoverDuplicates, DiscoverLinks, DrawDiagnosticsChart, DumpPartitions, FPGrowth, FrequentItems, JDBCOperator, KeyOperator, KNNClassifier, LoadActianVector, LoadVectorOnHadoop, LoadVectorOnHadoopDirect, LogisticRegressionLearner, NaiveBayesLearner, OpenComposite, OpenModelSink, OpenModelSource, OpenMultiModelSink, OpenMultiModelSource, OpenMultiRecordSink, OpenMultiRecordSource, OpenRecordSink, OpenRecordSource, ReadPMML, ReadSource, RemoveDuplicates, ReplaceMissingValues, RowsToColumns, RunRScript, SummaryStatistics

public abstract class CompositeOperator
extends AbstractLogicalOperator

To be implemented by operators that can be defined by chaining together other operations. CompositeOperators are commonly used in cases where an overall operation can be defined by a mixture parallelizable and non-parallelizable operations or a mixture of operations that have different distribution requirements. An example of this parallel/non-parallel mix would be the Group operator: in the case of a key-less aggregation, there is an initial parallel operator where partial aggregations are computed connected to a non-parallel operator that combines the one-row from each partition into a final result.

Here is an example of a CompositeOperator that consists of two sub-operators, linked together:

    public class MyComposite extends CompositeOperator {
       //the input port for the composite
       private final RecordPort input= newRecordInput("input");
       //the output port for the composite
       private final RecordPort output= newRecordOutput("output");
       
       public MyComposite() {
       }
       
       //the input port for the composite
       public RecordPort getInput() {
          return input;
       }
       
       //the output port for the composite
       public RecordPort getOutput() {
          return output;
       }

       @Override
       protected void compose(CompositionContext ctx) {
           //perform any validation
           if (...) {
              throw new ...;
           }
       
           //add the two sub-operators
           SubOp1 op1= ctx.add(new SubOp1());
           SubOp2 op2= ctx.add(new SubOp2());
           
           //connect the composite's input port to the op1's input
           ctx.connect(input,op1.getInput());
           //connect op1's output to op2's input
           ctx.connect(op1.getOutput(), op2.getInput());
           //connect op2's output to the composite's output port
           ctx.connect(op2.getOutput(), output);
       }
    }
 

Constructor Summary

Constructors

Constructor	Description
CompositeOperator()

Method Summary

Modifier and Type	Method	Description
protected abstract void	compose(CompositionContext ctx)	Compose the body of this operator.

Methods inherited from class com.pervasive.datarush.operators.AbstractLogicalOperator

disableParallelism, getInputPorts, getOutputPorts, newInput, newInput, newOutput, newRecordInput, newRecordInput, newRecordOutput, notifyError

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CompositeOperator

public CompositeOperator()

Method Detail

compose

protected abstract void compose(CompositionContext ctx)

Compose the body of this operator. Implementations should do the following:

1. Perform any validation of configuration, input types, etc
2. Instantiate and configure sub-operators, adding them to the provided context via the method OperatorComposable.add(O)
3. Create necessary connections via the method OperatorComposable.connect(P, P). This includes connections from the composite's input ports to sub-operators, connections between sub-operators, and connections from sub-operators output ports to the composite's output ports

Parameters:

ctx - the context