Module datarush.library

Package com.pervasive.datarush.operators

Class DeferredCompositeOperator

java.lang.Object

com.pervasive.datarush.operators.AbstractLogicalOperator

com.pervasive.datarush.operators.StreamingOperator

com.pervasive.datarush.operators.DeferredCompositeOperator

All Implemented Interfaces:

LogicalOperator

Direct Known Subclasses:

AbstractDeferredRecordOperator, ErrorSink, ErrorSource, ForceStaging, MergeModel, SimulatePartitions, UnionAll

public abstract class DeferredCompositeOperator
extends StreamingOperator

For rare use cases; implementors should use CompositeOperator whenever possible!

Similar to CompositeOperator in that this is composed of sub-operators. The key distinction, though, is that composition is deferred until execution time such that compose(DeferredCompositionContext) is invoked on a per-partition basis. Thus, subclasses of this class control input partitioning and output partitioning declaration. Beware that input partitioning requirements effectively override those that the components may have declared by-default, thus it is preferable to use CompositeOperator if possible. In addition be aware of the following restrictions:

1. DeferredCompositeOperator must not contain any iterative operators
2. DeferredCompositeOperator must not contain any operators that force staging
3. If any sub-operators declare metadata requirements they are treated as assertions (rather than being automatically fulfilled by the framework). Thus, it is the responsibility of DeferredCompositeOperator to ensure than any metadata requirements are fulfilled.

Constructor Summary

Constructors

Constructor	Description
DeferredCompositeOperator()

Method Summary

Modifier and Type	Method	Description
protected abstract void	compose(DeferredCompositionContext ctx)	Compose the body of this operator.
protected abstract void	computeMetadata(StreamingMetadataContext ctx)	Implementations must adhere to all of the contracts specified by StreamingOperator.computeMetadata(com.pervasive.datarush.operators.StreamingMetadataContext).
protected final void	computeOutputTypes(StreamingMetadataContext ctx)	Convenience method that implementations of computeMetadata may use in order to compute the output types.

Methods inherited from class com.pervasive.datarush.operators.AbstractLogicalOperator

disableParallelism, getInputPorts, getOutputPorts, newInput, newInput, newOutput, newRecordInput, newRecordInput, newRecordOutput, notifyError

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DeferredCompositeOperator

public DeferredCompositeOperator()

Method Details

computeMetadata

protected abstract void computeMetadata(StreamingMetadataContext ctx)

Implementations must adhere to all of the contracts specified by StreamingOperator.computeMetadata(com.pervasive.datarush.operators.StreamingMetadataContext). In addition, DeferredCompositeOperators must declare required metadata so as to satisfy requirements of the operators that are added during compose(com.pervasive.datarush.operators.DeferredCompositionContext).

Specified by:

computeMetadata in class StreamingOperator

Parameters:

ctx - the context

computeOutputTypes

protected final void computeOutputTypes(StreamingMetadataContext ctx)

Convenience method that implementations of computeMetadata may use in order to compute the output types. Beware that this method is implemented by invoking compose(com.pervasive.datarush.operators.DeferredCompositionContext) on the client in order to get the output types. Thus, operators whose compose method performs some machine-specific operation (i.e. validating the existence of a file on a machine in the cluster) should not invoke this method.

Parameters:

ctx - the context

compose

protected abstract void compose(DeferredCompositionContext ctx)

Compose the body of this operator. Implementations should do the following:

1. Instantiate and configure sub-operators, adding them to the provided context via the method OperatorComposable.add(O)
2. Create necessary connections via the method OperatorComposable.connect(P, P). This includes connections from the composite's input ports to sub-operators, connections between sub-operators, and connections from sub-operators output ports to the composite's output ports

Parameters:

ctx - the context