Module datarush.library

Package com.pervasive.datarush.operators

Provides classes and interfaces for developing dataflow operators. Operators represent processing units within a dataflow graph; they are the nodes in the graph. An operator consumes data from its inputs and produces data on its outputs. An alternative, function-oriented view is that an operator is a function which taking some number of flows as arguments and returning some number of flows.

All operators have:

• An interface defining the inputs and outputs of the operator. Graphs are constructed by linking operators together via their inputs and outputs.
• Zero or more properties which can be set which control the processing behavior of the operator.

A single operator represents processing to be done on the input data sets. This processing may be done in a single-threaded fashion or in parallel. When executed in parallel, multiple copies of an operator may execute concurrently, each receiving a different portion of the input data and producing a portion of the output data.

Operators may have requirements on the input or may provide guarantees about the output. These are expressed as metadata on the input and output ports. This information is then used by the framework in determining how to execute the dataflow graph.

Interface Summary

Interface	Description
CompositionContext	Composition context used by CompositeOperator's.
CompositionIterator	Interface to be implemented by iterative operators.
DeferredCompositionContext	Composition context passed to DeferredCompositeOperator.compose(DeferredCompositionContext).
ExecutionContext	Execution context passed to ExecutableOperator.execute(ExecutionContext).
IterativeExecutionContext	Context passed to CompositionIterator.execute(IterativeExecutionContext).
IterativeMetadataContext	Context used by IterativeOperator.computeMetadata(IterativeMetadataContext).
LogicalOperator	A logical operator is the fundamental unit of composition which may be added to a LogicalGraph.
MetadataCalculationContext	Base interface for all contexts in which metadata is being computed.
MetadataContext	Base interface for various contexts in which port metadata can be resolved.
MultiSinkOperator<T extends LogicalPort>	Mix-in interface to be implemented by some operators.
MultiSourceOperator<T extends LogicalPort>	Mix-in interface to be implemented by some operators.
OperatorComposable	Base interface for any context/graph that allows operators to be added and ports to be connected.
PipelineOperator<T extends LogicalPort>	Mix-in interface to be implemented by some operators.
RecordPipelineOperator	Refinement of PipelineOperator, specific to RecortPort's.
RecordSinkOperator	Refinement of SinkOperator, specific to RecortPort's.
RecordSourceOperator	Refinement of SourceOperator, specific to RecortPort's.
SinkOperator<T extends LogicalPort>	Mix-in interface to be implemented by some operators.
SourceOperator<T extends LogicalPort>	Mix-in interface to be implemented by some operators.
StagedSource<F extends LogicalPort>	A handle by which one can access the staged data associated with an iterative port.
StreamingMetadataContext	Context used when computing metadata via StreamingOperator.computeMetadata(StreamingMetadataContext).

Class Summary

Class	Description
AbstractDeferredRecordOperator	Convenient base class for DeferredCompositeOperator's that are also RecordPipelineOperator's.
AbstractExecutableRecordPipeline	A base class for simple record-to-record operators, providing common functions.
AbstractLogicalOperator	Base class for all LogicalOperators.
AbstractOperatorTestCase
AbstractRecordCompositeOperator	Convenient base class for CompositeOperator's that are also RecordPipelineOperator's.
CompositeOperator	To be implemented by operators that can be defined by chaining together other operations.
CoreTypeResolutionProvider	Public only as a matter of implementation; for internal use only.
DeferredCompositeOperator	For rare use cases; implementors should use CompositeOperator whenever possible!
ExecutableOperator	ExecutableOperators are the most commonly used operators.
IterativeOperator	To be implemented by operations that must make multiple passes over the input data.
OpenComposite	A composite operator that can be externally composed.
OpenModelSink<T>	A model sink that can be externally composed.
OpenModelSource<T>	A model source that can be externally composed.
OpenMultiModelSink<T>	A model sink that can be externally composed.
OpenMultiModelSource<T>	A model source that can be externally composed.
OpenMultiRecordSink	A record sink that can be externally composed.
OpenMultiRecordSource	A record source that can be externally composed.
OpenRecordSink	A record sink that can be externally composed.
OpenRecordSource	A record source that can be externally composed.
OperatorProxies	Public only as a matter of implementation; not intended for external use
OperatorProxies.CompositeOperatorProxy	Public only as a matter of implementation; not intended for external use
OperatorProxies.DeferredCompositeOperatorProxy	Public only as a matter of implementation; not intended for external use
OperatorProxies.ExecutableOperatorProxy	Public only as a matter of implementation; not intended for external use
OperatorProxies.IterativeOperatorProxy	Public only as a matter of implementation; not intended for external use
OperatorProxies.OperatorProxy	Public only as a matter of implementation; not intended for external use
OperatorProxies.StreamingOperatorProxy	Public only as a matter of implementation; not intended for external use
OperatorSettings	Advanced operator settings that can be specified for an operator.
OperatorTestHelper
ParallelismStrategy	A ParallelismStrategy is responsible for determining parallelization of an operator based on configured parallelism and source parallelism.
StagedSourceOptions	Various options that can be passed to StagedSource.getSource(StagedSourceOptions).
StreamingOperator	Base class for LogicalOperators that are a potentially streaming operation.

Enum Summary

Enum	Description
CompilationLevel	Enumeration of compilation levels.
OperatorType	Enumeration of possible operator types