com.pervasive.datarush.operators.io.staging

Class ForceStaging<P extends LogicalPort>

java.lang.Object

com.pervasive.datarush.operators.AbstractLogicalOperator

com.pervasive.datarush.operators.StreamingOperator

com.pervasive.datarush.operators.DeferredCompositeOperator

com.pervasive.datarush.operators.io.staging.ForceStaging<P>

Type Parameters:

P - the type of data port being staged

All Implemented Interfaces:

LogicalOperator

Direct Known Subclasses:

ForceRecordStaging

public abstract class ForceStaging<P extends LogicalPort>
extends DeferredCompositeOperator

Forces the operators on the input and output sides to execute sequentially, instead of concurrently. In a dataflow graph, consumers and produces run simultaneously, exchanging data as the operators are ready. This permits the graph to exploit pipeline parallelism.

In some cases, it may be necessary to avoid this concurrency. The simplest example is:

• Operator A produces data.
• Operator B consumes data from A, producing data only when all data from A has been consumed. (Calculating a total, for instance.)
• Operator C consumes the output of both A and B, but does not consume any data from A until consuming data from B.

In this case, a memory usage dilemma, known as queue expansion, arises. B needs all of A's data, but C doesn't read anything from A until B produces some data. As the data unread by C cannot be discarded, it must be saved until it is read. To minimize memory requirements, the lag between readers is limited. This limit is raised when required, but in this case, it requires the entire output of A, which could easily exceed memory.

Normally, this is automatically handled by the framework, saving data to disk. This is done in a optimistic fashion, avoiding disk I/O and associated performance degradation until necessary at cost of a (small) potential risk of graph failure due to memory being exhausted. This operator provides a pessimistic approach to the problem, forcing all data to disk, with all the associated performance costs.

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	ForceStaging(LogicalPortFactory<P> factory) Forces temporary staging of port data.

Method Summary

Methods

Modifier and Type	Method and Description
protected void	compose(DeferredCompositionContext ctx) Compose the body of this operator.
protected void	computeMetadata(StreamingMetadataContext ctx) Implementations must adhere to all of the contracts specified by StreamingOperator.computeMetadata(com.pervasive.datarush.operators.StreamingMetadataContext).
P	getInput() Gets the port for the data to be staged.
P	getOutput() Gets the port for the staged data.

Methods inherited from class com.pervasive.datarush.operators.DeferredCompositeOperator

computeOutputTypes

Methods inherited from class com.pervasive.datarush.operators.AbstractLogicalOperator

disableParallelism, getInputPorts, getOutputPorts, newInput, newInput, newOutput, newRecordInput, newRecordInput, newRecordOutput, notifyError

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ForceStaging

protected ForceStaging(LogicalPortFactory<P> factory)

Forces temporary staging of port data.

Parameters:

factory - the port factory for the type of ports being staged

Method Detail

getInput

public final P getInput()

Gets the port for the data to be staged. The operator supplying this port will run first.

Returns:

the input port for producer data

getOutput

public final P getOutput()

Gets the port for the staged data. The operator supplied from this will run second.

Returns:

the output port for consumer data

computeMetadata

protected final void computeMetadata(StreamingMetadataContext ctx)

Description copied from class: DeferredCompositeOperator

Implementations must adhere to all of the contracts specified by StreamingOperator.computeMetadata(com.pervasive.datarush.operators.StreamingMetadataContext). In addition, DeferredCompositeOperators must declare required metadata so as to satisfy requirements of the operators that are added during DeferredCompositeOperator.compose(com.pervasive.datarush.operators.DeferredCompositionContext).

Specified by:

computeMetadata in class DeferredCompositeOperator

Parameters:

ctx - the context

compose

protected final void compose(DeferredCompositionContext ctx)

Description copied from class: DeferredCompositeOperator

Compose the body of this operator. Implementations should do the following:

1. Instantiate and configure sub-operators, adding them to the provided context via the method OperatorComposable.add(O)
2. Create necessary connections via the method OperatorComposable.connect(P, P). This includes connections from the composite's input ports to sub-operators, connections between sub-operators, and connections from sub-operators output ports to the composite's output ports

Specified by:

compose in class DeferredCompositeOperator

Parameters:

ctx - the context