public interface IterativeMetadataContext extends MetadataCalculationContext
IterativeOperator.computeMetadata(IterativeMetadataContext)
.
With iterative operations, parallel vs. non-parallel is determined on a port-by-port basis. By default
all ports are assumed to be non-parallel. Operations must declare whether their ports are distributed
or local.Modifier and Type | Method and Description |
---|---|
void |
parallelize(ParallelismStrategy strategy)
Controls the parallelism of the ports of this operator.
|
void |
setIterationParallelizable(LogicalPort port,
boolean parallel)
Sets whether the operator will iterate on the given input data in parallel.
|
void |
setOutputMetadataDynamic(LogicalPort port,
boolean dynamic)
Indicates that the metadata for the given output port is dynamic.
|
void |
setOutputParallelizable(LogicalPort port,
boolean parallel)
Sets whether the given output port is parallel.
|
getCompilationLevel, getOperator, setOutputMetadata, setOutputMetadataDynamic, setRequiredMetadata, setStagingForced
getCombinedMetadata, getEngineConfig, getFileClient, getMaxParallelism, getPath, getRequiredMetadata, getSourceMaxParallelism, getSourceMetadata, isParallel, isSourceConnected, isSourceParallel
void parallelize(ParallelismStrategy strategy)
ParallelismStrategy.NON_PARALLELIZABLE
by default. If non-parallelizable, all data will be brought to a single processor on the machine/cluster.
In general, most operations must be parallelizable for them to be scalable. Some operations (composites
)
may consist of parallelizable operations that operator on large pieces of data, followed by non-parallelizable
operations that operate on smaller pieces.strategy
- the strategy to use to determine parallelization.ParallelismStrategy
void setOutputParallelizable(LogicalPort port, boolean parallel)
MetadataUtil#negotiateParallelismBasedOnSourceAssumingParallelizableRecords
,
you need not make this declaration.port
- the output portparallel
- whether the output is parallelvoid setIterationParallelizable(LogicalPort port, boolean parallel)
MetadataUtil#negotiateParallelismBasedOnSourceAssumingParallelizableRecords
,
you need not make this declaration.port
- the port providing a dataset to processparallel
- whether the data is to be processed in parallelvoid setOutputMetadataDynamic(LogicalPort port, boolean dynamic)
outputMetadata
must not be specified. The graph is compiled and
executed up until this operator. Following execution of this operator, downstream operators
can then be compiled. Note that dynamic metadata should be used sparingly as it causes
graph compilation errors to be postponed until after a (potentially) long operation.port
- the output portdynamic
- a value of true means metadata is dynamic.Copyright © 2021 Actian Corporation. All rights reserved.