com.pervasive.datarush.operators.io

Class AbstractWriter

java.lang.Object

com.pervasive.datarush.operators.AbstractLogicalOperator

com.pervasive.datarush.operators.CompositeOperator

com.pervasive.datarush.operators.io.AbstractWriter

All Implemented Interfaces:

LogicalOperator, RecordSinkOperator, SinkOperator<RecordPort>

Direct Known Subclasses:

AbstractTextWriter, WriteAvro, WriteORC, WriteStagingDataset

public abstract class AbstractWriter
extends CompositeOperator
implements RecordSinkOperator

A generic writer of byte data representing a stream of records. The writer encompasses the basic attributes any such reader should have:

• a sink for bytes, as identified by a ByteSink. This is most often a file, so convenience methods for specifying the target as a file are provided.
• common I/O tunables, such as buffer sizes

An AbstractWriter wraps a WriteSink operator. Implementations provide an interface specific to an appropriate class of files - delimited text files, as an example - hiding the more complex model used to describe data parsing in general. The composition structure is the same for all readers with only DataFormat used differing between implementations.

Field Summary

Fields

Modifier and Type	Field and Description
protected RecordPort	input The input port of the read operator
protected FormattingOptions	options Container for options related to formatting

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AbstractWriter() Writes an empty target with default settings.
protected	AbstractWriter(boolean includeDoneSignal) Writes an empty target with default settings, optionally providing a port for signaling completion of the write.
protected	AbstractWriter(ByteSink target, WriteMode mode) Writes to the specified target sink in the given mode.
protected	AbstractWriter(Path path, WriteMode mode) Writes to the specified path in the given mode.
protected	AbstractWriter(String path, WriteMode mode) Writes to the specified path in the given mode.

Method Summary

Methods

Modifier and Type	Method and Description
protected void	compose(CompositionContext ctx) Composes a writer for the target, using configured options and a derived format.
protected abstract DataFormat	computeFormat(CompositionContext ctx) Determines the data format for the target.
FormattingOptions	getFormatOptions() Gets the formatting options used by the writer.
RecordPort	getInput() Gets the record port providing the records to write to the target sink.
WriteMode	getMode() Gets how existing files should be handled by the writer.
boolean	getSaveMetadata() If set to true, the writer will attempt to save the metadata with the data if the implementation supports it.
ByteSink	getTarget() Gets the data sink for the writer.
int	getWriteBuffer() Gets the size of the I/O buffer, in bytes, to use for writes.
boolean	getWriteOnClient() Indicates whether the writer should write a file on the client.
boolean	getWriteSingleSink() Indicates whether the writer produces a single output file.
boolean	isIgnoreSortOrder() If set to true, the writer will write in any order.
void	setFormatOptions(FormattingOptions options) Sets the formatting options used by the reader.
void	setIgnoreSortOrder(boolean ignoreSortOrder) If set to true, the writer will write in any order.
void	setMode(WriteMode mode) Sets how the writer should handle an existing target.
void	setSaveMetadata(boolean saveMetadata) If set to true, the writer will attempt to save the metadata with the data if the implementation supports it.
void	setTarget(ByteSink sink) Sets the data sink for the writer.
void	setTarget(Path path) Sets the data sink for the writer.
void	setTarget(String path) Sets the data sink for the writer.
void	setWriteBuffer(int size) Sets the size of the I/O buffer, in bytes, to use for writes.
void	setWriteOnClient(boolean enabled) Set whether the writer should write a file on the client.
void	setWriteSingleSink(boolean enabled) Set whether the writer should produce a single output file or multiple ones.

Methods inherited from class com.pervasive.datarush.operators.AbstractLogicalOperator

disableParallelism, getInputPorts, getOutputPorts, newInput, newInput, newOutput, newRecordInput, newRecordInput, newRecordOutput, notifyError

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.pervasive.datarush.operators.LogicalOperator

disableParallelism, getInputPorts, getOutputPorts

Field Detail

input

protected final RecordPort input

The input port of the read operator

options

protected final FormattingOptions options

Container for options related to formatting

Constructor Detail

AbstractWriter

protected AbstractWriter()

Writes an empty target with default settings. The target must be set before execution or an error will be raised.

See Also:

setTarget(ByteSink)

AbstractWriter

protected AbstractWriter(boolean includeDoneSignal)

Writes an empty target with default settings, optionally providing a port for signaling completion of the write. The target must be set before execution or an error will be raised.

Parameters:

includeDoneSignal - indicates whether a done signal port should be created

See Also:

setTarget(ByteSink)

AbstractWriter

protected AbstractWriter(String path,
              WriteMode mode)

Writes to the specified path in the given mode.

If the writer is parallelized, this is interpreted as a directory in which each partition will write a fragment of the entire input stream. Otherwise, it is interpreted as the file to write.

Parameters:

path - the path to which to write

mode - how to handle existing files

AbstractWriter

protected AbstractWriter(Path path,
              WriteMode mode)

Writes to the specified path in the given mode.

If the writer is parallelized, this is interpreted as a directory in which each partition will write a fragment of the entire input stream. Otherwise, it is interpreted as the file to write.

Parameters:

path - the path to which to write

mode - how to handle existing files

AbstractWriter

protected AbstractWriter(ByteSink target,
              WriteMode mode)

Writes to the specified target sink in the given mode.

The writer can only be parallelized if the sink is fragmentable. In this case, each partition will be written as an independent sink. Otherwise, the writer will run non-parallel.

Parameters:

target - the sink to which to write

mode - how to handle an existing sink

Method Detail

getInput

public RecordPort getInput()

Gets the record port providing the records to write to the target sink.

Specified by:

getInput in interface RecordSinkOperator

Specified by:

getInput in interface SinkOperator<RecordPort>

Returns:

the input port for records to write

getTarget

public ByteSink getTarget()

Gets the data sink for the writer.

Returns:

the write sink

setTarget

public void setTarget(String path)

Sets the data sink for the writer.

If the writer is parallelized, this is interpreted as a directory in which each partition will write a fragment of the entire input stream. Otherwise, it is interpreted as the file to write.

Parameters:

path - the path to which to write

setTarget

public void setTarget(Path path)

Sets the data sink for the writer.

If the writer is parallelized, this is interpreted as a directory in which each partition will write a fragment of the entire input stream. Otherwise, it is interpreted as the file to write.

Parameters:

path - the path to which to write

setTarget

public void setTarget(ByteSink sink)

Sets the data sink for the writer.

Parameters:

sink - the sink to which to write

getMode

public WriteMode getMode()

Gets how existing files should be handled by the writer.

Returns:

the configured behavior with respect to target files

setMode

public void setMode(WriteMode mode)

Sets how the writer should handle an existing target.

Parameters:

mode - the behavior to use for existing files

getFormatOptions

public FormattingOptions getFormatOptions()

Gets the formatting options used by the writer.

Returns:

the format options

setFormatOptions

public void setFormatOptions(FormattingOptions options)

Sets the formatting options used by the reader. This sets all format options at once.

Parameters:

options - the format options to use

See Also:

DataFormat.DataFormatter

getWriteBuffer

public int getWriteBuffer()

Gets the size of the I/O buffer, in bytes, to use for writes.

Returns:

the size of the write buffer

setWriteBuffer

public void setWriteBuffer(int size)

Sets the size of the I/O buffer, in bytes, to use for writes. The default size is 64K.

Parameters:

size - the size of the write buffer

getWriteSingleSink

public boolean getWriteSingleSink()

Indicates whether the writer produces a single output file.

Returns:

true if the writer should write a single output, false otherwise.

setWriteSingleSink

public void setWriteSingleSink(boolean enabled)

Set whether the writer should produce a single output file or multiple ones. By default, an output file will be produced for each partition, if the target sink supports this.

Parameters:

enabled - indicates whether a single output file should be written

getWriteOnClient

public boolean getWriteOnClient()

Indicates whether the writer should write a file on the client. This also disables the parallelism on the sink.

Returns:

true if the writer should a file on the client, false otherwise.

setWriteOnClient

public void setWriteOnClient(boolean enabled)

Set whether the writer should write a file on the client. This has the effect of disabling the parallelism on the sink.

Parameters:

enabled - indicates whether the writer should write a file on the client.

isIgnoreSortOrder

public boolean isIgnoreSortOrder()

If set to true, the writer will write in any order. If false, the writer will preserve the current sorted state of the input.

Returns:

whether to ignore sort order.

setIgnoreSortOrder

public void setIgnoreSortOrder(boolean ignoreSortOrder)

If set to true, the writer will write in any order. If false, the writer will preserve the current sorted state of the input.

Parameters:

ignoreSortOrder - whether to ignore sort order.

getSaveMetadata

public boolean getSaveMetadata()

If set to true, the writer will attempt to save the metadata with the data if the implementation supports it. Otherwise metadata will not be saved.

Returns:

whether metadata is saved

setSaveMetadata

public void setSaveMetadata(boolean saveMetadata)

If set to true, the writer will attempt to save the metadata with the data if the implementation supports it. Otherwise metadata will not be saved.

Parameters:

saveMetadata - whether metadata is saved

computeFormat

protected abstract DataFormat computeFormat(CompositionContext ctx)

Determines the data format for the target. The returned format is used during composition to construct a WriteSink operator. If an implementation supports schema discovery, it must be performed in this method.

Parameters:

ctx - the composition context for the current invocation of compose(CompositionContext)

Returns:

the target format to use

compose

protected final void compose(CompositionContext ctx)

Composes a writer for the target, using configured options and a derived format.

Specified by:

compose in class CompositeOperator

Parameters:

ctx - the context