Module datarush.library
Package com.pervasive.datarush.operators.io

Class AbstractWriter

java.lang.Object
com.pervasive.datarush.operators.AbstractLogicalOperator
com.pervasive.datarush.operators.CompositeOperator
com.pervasive.datarush.operators.io.AbstractWriter
All Implemented Interfaces:
LogicalOperator, RecordSinkOperator, SinkOperator<RecordPort>
Direct Known Subclasses:
AbstractTextWriter, BinaryWriter, WriteAvro, WriteORC, WriteStagingDataset
public abstract class AbstractWriter extends CompositeOperator implements RecordSinkOperator
A generic writer of byte data representing a stream of records. The writer encompasses the basic attributes any such reader should have:
  • a sink for bytes, as identified by a ByteSink. This is most often a file, so convenience methods for specifying the target as a file are provided.
  • common I/O tunables, such as buffer sizes
An AbstractWriter wraps a WriteSink operator. Implementations provide an interface specific to an appropriate class of files - delimited text files, as an example - hiding the more complex model used to describe data parsing in general. The composition structure is the same for all readers with only DataFormat used differing between implementations.

  • Field Details

    • input

      protected final RecordPort input
      The input port of the read operator

    • options

      protected final FormattingOptions options
      Container for options related to formatting

  • Constructor Details

    • AbstractWriter

      protected AbstractWriter()
      Writes an empty target with default settings. The target must be set before execution or an error will be raised.
      See Also:

    • AbstractWriter

      protected AbstractWriter(boolean includeDoneSignal)
      Writes an empty target with default settings, optionally providing a port for signaling completion of the write. The target must be set before execution or an error will be raised.
      Parameters:
      includeDoneSignal - indicates whether a done signal port should be created
      See Also:

    • AbstractWriter

      protected AbstractWriter(String path, WriteMode mode)
      Writes to the specified path in the given mode.

      If the writer is parallelized, this is interpreted as a directory in which each partition will write a fragment of the entire input stream. Otherwise, it is interpreted as the file to write.

      Parameters:
      path - the path to which to write
      mode - how to handle existing files

    • AbstractWriter

      protected AbstractWriter(Path path, WriteMode mode)
      Writes to the specified path in the given mode.

      If the writer is parallelized, this is interpreted as a directory in which each partition will write a fragment of the entire input stream. Otherwise, it is interpreted as the file to write.

      Parameters:
      path - the path to which to write
      mode - how to handle existing files

    • AbstractWriter

      protected AbstractWriter(ByteSink target, WriteMode mode)
      Writes to the specified target sink in the given mode.

      The writer can only be parallelized if the sink is fragmentable. In this case, each partition will be written as an independent sink. Otherwise, the writer will run non-parallel.

      Parameters:
      target - the sink to which to write
      mode - how to handle an existing sink

  • Method Details

    • getInput

      public RecordPort getInput()
      Gets the record port providing the records to write to the target sink.
      Specified by:
      getInput in interface RecordSinkOperator
      Specified by:
      getInput in interface SinkOperator<RecordPort>
      Returns:
      the input port for records to write

    • getTarget

      public ByteSink getTarget()
      Gets the data sink for the writer.
      Returns:
      the write sink

    • setTarget

      public void setTarget(String path)
      Sets the data sink for the writer.

      If the writer is parallelized, this is interpreted as a directory in which each partition will write a fragment of the entire input stream. Otherwise, it is interpreted as the file to write.

      Parameters:
      path - the path to which to write

    • setTarget

      public void setTarget(Path path)
      Sets the data sink for the writer.

      If the writer is parallelized, this is interpreted as a directory in which each partition will write a fragment of the entire input stream. Otherwise, it is interpreted as the file to write.

      Parameters:
      path - the path to which to write

    • setTarget

      public void setTarget(ByteSink sink)
      Sets the data sink for the writer.
      Parameters:
      sink - the sink to which to write

    • getMode

      public WriteMode getMode()
      Gets how existing files should be handled by the writer.
      Returns:
      the configured behavior with respect to target files

    • setMode

      public void setMode(WriteMode mode)
      Sets how the writer should handle an existing target.
      Parameters:
      mode - the behavior to use for existing files

    • getFormatOptions

      public FormattingOptions getFormatOptions()
      Gets the formatting options used by the writer.
      Returns:
      the format options

    • setFormatOptions

      public void setFormatOptions(FormattingOptions options)
      Sets the formatting options used by the reader. This sets all format options at once.
      Parameters:
      options - the format options to use
      See Also:
      • DataFormatter

    • getWriteBuffer

      public int getWriteBuffer()
      Gets the size of the I/O buffer, in bytes, to use for writes.
      Returns:
      the size of the write buffer

    • setWriteBuffer

      public void setWriteBuffer(int size)
      Sets the size of the I/O buffer, in bytes, to use for writes. The default size is 64K.
      Parameters:
      size - the size of the write buffer

    • getWriteSingleSink

      public boolean getWriteSingleSink()
      Indicates whether the writer produces a single output file.
      Returns:
      true if the writer should write a single output, false otherwise.

    • setWriteSingleSink

      public void setWriteSingleSink(boolean enabled)
      Set whether the writer should produce a single output file or multiple ones. By default, an output file will be produced for each partition, if the target sink supports this.
      Parameters:
      enabled - indicates whether a single output file should be written

    • getWriteOnClient

      public boolean getWriteOnClient()
      Indicates whether the writer should write a file on the client. This also disables the parallelism on the sink.
      Returns:
      true if the writer should a file on the client, false otherwise.

    • setWriteOnClient

      public void setWriteOnClient(boolean enabled)
      Set whether the writer should write a file on the client. This has the effect of disabling the parallelism on the sink.
      Parameters:
      enabled - indicates whether the writer should write a file on the client.

    • isIgnoreSortOrder

      public boolean isIgnoreSortOrder()
      If set to true, the writer will write in any order. If false, the writer will preserve the current sorted state of the input.
      Returns:
      whether to ignore sort order.

    • setIgnoreSortOrder

      public void setIgnoreSortOrder(boolean ignoreSortOrder)
      If set to true, the writer will write in any order. If false, the writer will preserve the current sorted state of the input.
      Parameters:
      ignoreSortOrder - whether to ignore sort order.

    • getSaveMetadata

      public boolean getSaveMetadata()
      If set to true, the writer will attempt to save the metadata with the data if the implementation supports it. Otherwise metadata will not be saved.
      Returns:
      whether metadata is saved

    • setSaveMetadata

      public void setSaveMetadata(boolean saveMetadata)
      If set to true, the writer will attempt to save the metadata with the data if the implementation supports it. Otherwise metadata will not be saved.
      Parameters:
      saveMetadata - whether metadata is saved

    • computeFormat

      protected abstract DataFormat computeFormat(CompositionContext ctx)
      Determines the data format for the target. The returned format is used during composition to construct a WriteSink operator. If an implementation supports schema discovery, it must be performed in this method.
      Parameters:
      ctx - the composition context for the current invocation of compose(CompositionContext)
      Returns:
      the target format to use

    • compose

      protected final void compose(CompositionContext ctx)
      Composes a writer for the target, using configured options and a derived format.
      Specified by:
      compose in class CompositeOperator
      Parameters:
      ctx - the context