Module datarush.library
Package com.pervasive.datarush.operators.io

Class AbstractWriter

  • All Implemented Interfaces:
    LogicalOperator, RecordSinkOperator, SinkOperator<RecordPort>
    Direct Known Subclasses:
    AbstractTextWriter, BinaryWriter, WriteAvro, WriteORC, WriteStagingDataset
    public abstract class AbstractWriter
extends CompositeOperator
implements RecordSinkOperator
    A generic writer of byte data representing a stream of records. The writer encompasses the basic attributes any such reader should have:
    • a sink for bytes, as identified by a ByteSink. This is most often a file, so convenience methods for specifying the target as a file are provided.
    • common I/O tunables, such as buffer sizes
    An AbstractWriter wraps a WriteSink operator. Implementations provide an interface specific to an appropriate class of files - delimited text files, as an example - hiding the more complex model used to describe data parsing in general. The composition structure is the same for all readers with only DataFormat used differing between implementations.

    • Field Detail

      • input

        protected final RecordPort input
        The input port of the read operator

      • options

        protected final FormattingOptions options
        Container for options related to formatting

    • Constructor Detail

      • AbstractWriter

        protected AbstractWriter()
        Writes an empty target with default settings. The target must be set before execution or an error will be raised.
        See Also:
        setTarget(ByteSink)

      • AbstractWriter

        protected AbstractWriter​(boolean includeDoneSignal)
        Writes an empty target with default settings, optionally providing a port for signaling completion of the write. The target must be set before execution or an error will be raised.
        Parameters:
        includeDoneSignal - indicates whether a done signal port should be created
        See Also:
        setTarget(ByteSink)

      • AbstractWriter

        protected AbstractWriter​(String path,
                         WriteMode mode)
        Writes to the specified path in the given mode.

        If the writer is parallelized, this is interpreted as a directory in which each partition will write a fragment of the entire input stream. Otherwise, it is interpreted as the file to write.

        Parameters:
        path - the path to which to write
        mode - how to handle existing files

      • AbstractWriter

        protected AbstractWriter​(Path path,
                         WriteMode mode)
        Writes to the specified path in the given mode.

        If the writer is parallelized, this is interpreted as a directory in which each partition will write a fragment of the entire input stream. Otherwise, it is interpreted as the file to write.

        Parameters:
        path - the path to which to write
        mode - how to handle existing files

      • AbstractWriter

        protected AbstractWriter​(ByteSink target,
                         WriteMode mode)
        Writes to the specified target sink in the given mode.

        The writer can only be parallelized if the sink is fragmentable. In this case, each partition will be written as an independent sink. Otherwise, the writer will run non-parallel.

        Parameters:
        target - the sink to which to write
        mode - how to handle an existing sink

    • Method Detail

      • getTarget

        public ByteSink getTarget()
        Gets the data sink for the writer.
        Returns:
        the write sink

      • setTarget

        public void setTarget​(String path)
        Sets the data sink for the writer.

        If the writer is parallelized, this is interpreted as a directory in which each partition will write a fragment of the entire input stream. Otherwise, it is interpreted as the file to write.

        Parameters:
        path - the path to which to write

      • setTarget

        public void setTarget​(Path path)
        Sets the data sink for the writer.

        If the writer is parallelized, this is interpreted as a directory in which each partition will write a fragment of the entire input stream. Otherwise, it is interpreted as the file to write.

        Parameters:
        path - the path to which to write

      • setTarget

        public void setTarget​(ByteSink sink)
        Sets the data sink for the writer.
        Parameters:
        sink - the sink to which to write

      • getMode

        public WriteMode getMode()
        Gets how existing files should be handled by the writer.
        Returns:
        the configured behavior with respect to target files

      • setMode

        public void setMode​(WriteMode mode)
        Sets how the writer should handle an existing target.
        Parameters:
        mode - the behavior to use for existing files

      • getFormatOptions

        public FormattingOptions getFormatOptions()
        Gets the formatting options used by the writer.
        Returns:
        the format options

      • setFormatOptions

        public void setFormatOptions​(FormattingOptions options)
        Sets the formatting options used by the reader. This sets all format options at once.
        Parameters:
        options - the format options to use
        See Also:
        DataFormatter

      • getWriteBuffer

        public int getWriteBuffer()
        Gets the size of the I/O buffer, in bytes, to use for writes.
        Returns:
        the size of the write buffer

      • setWriteBuffer

        public void setWriteBuffer​(int size)
        Sets the size of the I/O buffer, in bytes, to use for writes. The default size is 64K.
        Parameters:
        size - the size of the write buffer

      • getWriteSingleSink

        public boolean getWriteSingleSink()
        Indicates whether the writer produces a single output file.
        Returns:
        true if the writer should write a single output, false otherwise.

      • setWriteSingleSink

        public void setWriteSingleSink​(boolean enabled)
        Set whether the writer should produce a single output file or multiple ones. By default, an output file will be produced for each partition, if the target sink supports this.
        Parameters:
        enabled - indicates whether a single output file should be written

      • getWriteOnClient

        public boolean getWriteOnClient()
        Indicates whether the writer should write a file on the client. This also disables the parallelism on the sink.
        Returns:
        true if the writer should a file on the client, false otherwise.

      • setWriteOnClient

        public void setWriteOnClient​(boolean enabled)
        Set whether the writer should write a file on the client. This has the effect of disabling the parallelism on the sink.
        Parameters:
        enabled - indicates whether the writer should write a file on the client.

      • isIgnoreSortOrder

        public boolean isIgnoreSortOrder()
        If set to true, the writer will write in any order. If false, the writer will preserve the current sorted state of the input.
        Returns:
        whether to ignore sort order.

      • setIgnoreSortOrder

        public void setIgnoreSortOrder​(boolean ignoreSortOrder)
        If set to true, the writer will write in any order. If false, the writer will preserve the current sorted state of the input.
        Parameters:
        ignoreSortOrder - whether to ignore sort order.

      • getSaveMetadata

        public boolean getSaveMetadata()
        If set to true, the writer will attempt to save the metadata with the data if the implementation supports it. Otherwise metadata will not be saved.
        Returns:
        whether metadata is saved

      • setSaveMetadata

        public void setSaveMetadata​(boolean saveMetadata)
        If set to true, the writer will attempt to save the metadata with the data if the implementation supports it. Otherwise metadata will not be saved.
        Parameters:
        saveMetadata - whether metadata is saved

      • computeFormat

        protected abstract DataFormat computeFormat​(CompositionContext ctx)
        Determines the data format for the target. The returned format is used during composition to construct a WriteSink operator. If an implementation supports schema discovery, it must be performed in this method.
        Parameters:
        ctx - the composition context for the current invocation of compose(CompositionContext)
        Returns:
        the target format to use

      • compose

        protected final void compose​(CompositionContext ctx)
        Composes a writer for the target, using configured options and a derived format.
        Specified by:
        compose in class CompositeOperator
        Parameters:
        ctx - the context