Module datarush.library
Package com.pervasive.datarush.operators.io.textfile

Class WriteDelimitedText

java.lang.Object
com.pervasive.datarush.operators.AbstractLogicalOperator
com.pervasive.datarush.operators.CompositeOperator
com.pervasive.datarush.operators.io.AbstractWriter
com.pervasive.datarush.operators.io.textfile.AbstractTextWriter
com.pervasive.datarush.operators.io.textfile.WriteDelimitedText
All Implemented Interfaces:
LogicalOperator, RecordSinkOperator, SinkOperator<RecordPort>
public final class WriteDelimitedText extends AbstractTextWriter
Writes a stream of records as delimited text. The writer performs no reordering or filtering of fields. If this is desired, the stream should be preprocessed with the appropriate operator.

Delimited text supports up to three distinct user-defined sequences within a record, used to identify field boundaries:

  • a field separator, found between individual fields; by default, this is a comma (the ',' character).
  • a field start delimiter, marking the beginning of a field value; by default this is a double quote (the '"' character).
  • a field end delimiter, marking the end of a field value; ; by default this is a double quote (the '"' character).
The field separator cannot be empty. The start and end delimiters can be the same value. They can also both (but not individually) be empty, signifying the absence of field delimiters. The writer will always delimit fields, regardless of whether values require it. If a value contains the end delimiter, it will be escaped by doubling it. For instance, the value ab""c will be formatted as "ab"c".

The writer accepts a RecordTextSchema to provide formatting information for fields, as well as header row information, if requested. It is not required to provide a schema however, as one can be generated from the input data type, using default formatting based on datatype. The writer also supports a pluggable discovery mechanism for creating a schema based on the input type, should fine-grained dynamic control be required. Any schema, supplied or discovered, must be compatible with the input to the reader. To be compatible, a schema must contain a field definition with an assignable type for each field named in the input. Fields present in the schema, but not the input are permitted with the missing field assuming a null value.

Delimited text data may or may not have a header row. The header row is delimited as usual but contains the names of the fields in the data portion of the record. The writer will emit a header row if it should be present and the write is not appending to an existing file.

  • Constructor Details

    • WriteDelimitedText

      public WriteDelimitedText()
      Writes delimited text to an empty target with default settings. The target must be set before execution or an error will be raised.

      A default schema will be generated based on the input type, unless otherwise configured via setSchema(RecordTextSchema) or setSchemaDiscovery(TextRecordDiscoverer).

      See Also:

    • WriteDelimitedText

      public WriteDelimitedText(boolean provideDoneSignal)
      Writes delimited text to an empty target with default settings. The target must be set before execution or an error will be raised.

      A default schema will be generated based on the input type, unless otherwise configured via setSchema(RecordTextSchema) or setSchemaDiscovery(TextRecordDiscoverer).

      Parameters:
      provideDoneSignal - indicates whether a done signal port should be created
      See Also:

    • WriteDelimitedText

      public WriteDelimitedText(String path, WriteMode mode)
      Writes delimited text to the specified path in the given mode, using default settings.

      If the writer is parallelized, this is interpreted as a directory in which each partition will write a fragment of the entire input stream. Otherwise, it is interpreted as the file to write.

      A default schema will be generated based on the input type, unless otherwise configured via setSchema(RecordTextSchema) or setSchemaDiscovery(TextRecordDiscoverer).

      Parameters:
      path - the path to which to write
      mode - how to handle existing files

    • WriteDelimitedText

      public WriteDelimitedText(Path path, WriteMode mode)
      Writes delimited text to the specified path in the given mode, using default settings.

      If the writer is parallelized, this is interpreted as a directory in which each partition will write a fragment of the entire input stream. Otherwise, it is interpreted as the file to write.

      A default schema will be generated based on the input type, unless otherwise configured via setSchema(RecordTextSchema) or setSchemaDiscovery(TextRecordDiscoverer).

      Parameters:
      path - the path to which to write
      mode - how to handle existing files

    • WriteDelimitedText

      public WriteDelimitedText(ByteSink target, WriteMode mode)
      Writes delimited text to the specified target sink in the given mode.

      The writer can only be parallelized if the sink is fragmentable. In this case, each partition will be written as an independent sink. Otherwise, the writer will run non-parallel.

      A default schema will be generated based on the input type, unless otherwise configured via setSchema(RecordTextSchema) or setSchemaDiscovery(TextRecordDiscoverer).

      Parameters:
      target - the sink to which to write
      mode - how to handle an existing sink

  • Method Details

    • getSchema

      public RecordTextSchema<?> getSchema()
      Gets the record schema for delimited text source. If schema discovery is enabled, this will return null.
      Returns:
      the record schema for the target

    • setSchema

      public void setSchema(RecordTextSchema<?> schema)
      Sets the record schema for the delimited text source. The output text will follow the formatting and ordering of fields in the schema.

      Setting a schema overrides any previously configured schema discovery.

      Parameters:
      schema - the desired record schema for the target
      See Also:

    • getSchemaDiscovery

      public TextRecordDiscoverer getSchemaDiscovery()
      Gets the schema discoverer to use on for the delimited text. If schema discovery is disabled, this will return null.
      Returns:
      the configured schema discoverer

    • setSchemaDiscovery

      public void setSchemaDiscovery(TextRecordDiscoverer discoverer)
      Sets the schema discoverer to use for the delimited text. Just prior to graph execution the type of the input records will be used to determine the schema to use for formatting and ordering fields in the output text.

      By default, the schema will be discovered automatically. This schema preserves field order from the input and uses default formatting appropriate for the field's datatype. If a header row is written, it will use the field names from the input.

      Setting schema discovery overrides any previously configured schema.

      Parameters:
      discoverer - the schema discoverer to use.
      See Also:

    • getNullIndicator

      public String getNullIndicator()
      Gets the text value used to represent null values by default in generated schemas.
      Returns:
      the string indicating a null value

    • setNullIndicator

      public void setNullIndicator(String value)
      Sets the text value used to represent null values by default in generated schemas. By default, this is the empty string. If schema is provided, this setting is ignored.
      Parameters:
      value - the string indicating a null value

    • getHeader

      public boolean getHeader()
      Indicates whether a header row should be written in the target.
      Returns:
      whether a header row should be written

    • setHeader

      public void setHeader(boolean header)
      Configures whether to write a header row in the target. A header will not be written if appending to a target file.
      Parameters:
      header - indicates whether to write a header row

    • getLineComment

      public String getLineComment()
      Gets the character sequence indicating a line comment.
      Returns:
      the sequence marking a line comment

    • setLineComment

      public void setLineComment(String lineComment)
      Sets the character sequence indicating a line comment. This sequence is used immediately following a record to indicate everything up to the next record separator is to be ignored.
      Parameters:
      lineComment - the character sequence marking the start of a line comment

    • getDelimiters

      public FieldDelimiterSettings getDelimiters()
      Gets the field delimiter settings used by the writer.
      Returns:
      the field delimiter settings

    • setDelimiters

      public void setDelimiters(FieldDelimiterSettings settings)
      Sets the field delimiter settings for the writer. This sets all field delimiter settings at once.
      Parameters:
      settings - the field delimiter settings to use

    • getRecordSeparator

      public String getRecordSeparator()
      Gets the value used as a record separator.
      Returns:
      the text value of the record separator

    • setRecordSeparator

      public void setRecordSeparator(String separator)
      Sets the value to use as a record separator. The separator is used to delimit records in the output text.

      By default the record separator is set to the default record separator for the installed operating system of the execution environment.

      Parameters:
      separator - the value to use as a record separator
      Throws:
      com.pervasive.datarush.graphs.physical.InvalidPropertyValueException - if the separator is null or the empty string

    • getFieldSeparator

      public String getFieldSeparator()
      Returns the delimiter used to distinguish field boundaries.
      Returns:
      the string used to separate fields

    • setFieldSeparator

      public void setFieldSeparator(String separator)
      Sets the delimiter used to define the boundary between data fields.
      Parameters:
      separator - string used to separate fields
      Throws:
      com.pervasive.datarush.graphs.physical.InvalidPropertyValueException - if the delimiter is null or the empty string

    • setFieldDelimiter

      public void setFieldDelimiter(String delimiter)
      Sets the delimiter used to denote the boundaries of a data field.

      This method is generally equivalent to calling setFieldStartDelimiter() and setFieldEndDelimiter() with the same parameter values. However, those methods do not allow the empty string as a parameter.

      Parameters:
      delimiter - string used to optionally mark the start and end of a field value. An empty string indicates field values are not delimited.
      Throws:
      com.pervasive.datarush.graphs.physical.InvalidPropertyValueException - if the delimiter is null

    • getFieldStartDelimiter

      public String getFieldStartDelimiter()
      Returns the start of field delimiter.
      Returns:
      the string used to mark the beginning of a field value

    • getFieldEndDelimiter

      public String getFieldEndDelimiter()
      Gets the end of field delimiter.
      Returns:
      the string used to mark the end of a field value

    • setFieldStartDelimiter

      public void setFieldStartDelimiter(String delimiter)
      Sets the delimiter used to denote the beginning of a data field. It not permitted to set the start delimiter to the empty string; use setFieldDelimiter(String) instead to indicate no delimiters.
      Parameters:
      delimiter - string used to mark the start of a field value
      Throws:
      com.pervasive.datarush.graphs.physical.InvalidPropertyValueException - if the delimiter is null or the empty string

    • setFieldEndDelimiter

      public void setFieldEndDelimiter(String delimiter)
      Sets the delimiter used to denote the end of a data field. It not permitted to set the end delimiter to the empty string; use setFieldDelimiter(String) instead to indicate no delimiters.
      Parameters:
      delimiter - string used to mark the start of a field value
      Throws:
      com.pervasive.datarush.graphs.physical.InvalidPropertyValueException - if the delimiter is null or the empty string

    • computeFormat

      protected DataFormat computeFormat(CompositionContext ctx)
      Description copied from class: AbstractWriter
      Determines the data format for the target. The returned format is used during composition to construct a WriteSink operator. If an implementation supports schema discovery, it must be performed in this method.
      Specified by:
      computeFormat in class AbstractWriter
      Parameters:
      ctx - the composition context for the current invocation of AbstractWriter.compose(CompositionContext)
      Returns:
      the target format to use