com.pervasive.datarush.operators.io.textfile

Class WriteDelimitedText

java.lang.Object

com.pervasive.datarush.operators.AbstractLogicalOperator

com.pervasive.datarush.operators.CompositeOperator

com.pervasive.datarush.operators.io.AbstractWriter

com.pervasive.datarush.operators.io.textfile.AbstractTextWriter

com.pervasive.datarush.operators.io.textfile.WriteDelimitedText

All Implemented Interfaces:

LogicalOperator, RecordSinkOperator, SinkOperator<RecordPort>

public final class WriteDelimitedText
extends AbstractTextWriter

Writes a stream of records as delimited text. The writer performs no reordering or filtering of fields. If this is desired, the stream should be preprocessed with the appropriate operator.

Delimited text supports up to three distinct user-defined sequences within a record, used to identify field boundaries:

• a field separator, found between individual fields; by default, this is a comma (the ',' character).
• a field start delimiter, marking the beginning of a field value; by default this is a double quote (the '"' character).
• a field end delimiter, marking the end of a field value; ; by default this is a double quote (the '"' character).

The field separator cannot be empty. The start and end delimiters can be the same value. They can also both (but not individually) be empty, signifying the absence of field delimiters. The writer will always delimit fields, regardless of whether values require it. If a value contains the end delimiter, it will be escaped by doubling it. For instance, the value ab""c will be formatted as "ab"c".

The writer accepts a RecordTextSchema to provide formatting information for fields, as well as header row information, if requested. It is not required to provide a schema however, as one can be generated from the input data type, using default formatting based on datatype. The writer also supports a pluggable discovery mechanism for creating a schema based on the input type, should fine-grained dynamic control be required. Any schema, supplied or discovered, must be compatible with the input to the reader. To be compatible, a schema must contain a field definition with an assignable type for each field named in the input. Fields present in the schema, but not the input are permitted with the missing field assuming a null value.

Delimited text data may or may not have a header row. The header row is delimited as usual but contains the names of the fields in the data portion of the record. The writer will emit a header row if it should be present and the write is not appending to an existing file.

Field Summary

Fields inherited from class com.pervasive.datarush.operators.io.textfile.AbstractTextWriter

encodingProps

Fields inherited from class com.pervasive.datarush.operators.io.AbstractWriter

input, options

Constructor Summary

Constructors

Constructor and Description
WriteDelimitedText() Writes delimited text to an empty target with default settings.
WriteDelimitedText(boolean provideDoneSignal) Writes delimited text to an empty target with default settings.
WriteDelimitedText(ByteSink target, WriteMode mode) Writes delimited text to the specified target sink in the given mode.
WriteDelimitedText(Path path, WriteMode mode) Writes delimited text to the specified path in the given mode, using default settings.
WriteDelimitedText(String path, WriteMode mode) Writes delimited text to the specified path in the given mode, using default settings.

Method Summary

Modifier and Type	Method and Description
protected DataFormat	computeFormat(CompositionContext ctx) Determines the data format for the target.
FieldDelimiterSettings	getDelimiters() Gets the field delimiter settings used by the writer.
String	getFieldEndDelimiter() Gets the end of field delimiter.
String	getFieldSeparator() Returns the delimiter used to distinguish field boundaries.
String	getFieldStartDelimiter() Returns the start of field delimiter.
boolean	getHeader() Indicates whether a header row should be written in the target.
String	getLineComment() Gets the character sequence indicating a line comment.
String	getNullIndicator() Gets the text value used to represent null values by default in generated schemas.
String	getRecordSeparator() Gets the value used as a record separator.
RecordTextSchema<?>	getSchema() Gets the record schema for delimited text source.
TextRecordDiscoverer	getSchemaDiscovery() Gets the schema discoverer to use on for the delimited text.
void	setDelimiters(FieldDelimiterSettings settings) Sets the field delimiter settings for the writer.
void	setFieldDelimiter(String delimiter) Sets the delimiter used to denote the boundaries of a data field.
void	setFieldEndDelimiter(String delimiter) Sets the delimiter used to denote the end of a data field.
void	setFieldSeparator(String separator) Sets the delimiter used to define the boundary between data fields.
void	setFieldStartDelimiter(String delimiter) Sets the delimiter used to denote the beginning of a data field.
void	setHeader(boolean header) Configures whether to write a header row in the target.
void	setLineComment(String lineComment) Sets the character sequence indicating a line comment.
void	setNullIndicator(String value) Sets the text value used to represent null values by default in generated schemas.
void	setRecordSeparator(String separator) Sets the value to use as a record separator.
void	setSchema(RecordTextSchema<?> schema) Sets the record schema for the delimited text source.
void	setSchemaDiscovery(TextRecordDiscoverer discoverer) Sets the schema discoverer to use for the delimited text.

Methods inherited from class com.pervasive.datarush.operators.io.textfile.AbstractTextWriter

getCharset, getCharsetName, getEncodeBuffer, getEncoding, getErrorAction, getReplacement, setCharset, setCharsetName, setEncodeBuffer, setEncoding, setErrorAction, setReplacement

Methods inherited from class com.pervasive.datarush.operators.io.AbstractWriter

compose, getFormatOptions, getInput, getMode, getSaveMetadata, getTarget, getWriteBuffer, getWriteOnClient, getWriteSingleSink, isIgnoreSortOrder, setFormatOptions, setIgnoreSortOrder, setMode, setSaveMetadata, setTarget, setTarget, setTarget, setWriteBuffer, setWriteOnClient, setWriteSingleSink

Methods inherited from class com.pervasive.datarush.operators.AbstractLogicalOperator

disableParallelism, getInputPorts, getOutputPorts, newInput, newInput, newOutput, newRecordInput, newRecordInput, newRecordOutput, notifyError

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.pervasive.datarush.operators.LogicalOperator

disableParallelism, getInputPorts, getOutputPorts

Constructor Detail

WriteDelimitedText

public WriteDelimitedText()

Writes delimited text to an empty target with default settings. The target must be set before execution or an error will be raised.

A default schema will be generated based on the input type, unless otherwise configured via setSchema(RecordTextSchema) or setSchemaDiscovery(TextRecordDiscoverer).

See Also:

AbstractWriter.setTarget(ByteSink)

WriteDelimitedText

public WriteDelimitedText(boolean provideDoneSignal)

Writes delimited text to an empty target with default settings. The target must be set before execution or an error will be raised.

A default schema will be generated based on the input type, unless otherwise configured via setSchema(RecordTextSchema) or setSchemaDiscovery(TextRecordDiscoverer).

Parameters:

provideDoneSignal - indicates whether a done signal port should be created

See Also:

AbstractWriter.setTarget(ByteSink)

WriteDelimitedText

public WriteDelimitedText(String path,
                          WriteMode mode)

Writes delimited text to the specified path in the given mode, using default settings.

If the writer is parallelized, this is interpreted as a directory in which each partition will write a fragment of the entire input stream. Otherwise, it is interpreted as the file to write.

A default schema will be generated based on the input type, unless otherwise configured via setSchema(RecordTextSchema) or setSchemaDiscovery(TextRecordDiscoverer).

Parameters:

path - the path to which to write

mode - how to handle existing files

WriteDelimitedText

public WriteDelimitedText(Path path,
                          WriteMode mode)

Writes delimited text to the specified path in the given mode, using default settings.

If the writer is parallelized, this is interpreted as a directory in which each partition will write a fragment of the entire input stream. Otherwise, it is interpreted as the file to write.

A default schema will be generated based on the input type, unless otherwise configured via setSchema(RecordTextSchema) or setSchemaDiscovery(TextRecordDiscoverer).

Parameters:

path - the path to which to write

mode - how to handle existing files

WriteDelimitedText

public WriteDelimitedText(ByteSink target,
                          WriteMode mode)

Writes delimited text to the specified target sink in the given mode.

The writer can only be parallelized if the sink is fragmentable. In this case, each partition will be written as an independent sink. Otherwise, the writer will run non-parallel.

A default schema will be generated based on the input type, unless otherwise configured via setSchema(RecordTextSchema) or setSchemaDiscovery(TextRecordDiscoverer).

Parameters:

target - the sink to which to write

mode - how to handle an existing sink

Method Detail

getSchema

public RecordTextSchema<?> getSchema()

Gets the record schema for delimited text source. If schema discovery is enabled, this will return null.

Returns:

the record schema for the target

setSchema

public void setSchema(RecordTextSchema<?> schema)

Sets the record schema for the delimited text source. The output text will follow the formatting and ordering of fields in the schema.

Setting a schema overrides any previously configured schema discovery.

Parameters:

schema - the desired record schema for the target

See Also:

setSchemaDiscovery(TextRecordDiscoverer)

getSchemaDiscovery

public TextRecordDiscoverer getSchemaDiscovery()

Gets the schema discoverer to use on for the delimited text. If schema discovery is disabled, this will return null.

Returns:

the configured schema discoverer

setSchemaDiscovery

public void setSchemaDiscovery(TextRecordDiscoverer discoverer)

Sets the schema discoverer to use for the delimited text. Just prior to graph execution the type of the input records will be used to determine the schema to use for formatting and ordering fields in the output text.

By default, the schema will be discovered automatically. This schema preserves field order from the input and uses default formatting appropriate for the field's datatype. If a header row is written, it will use the field names from the input.

Setting schema discovery overrides any previously configured schema.

Parameters:

discoverer - the schema discoverer to use.

See Also:

setSchema(RecordTextSchema)

getNullIndicator

public String getNullIndicator()

Gets the text value used to represent null values by default in generated schemas.

Returns:

the string indicating a null value

setNullIndicator

public void setNullIndicator(String value)

Sets the text value used to represent null values by default in generated schemas. By default, this is the empty string. If schema is provided, this setting is ignored.

Parameters:

value - the string indicating a null value

getHeader

public boolean getHeader()

Indicates whether a header row should be written in the target.

Returns:

whether a header row should be written

setHeader

public void setHeader(boolean header)

Configures whether to write a header row in the target. A header will not be written if appending to a target file.

Parameters:

header - indicates whether to write a header row

getLineComment

public String getLineComment()

Gets the character sequence indicating a line comment.

Returns:

the sequence marking a line comment

setLineComment

public void setLineComment(String lineComment)

Sets the character sequence indicating a line comment. This sequence is used immediately following a record to indicate everything up to the next record separator is to be ignored.

Parameters:

lineComment - the character sequence marking the start of a line comment

getDelimiters

public FieldDelimiterSettings getDelimiters()

Gets the field delimiter settings used by the writer.

Returns:

the field delimiter settings

setDelimiters

public void setDelimiters(FieldDelimiterSettings settings)

Sets the field delimiter settings for the writer. This sets all field delimiter settings at once.

Parameters:

settings - the field delimiter settings to use

getRecordSeparator

public String getRecordSeparator()

Gets the value used as a record separator.

Returns:

the text value of the record separator

setRecordSeparator

public void setRecordSeparator(String separator)

Sets the value to use as a record separator. The separator is used to delimit records in the output text.

By default the record separator is set to the default record separator for the installed operating system of the execution environment.

Parameters:

separator - the value to use as a record separator

Throws:

com.pervasive.datarush.graphs.physical.InvalidPropertyValueException - if the separator is null or the empty string

getFieldSeparator

public String getFieldSeparator()

Returns the delimiter used to distinguish field boundaries.

Returns:

the string used to separate fields

setFieldSeparator

public void setFieldSeparator(String separator)

Sets the delimiter used to define the boundary between data fields.

Parameters:

separator - string used to separate fields

Throws:

com.pervasive.datarush.graphs.physical.InvalidPropertyValueException - if the delimiter is null or the empty string

setFieldDelimiter

public void setFieldDelimiter(String delimiter)

Sets the delimiter used to denote the boundaries of a data field.

This method is generally equivalent to calling setFieldStartDelimiter() and setFieldEndDelimiter() with the same parameter values. However, those methods do not allow the empty string as a parameter.

Parameters:

delimiter - string used to optionally mark the start and end of a field value. An empty string indicates field values are not delimited.

Throws:

com.pervasive.datarush.graphs.physical.InvalidPropertyValueException - if the delimiter is null

getFieldStartDelimiter

public String getFieldStartDelimiter()

Returns the start of field delimiter.

Returns:

the string used to mark the beginning of a field value

getFieldEndDelimiter

public String getFieldEndDelimiter()

Gets the end of field delimiter.

Returns:

the string used to mark the end of a field value

setFieldStartDelimiter

public void setFieldStartDelimiter(String delimiter)

Sets the delimiter used to denote the beginning of a data field. It not permitted to set the start delimiter to the empty string; use setFieldDelimiter(String) instead to indicate no delimiters.

Parameters:

delimiter - string used to mark the start of a field value

Throws:

com.pervasive.datarush.graphs.physical.InvalidPropertyValueException - if the delimiter is null or the empty string

setFieldEndDelimiter

public void setFieldEndDelimiter(String delimiter)

Sets the delimiter used to denote the end of a data field. It not permitted to set the end delimiter to the empty string; use setFieldDelimiter(String) instead to indicate no delimiters.

Parameters:

delimiter - string used to mark the start of a field value

Throws:

com.pervasive.datarush.graphs.physical.InvalidPropertyValueException - if the delimiter is null or the empty string

computeFormat

protected DataFormat computeFormat(CompositionContext ctx)

Description copied from class: AbstractWriter

Determines the data format for the target. The returned format is used during composition to construct a WriteSink operator. If an implementation supports schema discovery, it must be performed in this method.

Specified by:

computeFormat in class AbstractWriter

Parameters:

ctx - the composition context for the current invocation of AbstractWriter.compose(CompositionContext)

Returns:

the target format to use