com.pervasive.datarush.operators.io.textfile

Class AbstractTextReader

java.lang.Object

com.pervasive.datarush.operators.AbstractLogicalOperator

com.pervasive.datarush.operators.CompositeOperator

com.pervasive.datarush.operators.io.AbstractReader

com.pervasive.datarush.operators.io.textfile.AbstractTextReader

All Implemented Interfaces:

LogicalOperator, RecordSourceOperator, SourceOperator<RecordPort>

Direct Known Subclasses:

ReadARFF, ReadDelimitedText, ReadFixedText, ReadJSON, ReadLog

public abstract class AbstractTextReader
extends AbstractReader

A generic reader of text data representing a stream of records. The reader encompasses the basic attributes any such reader should have beyond a standard byte-oriented reader, namely information on how to decode the bytes into characters.

See Also:

CharsetEncoding

Field Summary

Fields

Modifier and Type	Field and Description
protected CharsetEncoding	encodingProps Container for character encoding related attributes

Fields inherited from class com.pervasive.datarush.operators.io.AbstractReader

options, output

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AbstractTextReader() Reads an empty source with default settings.
protected	AbstractTextReader(ByteSource source) Reads the specified data source using default options.
protected	AbstractTextReader(Path path) Reads the file specified by the path using default options.
protected	AbstractTextReader(String pattern) Reads all paths matching the specified pattern using default options.

Method Summary

Modifier and Type	Method and Description
Charset	getCharset() Gets the character set used by the data source.
String	getCharsetName() Gets the name of the character set used by the data source.
int	getDecodeBuffer() Gets the size of the buffer, in bytes, used to decode character data.
CharsetEncoding	getEncoding() Get the character set encoding properties.
CodingErrorAction	getErrorAction() Get the configured encoding error action.
String	getReplacement() Get the text used by the replacement error action.
void	setCharset(Charset charset) Sets the character set used by the data source.
void	setCharsetName(String charsetName) Sets the character set used by the data source.
void	setDecodeBuffer(int size) Sets the size of the buffer, in bytes, used to decode character data.
void	setEncoding(CharsetEncoding settings) Set the properties that control character set encoding.
void	setErrorAction(CodingErrorAction errorAction) Set the encoding error action.
void	setReplacement(String replacement) Sets the error policy to be replacement with the specified string.

Methods inherited from class com.pervasive.datarush.operators.io.AbstractReader

compose, computeFormat, getExtraFieldAction, getFieldErrorAction, getFieldLengthThreshold, getIncludeSourceInfo, getMissingFieldAction, getOutput, getParseOptions, getPessimisticSplitting, getReadBuffer, getReadOnClient, getRecordWarningThreshold, getSelectedFields, getSource, getSplitOptions, getUseMetadata, setExtraFieldAction, setFieldErrorAction, setFieldLengthThreshold, setIncludeSourceInfo, setMissingFieldAction, setParseErrorAction, setParseOptions, setPessimisticSplitting, setReadBuffer, setReadOnClient, setRecordWarningThreshold, setSelectedFields, setSelectedFields, setSource, setSource, setSource, setSplitOptions, setUseMetadata

Methods inherited from class com.pervasive.datarush.operators.AbstractLogicalOperator

disableParallelism, getInputPorts, getOutputPorts, newInput, newInput, newOutput, newRecordInput, newRecordInput, newRecordOutput, notifyError

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.pervasive.datarush.operators.LogicalOperator

disableParallelism, getInputPorts, getOutputPorts

Field Detail

encodingProps

protected final CharsetEncoding encodingProps

Container for character encoding related attributes

Constructor Detail

AbstractTextReader

protected AbstractTextReader()

Reads an empty source with default settings. The source must be set before execution or an error will be raised.

See Also:

AbstractReader.setSource(ByteSource)

AbstractTextReader

protected AbstractTextReader(String pattern)

Reads all paths matching the specified pattern using default options. Any matching path which is a directory is replaced with all files in the directory; this expansion is not applied recursively.

Parameters:

pattern - a path-matching pattern

See Also:

FileClient#matchPaths(String)

AbstractTextReader

protected AbstractTextReader(Path path)

Reads the file specified by the path using default options. If the path refers to a a directory, all files in the directory are read; this expansion is not applied recursively.

Parameters:

path - the path to read

AbstractTextReader

protected AbstractTextReader(ByteSource source)

Reads the specified data source using default options.

Parameters:

source - the data source to read

Method Detail

getDecodeBuffer

public int getDecodeBuffer()

Gets the size of the buffer, in bytes, used to decode character data.

Returns:

the decoding buffer size

setDecodeBuffer

public void setDecodeBuffer(int size)

Sets the size of the buffer, in bytes, used to decode character data. By default, this will be automatically derived using the character set and read buffer size.

Parameters:

size - the decoding buffer size to use

getEncoding

public CharsetEncoding getEncoding()

Get the character set encoding properties.

Returns:

properties used for character set encoding

setEncoding

public void setEncoding(CharsetEncoding settings)

Set the properties that control character set encoding.

Parameters:

settings - character set encoding properties

getCharset

public Charset getCharset()

Gets the character set used by the data source.

Returns:

the character set of the source

setCharset

public void setCharset(Charset charset)

Sets the character set used by the data source. By default ISO-8859-1 is used.

Parameters:

charset - the character set to use

getCharsetName

public String getCharsetName()

Gets the name of the character set used by the data source.

Returns:

the name of character set of the source

setCharsetName

public void setCharsetName(String charsetName)

Sets the character set used by the data source.

Parameters:

charsetName - name of the character set

Throws:

InvalidPropertyValueException - if the named character set is not supported.

getErrorAction

public CodingErrorAction getErrorAction()

Get the configured encoding error action.

Returns:

encoding error action

setErrorAction

public void setErrorAction(CodingErrorAction errorAction)

Set the encoding error action. The error action determines how to handle errors encoding the input data into the configured character set. The default action is to replace the faulty data with a replacement character.

Parameters:

errorAction - encoding error action

getReplacement

public String getReplacement()

Get the text used by the replacement error action. This value is only used if the error action is to replace.

Returns:

replacement text used for encoding errors

setReplacement

public void setReplacement(String replacement)

Sets the error policy to be replacement with the specified string. By default, "?" is used.

Parameters:

replacement - replacement value to use for encoding errors