Module datarush.library
Class AbstractTextReader
- java.lang.Object
-
- com.pervasive.datarush.operators.AbstractLogicalOperator
-
- com.pervasive.datarush.operators.CompositeOperator
-
- com.pervasive.datarush.operators.io.AbstractReader
-
- com.pervasive.datarush.operators.io.textfile.AbstractTextReader
-
- All Implemented Interfaces:
LogicalOperator
,RecordSourceOperator
,SourceOperator<RecordPort>
- Direct Known Subclasses:
ReadARFF
,ReadDelimitedText
,ReadFixedText
,ReadJSON
,ReadLog
public abstract class AbstractTextReader extends AbstractReader
A generic reader of text data representing a stream of records. The reader encompasses the basic attributes any such reader should have beyond a standard byte-oriented reader, namely information on how to decode the bytes into characters.- See Also:
CharsetEncoding
-
-
Field Summary
Fields Modifier and Type Field Description protected CharsetEncoding
encodingProps
Container for character encoding related attributes-
Fields inherited from class com.pervasive.datarush.operators.io.AbstractReader
options, output
-
-
Constructor Summary
Constructors Modifier Constructor Description protected
AbstractTextReader()
Reads an empty source with default settings.protected
AbstractTextReader(Path path)
Reads the file specified by the path using default options.protected
AbstractTextReader(ByteSource source)
Reads the specified data source using default options.protected
AbstractTextReader(String pattern)
Reads all paths matching the specified pattern using default options.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description Charset
getCharset()
Gets the character set used by the data source.String
getCharsetName()
Gets the name of the character set used by the data source.int
getDecodeBuffer()
Gets the size of the buffer, in bytes, used to decode character data.CharsetEncoding
getEncoding()
Get the character set encoding properties.CodingErrorAction
getErrorAction()
Get the configured encoding error action.String
getReplacement()
Get the text used by the replacement error action.void
setCharset(Charset charset)
Sets the character set used by the data source.void
setCharsetName(String charsetName)
Sets the character set used by the data source.void
setDecodeBuffer(int size)
Sets the size of the buffer, in bytes, used to decode character data.void
setEncoding(CharsetEncoding settings)
Set the properties that control character set encoding.void
setErrorAction(CodingErrorAction errorAction)
Set the encoding error action.void
setReplacement(String replacement)
Sets the error policy to be replacement with the specified string.-
Methods inherited from class com.pervasive.datarush.operators.io.AbstractReader
compose, computeFormat, getExtraFieldAction, getFieldErrorAction, getFieldLengthThreshold, getIncludeSourceInfo, getMissingFieldAction, getOutput, getParseOptions, getPessimisticSplitting, getReadBuffer, getReadOnClient, getRecordWarningThreshold, getSelectedFields, getSource, getSplitOptions, getUseMetadata, setExtraFieldAction, setFieldErrorAction, setFieldLengthThreshold, setIncludeSourceInfo, setMissingFieldAction, setParseErrorAction, setParseOptions, setPessimisticSplitting, setReadBuffer, setReadOnClient, setRecordWarningThreshold, setSelectedFields, setSelectedFields, setSource, setSource, setSource, setSplitOptions, setUseMetadata
-
Methods inherited from class com.pervasive.datarush.operators.AbstractLogicalOperator
disableParallelism, getInputPorts, getOutputPorts, newInput, newInput, newOutput, newRecordInput, newRecordInput, newRecordOutput, notifyError
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface com.pervasive.datarush.operators.LogicalOperator
disableParallelism, getInputPorts, getOutputPorts
-
-
-
-
Field Detail
-
encodingProps
protected final CharsetEncoding encodingProps
Container for character encoding related attributes
-
-
Constructor Detail
-
AbstractTextReader
protected AbstractTextReader()
Reads an empty source with default settings. The source must be set before execution or an error will be raised.- See Also:
AbstractReader.setSource(ByteSource)
-
AbstractTextReader
protected AbstractTextReader(String pattern)
Reads all paths matching the specified pattern using default options. Any matching path which is a directory is replaced with all files in the directory; this expansion is not applied recursively.- Parameters:
pattern
- a path-matching pattern- See Also:
FileClient#matchPaths(String)
-
AbstractTextReader
protected AbstractTextReader(Path path)
Reads the file specified by the path using default options. If the path refers to a a directory, all files in the directory are read; this expansion is not applied recursively.- Parameters:
path
- the path to read
-
AbstractTextReader
protected AbstractTextReader(ByteSource source)
Reads the specified data source using default options.- Parameters:
source
- the data source to read
-
-
Method Detail
-
getDecodeBuffer
public int getDecodeBuffer()
Gets the size of the buffer, in bytes, used to decode character data.- Returns:
- the decoding buffer size
-
setDecodeBuffer
public void setDecodeBuffer(int size)
Sets the size of the buffer, in bytes, used to decode character data. By default, this will be automatically derived using the character set and read buffer size.- Parameters:
size
- the decoding buffer size to use
-
getEncoding
public CharsetEncoding getEncoding()
Get the character set encoding properties.- Returns:
- properties used for character set encoding
-
setEncoding
public void setEncoding(CharsetEncoding settings)
Set the properties that control character set encoding.- Parameters:
settings
- character set encoding properties
-
getCharset
public Charset getCharset()
Gets the character set used by the data source.- Returns:
- the character set of the source
-
setCharset
public void setCharset(Charset charset)
Sets the character set used by the data source. By default ISO-8859-1 is used.- Parameters:
charset
- the character set to use
-
getCharsetName
public String getCharsetName()
Gets the name of the character set used by the data source.- Returns:
- the name of character set of the source
-
setCharsetName
public void setCharsetName(String charsetName)
Sets the character set used by the data source.- Parameters:
charsetName
- name of the character set- Throws:
InvalidPropertyValueException
- if the named character set is not supported.
-
getErrorAction
public CodingErrorAction getErrorAction()
Get the configured encoding error action.- Returns:
- encoding error action
-
setErrorAction
public void setErrorAction(CodingErrorAction errorAction)
Set the encoding error action. The error action determines how to handle errors encoding the input data into the configured character set. The default action is to replace the faulty data with a replacement character.- Parameters:
errorAction
- encoding error action
-
getReplacement
public String getReplacement()
Get the text used by the replacement error action. This value is only used if the error action is to replace.- Returns:
- replacement text used for encoding errors
-
setReplacement
public void setReplacement(String replacement)
Sets the error policy to be replacement with the specified string. By default, "?" is used.- Parameters:
replacement
- replacement value to use for encoding errors
-
-