com.actian.dataflow.operators.io.parquet

Class ParquetFormat

java.lang.Object

com.actian.dataflow.operators.io.parquet.ParquetFormat

All Implemented Interfaces:

DataFormat

public class ParquetFormat
extends Object
implements DataFormat

Data format implementation for Parquet files.

Nested Class Summary

Nested classes/interfaces inherited from interface com.pervasive.datarush.operators.io.DataFormat

DataFormat.DataFormatter, DataFormat.DataParser

Constructor Summary

Constructors

Constructor and Description
ParquetFormat(ParquetMetadata parquetMetadata)
ParquetFormat(ParquetMetadata parquetMetadata, FileMetadata fileMetadata)

Method Summary

Modifier and Type	Method and Description
DataFormat.DataParser	createParser(ParsingOptions options) Create a new parser for the format using the specified parsing options.
DataFormat.DataFormatter	createWriter(FormattingOptions options) Create a new writer for the format using the specified formatting options.
FileMetadata	getMetadata() Gets the metadata associated with the format.
RecordTokenType	getType() Gets the record type associated with the format.
boolean	isSplittable() Indicates if the format supports parsing of subsections of a file.
FileMetadata	readMetadata(FileClient fileClient, ByteSource source) Reads the metadata associated with the format.
void	setMetadata(FileMetadata metadata) Sets the metadata associated with the format.
void	writeMetadata(FileMetadata metadata, FileClient fileClient, ByteSink target) Writes the provided metadata associated with the format.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ParquetFormat

public ParquetFormat(ParquetMetadata parquetMetadata)

ParquetFormat

public ParquetFormat(ParquetMetadata parquetMetadata,
                     FileMetadata fileMetadata)

Method Detail

getType

public RecordTokenType getType()

Description copied from interface: DataFormat

Gets the record type associated with the format. Records produced by the associated parser or consumed by the associated formatter will be of this type.

For many formats, this may be derived from a schema object describing the format layout.

Specified by:

getType in interface DataFormat

Returns:

the format's record type

getMetadata

public FileMetadata getMetadata()

Description copied from interface: DataFormat

Gets the metadata associated with the format. Records produces by the associated parser or consumed by the associated formatter will use this metadata.

Specified by:

getMetadata in interface DataFormat

Returns:

the format's metadata

setMetadata

public void setMetadata(FileMetadata metadata)

Description copied from interface: DataFormat

Sets the metadata associated with the format.

Specified by:

setMetadata in interface DataFormat

readMetadata

public FileMetadata readMetadata(FileClient fileClient,
                                 ByteSource source)

Description copied from interface: DataFormat

Reads the metadata associated with the format.

Specified by:

readMetadata in interface DataFormat

Parameters:

fileClient - client used to read file

source - location of the files

writeMetadata

public void writeMetadata(FileMetadata metadata,
                          FileClient fileClient,
                          ByteSink target)

Description copied from interface: DataFormat

Writes the provided metadata associated with the format.

Specified by:

writeMetadata in interface DataFormat

Parameters:

metadata - the metadata to write

fileClient - client used to write file

createParser

public DataFormat.DataParser createParser(ParsingOptions options)

Description copied from interface: DataFormat

Create a new parser for the format using the specified parsing options.

Specified by:

createParser in interface DataFormat

Parameters:

options - parsing options to use

Returns:

a new parser for reading external data

createWriter

public DataFormat.DataFormatter createWriter(FormattingOptions options)

Description copied from interface: DataFormat

Create a new writer for the format using the specified formatting options.

Specified by:

createWriter in interface DataFormat

Parameters:

options - formatting options to use

Returns:

a new formatter for writing external data

isSplittable

public boolean isSplittable()

Description copied from interface: DataFormat

Indicates if the format supports parsing of subsections of a file.

A format should only return true if it can, at least in some situations, support this sort of parsing. If a format requires reading the entire file, it must return false.

If a format is not splittable, a file in the format cannot be parsed in parallel; however, individual files can still be parsed independently in parallel, as when reading the contents of a directory or using a file globbing pattern.

Specified by:

isSplittable in interface DataFormat

Returns:

true if the format supports parsing only a portion of the file, false otherwise