Module datarush.hadoop.core

Package com.pervasive.datarush.hbase

Class DeleteHBase

java.lang.Object

com.pervasive.datarush.operators.AbstractLogicalOperator

com.pervasive.datarush.operators.CompositeOperator

com.pervasive.datarush.hbase.KeyOperator

com.pervasive.datarush.hbase.DeleteHBase

All Implemented Interfaces:

LogicalOperator

public class DeleteHBase
extends KeyOperator

Write delete markers to HBase

A DeleteMarker can be mapped to a select HBase cell or it can be mapped to a family as a sub table:

1. mapCellMarker(java.lang.String, java.lang.String, com.pervasive.datarush.hbase.DeleteHBase.DeleteMarker) - Map a DeleteMarker to a select cell within a column family. A row key field is required to uniquely id cells. A DeleteMarker will be inserted for each mapped cell qualifier, for each input record.

2. mapFamilyMarker(java.lang.String, com.pervasive.datarush.hbase.DeleteHBase.DeleteMarker) - Map a DeleteMarker to a column family. If mapping DeleteMarker.DeleteFamily then only a row key field is required, otherwise both a row key field and a qualifier key field are required to uniquely id cells. A single DeleteMarker will be inserted for each input record.

A time key field can optionally be specified to allow the user to provide a timestamp value as part of the input record. If a time key field is not specified then each record will default to current time. If mapping DeleteMarker.Delete to delete a specific version of a cell then a time key field is required to uniquely id cells. DeleteMarkers that delete past versions will use the time key field or default to current time.

The input will be repartitioned using HBase table region row key ranges. Each partition will sort its DeleteMarkers in row key ascending, qualifier key ascending, time key descending order, and then write the DeleteMarkers to the appropriate regions.

If the specified HBase table does not exist then it will be created. The number of regions created will be MAX(4, the level of parallelism).

See Also:

WriteHBase, ReadHBase

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	DeleteHBase.DeleteMarker

Field Summary

Fields inherited from class com.pervasive.datarush.hbase.KeyOperator

catalogTableName, cellSchemaFamilyName, keySchemaFamilyName, statsFamilyName

Constructor Summary

Constructors

Constructor	Description
DeleteHBase()

Method Summary

Modifier and Type	Method	Description
void	compose(CompositionContext ctx)	Compose the body of this operator.
Map<String,Map<String,DeleteHBase.DeleteMarker>>	getCellMarkerMap()
Map<String,DeleteHBase.DeleteMarker>	getFamilyMarkerMap()
RecordPort	getInput()
String	getOutputPath()	Get the output path.
void	mapCellMarker(String familyName, String qualifier, DeleteHBase.DeleteMarker marker)	Map DeleteMarker to a Cell.
void	mapFamilyMarker(String familyName, DeleteHBase.DeleteMarker marker)	Map DeleteMarker to a family.
void	setCellMarkerMap(Map<String,Map<String,DeleteHBase.DeleteMarker>> cellMarkerMap)
void	setFamilyMarkerMap(Map<String,DeleteHBase.DeleteMarker> familyMarkerMap)
void	setOutputPath(String outputPath)	Set the output path.

Methods inherited from class com.pervasive.datarush.hbase.KeyOperator

addFamily, effectiveConfiguration, getConfiguration, getFamilies, getFilesystem, getHiveMetastore, getQualifierFieldMap, getRootDirectory, getRowFieldMap, getTableName, getTimeFieldName, getZookeeperParentZNode, getZookeeperPort, getZookeeperQuorum, mapQualifier, mapRow, mapRowRecord, setConfiguration, setFamilies, setFilesystem, setHiveMetastore, setQualifierFieldMap, setRootDirectory, setRowFieldMap, setTableName, setTimeFieldName, setZookeeperParentZNode, setZookeeperPort, setZookeeperQuorum

Methods inherited from class com.pervasive.datarush.operators.AbstractLogicalOperator

disableParallelism, getInputPorts, getOutputPorts, newInput, newInput, newOutput, newRecordInput, newRecordInput, newRecordOutput, notifyError

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DeleteHBase

public DeleteHBase()

Method Detail

getFamilyMarkerMap

public Map<String,DeleteHBase.DeleteMarker> getFamilyMarkerMap()

setFamilyMarkerMap

public void setFamilyMarkerMap(Map<String,DeleteHBase.DeleteMarker> familyMarkerMap)

getCellMarkerMap

public Map<String,Map<String,DeleteHBase.DeleteMarker>> getCellMarkerMap()

setCellMarkerMap

public void setCellMarkerMap(Map<String,Map<String,DeleteHBase.DeleteMarker>> cellMarkerMap)

getOutputPath

public String getOutputPath()

Get the output path.

Returns:

output path.

setOutputPath

public void setOutputPath(String outputPath)

Set the output path. Set the location used to create HFiles. The Hfiles will be input to HBase by renaming if the same file system as HBase, by copying if a different file system than HBase. All files and directories created during the load process will be deleted upon completion.

Optional property. Defaults to hdfs:/user/userName/

Parameters:

outputPath - the output path.

getInput

public RecordPort getInput()

mapCellMarker

public void mapCellMarker(String familyName,
                          String qualifier,
                          DeleteHBase.DeleteMarker marker)

Map DeleteMarker to a Cell. This method is used to map a DeleteMarker to a select cell. Any previous mapping for the specified family, qualifier is replaced.

Parameters:

familyName - the name of the HBase column family.

qualifier - the HBase column qualifier identifying the cell to be mapped.

marker - the DeleteMarker to be mapped to the specified cell qualifier.

mapFamilyMarker

public void mapFamilyMarker(String familyName,
                            DeleteHBase.DeleteMarker marker)

Map DeleteMarker to a family. This method is used to map a DeleteMarker to a family as a sub table. Any previous mapping for the specified family is replaced.

Parameters:

familyName - the name of the HBase column family.

marker - the DeleteMarker to be mapped to the specified family.

compose

public void compose(CompositionContext ctx)

Description copied from class: CompositeOperator

Compose the body of this operator. Implementations should do the following:

1. Perform any validation of configuration, input types, etc
2. Instantiate and configure sub-operators, adding them to the provided context via the method OperatorComposable.add(O)
3. Create necessary connections via the method OperatorComposable.connect(P, P). This includes connections from the composite's input ports to sub-operators, connections between sub-operators, and connections from sub-operators output ports to the composite's output ports

Specified by:

compose in class CompositeOperator

Parameters:

ctx - the context