Module datarush.hadoop.core

Package com.actian.dataflow.cal.yarn.client

Class YarnClusterProvider

java.lang.Object

com.actian.dataflow.cal.yarn.client.YarnClusterProvider

All Implemented Interfaces:

ClusterProvider

public class YarnClusterProvider
extends Object
implements ClusterProvider

Implementation of the ClusterProvider

interface that supports interacting with a YARN-based Hadoop cluster. This is the entry point into providing the needed services for interacting with YARN.

The cluster specifier uses the scheme yarn. The host name and port are the address of the Dataflow Cluster Manager. The Cluster Manager is used as the gateway into YARN. Hadoop configuration information can be bootstrapped from the Cluster Manager and used to create initial connections into the YARN Resource Manager.

Field Summary

Fields

Modifier and Type	Field	Description
static String	PROTOCOL

Constructor Summary

Constructors

Constructor	Description
YarnClusterProvider()

Method Summary

Modifier and Type	Method	Description
JobExecutorService	createExecutor(FileClient fileClient, JobSpecifier spec)	Create a new job executor service.
String	getDefaultStorageManagementPath(ClusterSpecifier spec)	Returns the default root path for this cluster provider.
JobHistoryService	getJobHistoryService(FileClient fileClient, ClusterSpecifier spec)
boolean	isProviderFor(ClusterSpecifier spec)	Returns whether this provider is the correct provider for the given specifier.
void	validateConnection(FileClient fileClient, ClusterSpecifier spec)	Gives the provider an opportunity to validate the specifier

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PROTOCOL

public static final String PROTOCOL

See Also:

Constant Field Values

Constructor Detail

YarnClusterProvider

public YarnClusterProvider()

Method Detail

isProviderFor

public boolean isProviderFor(ClusterSpecifier spec)

Description copied from interface: ClusterProvider

Returns whether this provider is the correct provider for the given specifier.

Specified by:

isProviderFor in interface ClusterProvider

Parameters:

spec - the specifier

Returns:

whether this provider knows how to handle the given specifier

validateConnection

public void validateConnection(FileClient fileClient,
                               ClusterSpecifier spec)

Description copied from interface: ClusterProvider

Gives the provider an opportunity to validate the specifier

Specified by:

validateConnection in interface ClusterProvider

spec - the specifier

getDefaultStorageManagementPath

public String getDefaultStorageManagementPath(ClusterSpecifier spec)

Description copied from interface: ClusterProvider

Returns the default root path for this cluster provider. For example, within the DR cluster, the default path is drfs://clustermanager:port.

Specified by:

getDefaultStorageManagementPath in interface ClusterProvider

Parameters:

spec - the cluster specifier

Returns:

the default root path

createExecutor

public JobExecutorService createExecutor(FileClient fileClient,
                                         JobSpecifier spec)

Description copied from interface: ClusterProvider

Create a new job executor service.

Specified by:

createExecutor in interface ClusterProvider

Parameters:

fileClient - the file client to use

spec - the job specifier

Returns:

a new job executor service

getJobHistoryService

public JobHistoryService getJobHistoryService(FileClient fileClient,
                                              ClusterSpecifier spec)

Specified by:

getJobHistoryService in interface ClusterProvider