com.pervasive.datarush.cluster

Class ClusterManagerImpl

java.lang.Object

com.pervasive.datarush.cluster.ClusterManagerImpl

All Implemented Interfaces:

JobHistoryService, ClusterManager, NodeTracker.NodeEventListener, ClusterPreferencesChangeListener, SimpleRMI.ServerListener

public class ClusterManagerImpl
extends Object
implements ClusterManager, SimpleRMI.ServerListener, ClusterPreferencesChangeListener, NodeTracker.NodeEventListener

Method Summary

Methods

Modifier and Type	Method and Description
void	addContainersToAppMaster(JobInformation jobInfo, List<ContainerDescriptor> containers) Add descriptors of containers that are allocated to the application master.
void	addListener(ClusterManagerListener listener)
String	allocateMaster(GUID jobGUID) Allocates the master node for the given job
NodeAllocationPlan	allocateWorkers(GUID jobGUID) Allocates the worker nodes for the given job
void	checkStatus()
void	clearClasspathCache() Clears the shared classpath cache
void	deregisterAppMaster(JobInformation jobInfo) Deregister an external application master.
void	destroy() Called upon server shutdown.
void	ensureRegistered(NodeDescriptor node) Adds a node to the cluster registry.
void	failJob(GUID jobGUID) Manually force job fail
List<MasterDescriptor>	getActiveAppMasters() Get a list of active (external) application masters.
List<JobInformation>	getAllocatedJobs()
InetSocketAddress	getClusterAddress()
ClusterPreferences	getConfiguration() Get the global cluster configuration (not for a specific node).
ClusterPreferences	getConfiguration(String node) Requests the configuration for the given node.
List<ContainerDescriptor>	getContainers(MasterDescriptor masterDescriptor) Get the list of containers allocated to an application master.
ExecutorOptions	getDefaultOptions() Gets the default options configured for remote fragment execution.
protected Integer	getJobServiceStatus(GUID jobGUID, WorkerRole role)
Integer	getJobStatus(GUID jobGUID) Get the general run status of the job -1 - job in progress 0 - job completed normally 1+ - job has errors
NodeDescriptor	getNodeDescriptor(String name) Gets the node descriptor for the specified node
NodeMapping	getNodes(boolean refresh) Gets node info for all nodes in the cluster
List<MasterDescriptor>	getPendingAppMasters() Get a list of pending (external) application masters.
List<NodeAllocationRequest>	getPendingJobs()
String	getStatus() Gets a description of the current status of the node
String	getVersion() Returns the version number of this ClusterManager instance
void	init(GenericTCPServer.ServerContext context) Called upon server start to inform the handler of context information such as the listening address.
String	initializeNodeAllocation(SerializationBridge<NodeAllocationRequest> request)
void	jobFinished(JobStatistics statistics) Called by command server upon exit to report final statistics
void	killJob(GUID jobGUID) Administrative job kill
void	nodeStarted(NodeDescriptor node)
InputStream	openClusterLogFile(LogFileOptions options) Opens the log file for cluster manager for reading
void	preferencesChanged(ClusterPreferenceChangeEvent e) Event fired to indicate change of cluster preferences
void	registerAppMaster(MasterDescriptor masterDescriptor) Register an application master started outside of the context of the cluster manager.
void	removeListener(ClusterManagerListener listener)
void	rotateClusterLog() Rotates the cluster manager log
void	rotateNodeLogs() Rotates all of the node manager logs
void	shutdown() Requests a shutdown of the cluster.
void	shutdownNode(String name) Shuts down the given node.
static ClusterManagerImpl	start(ClusterPreferencesProvider prefProvider, int port, boolean daemon, String ipaddress)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

checkStatus

public void checkStatus()

getClusterAddress

public InetSocketAddress getClusterAddress()

getDefaultOptions

public ExecutorOptions getDefaultOptions()

Description copied from interface: ClusterManager

Gets the default options configured for remote fragment execution.

Specified by:

getDefaultOptions in interface ClusterManager

Returns:

the default settings for execution of remote fragments

nodeStarted

public void nodeStarted(NodeDescriptor node)

Specified by:

nodeStarted in interface NodeTracker.NodeEventListener

getNodes

public NodeMapping getNodes(boolean refresh)

Description copied from interface: ClusterManager

Gets node info for all nodes in the cluster

Specified by:

getNodes in interface ClusterManager

Parameters:

refresh - if true, pings each node to ensure that it only returns live nodes

Returns:

valid node names for the cluster

ensureRegistered

public void ensureRegistered(NodeDescriptor node)

Description copied from interface: ClusterManager

Adds a node to the cluster registry. If a node is already registered with the specified name, it is replaced.

Specified by:

ensureRegistered in interface ClusterManager

Parameters:

node - the node information

getNodeDescriptor

public NodeDescriptor getNodeDescriptor(String name)

Description copied from interface: ClusterManager

Gets the node descriptor for the specified node

Specified by:

getNodeDescriptor in interface ClusterManager

Parameters:

name - the name of the node requested

Returns:

the node descriptor

getConfiguration

public ClusterPreferences getConfiguration()
                                    throws RMIException

Description copied from interface: ClusterManager

Get the global cluster configuration (not for a specific node).

Specified by:

getConfiguration in interface ClusterManager

Returns:

global configuration

Throws:

RMIException - thrown for communication errors

registerAppMaster

public void registerAppMaster(MasterDescriptor masterDescriptor)
                       throws RMIException

Description copied from interface: ClusterManager

Register an application master started outside of the context of the cluster manager.

Specified by:

registerAppMaster in interface ClusterManager

Parameters:

masterDescriptor - descriptor of the application master

Throws:

RMIException - thrown for communication errors

deregisterAppMaster

public void deregisterAppMaster(JobInformation jobInfo)
                         throws RMIException

Description copied from interface: ClusterManager

Deregister an external application master. The status of the application will no longer be tracked by the cluster manager.

Specified by:

deregisterAppMaster in interface ClusterManager

Throws:

RMIException - thrown for communication errors

getPendingAppMasters

public List<MasterDescriptor> getPendingAppMasters()
                                            throws RMIException

Description copied from interface: ClusterManager

Get a list of pending (external) application masters. A pending application master is up and running, but is awaiting worker container resources.

Specified by:

getPendingAppMasters in interface ClusterManager

Returns:

pending application masters

Throws:

RMIException - throw for communication errors

getActiveAppMasters

public List<MasterDescriptor> getActiveAppMasters()
                                           throws RMIException

Description copied from interface: ClusterManager

Get a list of active (external) application masters. Active application masters have allocated resources and running containers.

Specified by:

getActiveAppMasters in interface ClusterManager

Returns:

active application masters

Throws:

RMIException - throw for communication errors

addContainersToAppMaster

public void addContainersToAppMaster(JobInformation jobInfo,
                            List<ContainerDescriptor> containers)
                              throws RMIException

Description copied from interface: ClusterManager

Add descriptors of containers that are allocated to the application master. This may be called more than once per application master.

Specified by:

addContainersToAppMaster in interface ClusterManager

containers - containers allocated to the application master

Throws:

RMIException - thrown for communication errors

getContainers

public List<ContainerDescriptor> getContainers(MasterDescriptor masterDescriptor)
                                        throws RMIException

Description copied from interface: ClusterManager

Get the list of containers allocated to an application master.

Specified by:

getContainers in interface ClusterManager

Parameters:

masterDescriptor - application master

Returns:

list of containers allocated to the application master

Throws:

RMIException - throws for communication errors

shutdown

public void shutdown()

Description copied from interface: ClusterManager

Requests a shutdown of the cluster. All registered node managers will also be shutdown.

Specified by:

shutdown in interface ClusterManager

addListener

public void addListener(ClusterManagerListener listener)

removeListener

public void removeListener(ClusterManagerListener listener)

start

public static ClusterManagerImpl start(ClusterPreferencesProvider prefProvider,
                       int port,
                       boolean daemon,
                       String ipaddress)

getStatus

public String getStatus()

Description copied from interface: ClusterManager

Gets a description of the current status of the node

Specified by:

getStatus in interface ClusterManager

Returns:

a description of the node's status

init

public void init(GenericTCPServer.ServerContext context)

Description copied from interface: SimpleRMI.ServerListener

Called upon server start to inform the handler of context information such as the listening address. This method will be called prior to accepting any requests.

Specified by:

init in interface SimpleRMI.ServerListener

Parameters:

context - the server context

destroy

public void destroy()

Description copied from interface: SimpleRMI.ServerListener

Called upon server shutdown. This method will be called only after the last request has finished.

Specified by:

destroy in interface SimpleRMI.ServerListener

getConfiguration

public ClusterPreferences getConfiguration(String node)

Description copied from interface: ClusterManager

Requests the configuration for the given node.

Specified by:

getConfiguration in interface ClusterManager

Parameters:

node - the node requesting the configuration

Returns:

the configuration for the given node

preferencesChanged

public void preferencesChanged(ClusterPreferenceChangeEvent e)

Description copied from interface: ClusterPreferencesChangeListener

Event fired to indicate change of cluster preferences

Specified by:

preferencesChanged in interface ClusterPreferencesChangeListener

Parameters:

e - the change event

jobFinished

public void jobFinished(JobStatistics statistics)

Description copied from interface: JobHistoryService

Called by command server upon exit to report final statistics

Specified by:

jobFinished in interface JobHistoryService

Parameters:

statistics - the final statistics

openClusterLogFile

public InputStream openClusterLogFile(LogFileOptions options)
                               throws IOException

Description copied from interface: ClusterManager

Opens the log file for cluster manager for reading

Specified by:

openClusterLogFile in interface ClusterManager

Returns:

stream to read the log file

Throws:

IOException

killJob

public void killJob(GUID jobGUID)

Description copied from interface: ClusterManager

Administrative job kill

Specified by:

killJob in interface ClusterManager

failJob

public void failJob(GUID jobGUID)

Description copied from interface: ClusterManager

Manually force job fail

Specified by:

failJob in interface ClusterManager

getJobStatus

public Integer getJobStatus(GUID jobGUID)

Description copied from interface: ClusterManager

Get the general run status of the job -1 - job in progress 0 - job completed normally 1+ - job has errors

Specified by:

getJobStatus in interface ClusterManager

Returns:

integer describing run status

getJobServiceStatus

protected Integer getJobServiceStatus(GUID jobGUID,
                          WorkerRole role)

clearClasspathCache

public void clearClasspathCache()
                         throws RMIException,
                                IOException

Description copied from interface: ClusterManager

Clears the shared classpath cache

Specified by:

clearClasspathCache in interface ClusterManager

Throws:

IOException - if unable to delete (would typically happen if job is running). Generally we don't care so much about cleanup failures, but here we need to let the user know because otherwise case may still be dirty.

RMIException

getVersion

public String getVersion()

Description copied from interface: ClusterManager

Returns the version number of this ClusterManager instance

Specified by:

getVersion in interface ClusterManager

Returns:

the version

shutdownNode

public void shutdownNode(String name)
                  throws RMIException

Description copied from interface: ClusterManager

Shuts down the given node.

Specified by:

shutdownNode in interface ClusterManager

Parameters:

name - the entry to remove

Throws:

RMIException

rotateClusterLog

public void rotateClusterLog()
                      throws RMIException

Description copied from interface: ClusterManager

Rotates the cluster manager log

Specified by:

rotateClusterLog in interface ClusterManager

Throws:

RMIException

rotateNodeLogs

public void rotateNodeLogs()
                    throws RMIException,
                           IOException

Description copied from interface: ClusterManager

Rotates all of the node manager logs

Specified by:

rotateNodeLogs in interface ClusterManager

Throws:

RMIException

IOException - if unable to rotate

initializeNodeAllocation

public String initializeNodeAllocation(SerializationBridge<NodeAllocationRequest> request)

Specified by:

initializeNodeAllocation in interface ClusterManager

allocateMaster

public String allocateMaster(GUID jobGUID)

Description copied from interface: ClusterManager

Allocates the master node for the given job

Specified by:

allocateMaster in interface ClusterManager

Returns:

allocateWorkers

public NodeAllocationPlan allocateWorkers(GUID jobGUID)

Description copied from interface: ClusterManager

Allocates the worker nodes for the given job

Specified by:

allocateWorkers in interface ClusterManager

Returns:

the node allocation plan

getAllocatedJobs

public List<JobInformation> getAllocatedJobs()

Specified by:

getAllocatedJobs in interface ClusterManager

getPendingJobs

public List<NodeAllocationRequest> getPendingJobs()

Specified by:

getPendingJobs in interface ClusterManager