com.pervasive.datarush.cal.dr

Class AbstractDRClusterExecutor

java.lang.Object

com.pervasive.datarush.cal.dr.AbstractDRClusterExecutor

All Implemented Interfaces:

DistributedExecutorService

Direct Known Subclasses:

DRClusterExecutor, PseudoDistributedClusterExecutor, YarnClusterExecutor

public abstract class AbstractDRClusterExecutor
extends Object
implements DistributedExecutorService

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	AbstractDRClusterExecutor.CommandServiceHelper
protected static class	AbstractDRClusterExecutor.CommandServiceInfo

Field Summary

Fields

Modifier and Type	Field and Description
protected NodeAllocationProvider	allocationProvider
protected Object	DIST_MUTEX
protected List<AbstractDRClusterExecutor.CommandServiceInfo>	distExecutors
protected FileClient	fileClient
protected JobSpecifier	jobSpec

Constructor Summary

Constructors

Constructor and Description
AbstractDRClusterExecutor(FileClient fileClient, JobSpecifier jobSpec, NodeAllocationProvider allocationProvider)

Method Summary

Methods

Modifier and Type	Method and Description
protected abstract <R> MonitoredFuture<R>	createFuture(DistributedCallableBatch<R> batch, List<CommandHandleInfo> handles, ExecutionOptions options)
Map<String,Path>	drfsRoots()
NodeAllocationPlan	ensureInitialNodeAllocation() Returns the original node allocation (that which was allocated before any nodes failed).
<T> T	execute(DistributedCallable<T> command, ExecutionOptions options) Synchronously invokes the given callable.
<T> T	execute(DistributedCallableBatch<T> command, ExecutionOptions options) Synchronously invokes the given callable.
protected List<AbstractDRClusterExecutor.CommandServiceInfo>	fullPartitions()
DistributedCallableContext	getMasterContext() Returns the job master context.
protected ResourceAllocation	getMasterResources()
NodeStatistics	getNonParallelStatistics() Returns the non-parallel stats ( these are treated as the stats for the master )
protected abstract List<AbstractDRClusterExecutor.CommandServiceInfo>	start()
List<NodeStatistics>	startShutdown()
<R> MonitoredFuture<R>	submit(DistributedCallable<R> command, ExecutionOptions options) Asynchronously invokes the given callable.
<R> MonitoredFuture<R>	submit(DistributedCallableBatch<R> command, ExecutionOptions options) Asynchronously invokes the given callable.
protected abstract <R> List<CommandHandleInfo>	submit(DistributedCallableBatch<R> command, List<AbstractDRClusterExecutor.CommandServiceHelper> services) Overridden to either do parallel submit (for dist) or non-parallel (for pseudo-dist)
<T> MonitoredFuture<T>	submitMaster(MonitoredCallable<T> command) Asynchronous, local, and non-parallel submission of the given callable.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

fileClient

protected final FileClient fileClient

jobSpec

protected final JobSpecifier jobSpec

allocationProvider

protected final NodeAllocationProvider allocationProvider

DIST_MUTEX

protected final Object DIST_MUTEX

distExecutors

protected List<AbstractDRClusterExecutor.CommandServiceInfo> distExecutors

Constructor Detail

AbstractDRClusterExecutor

public AbstractDRClusterExecutor(FileClient fileClient,
                         JobSpecifier jobSpec,
                         NodeAllocationProvider allocationProvider)

Method Detail

ensureInitialNodeAllocation

public final NodeAllocationPlan ensureInitialNodeAllocation()

Description copied from interface: DistributedExecutorService

Returns the original node allocation (that which was allocated before any nodes failed). This method forces allocation of nodes if not already allocated

Specified by:

ensureInitialNodeAllocation in interface DistributedExecutorService

getNonParallelStatistics

public final NodeStatistics getNonParallelStatistics()

Returns the non-parallel stats ( these are treated as the stats for the master )

getMasterResources

protected ResourceAllocation getMasterResources()

fullPartitions

protected final List<AbstractDRClusterExecutor.CommandServiceInfo> fullPartitions()

start

protected abstract List<AbstractDRClusterExecutor.CommandServiceInfo> start()

getMasterContext

public final DistributedCallableContext getMasterContext()

Description copied from interface: DistributedExecutorService

Returns the job master context. This is the context that nodes will be handled when submitted using ExecutionOptions.MASTER.

Specified by:

getMasterContext in interface DistributedExecutorService

Returns:

the non-parallel/non-distributed context

submitMaster

public final <T> MonitoredFuture<T> submitMaster(MonitoredCallable<T> command)

Description copied from interface: DistributedExecutorService

Asynchronous, local, and non-parallel submission of the given callable.

Specified by:

submitMaster in interface DistributedExecutorService

Type Parameters:

T - the result type

Parameters:

command - the callable to run

Returns:

an execution handle

startShutdown

public final List<NodeStatistics> startShutdown()

submit

public final <R> MonitoredFuture<R> submit(DistributedCallable<R> command,
                            ExecutionOptions options)

Description copied from interface: DistributedExecutorService

Asynchronously invokes the given callable. The command will be invoked once per VirtualNodeID and results/status/exceptions merged together.

NOTE: Implementations are advised to implement this in terms of DistributedCallableBatch. This has the advantage of minimizing communication overhead. It also has the advantage of deserializing the command once-per-JVM which provides command implementations a convenient way of sharing (larger) objects within the same JVM. (DR relies on the assumption for certain performance optimizations).

Specified by:

submit in interface DistributedExecutorService

Type Parameters:

R - the type of results/status

Parameters:

command - the callable to execute

options - miscellaneous execution options

Returns:

an execution handle

submit

public final <R> MonitoredFuture<R> submit(DistributedCallableBatch<R> command,
                            ExecutionOptions options)

Description copied from interface: DistributedExecutorService

Asynchronously invokes the given callable. The command will be invoked once per physicalNodeID and results/status/exceptions merged together.

Specified by:

submit in interface DistributedExecutorService

Type Parameters:

R - the type of results/status

Parameters:

command - the callable to execute

options - miscellaneous execution options

Returns:

an execution handle

submit

protected abstract <R> List<CommandHandleInfo> submit(DistributedCallableBatch<R> command,
                                 List<AbstractDRClusterExecutor.CommandServiceHelper> services)

Overridden to either do parallel submit (for dist) or non-parallel (for pseudo-dist)

Parameters:

command -

services -

Returns:

execute

public final <T> T execute(DistributedCallable<T> command,
            ExecutionOptions options)
                throws DistributedExecutionException,
                       InterruptedException

Description copied from interface: DistributedExecutorService

Synchronously invokes the given callable. The command will be invoked once per VirtualNodeID and results/status/exceptions merged together.

NOTE: Implementations are advised to implement this in terms of DistributedCallableBatch. This has the advantage of minimizing communication overhead. It also has the advantage of deserializing the command once-per-JVM which provides command implementations a convenient way of sharing (larger) objects within the same JVM. (DR relies on the assumption for certain performance optimizations).

Specified by:

execute in interface DistributedExecutorService

Type Parameters:

T - the type of results/status

Parameters:

command - the command to execute

options - miscellaneous execution options

Returns:

the result

Throws:

DistributedExecutionException - if an exception occurs during execution

InterruptedException - if interrupted while waiting for results

execute

public final <T> T execute(DistributedCallableBatch<T> command,
            ExecutionOptions options)
                throws DistributedExecutionException,
                       InterruptedException

Description copied from interface: DistributedExecutorService

Synchronously invokes the given callable. The command will be invoked once per physicalNodeID and results/status/exceptions merged together.

Specified by:

execute in interface DistributedExecutorService

Type Parameters:

T - the type of results/status

Parameters:

command - the command to execute

options - miscellaneous execution options

Returns:

an execution handle

Throws:

DistributedExecutionException - if an exception occurs during execution

InterruptedException - if interrupted while waiting for results

createFuture

protected abstract <R> MonitoredFuture<R> createFuture(DistributedCallableBatch<R> batch,
                                  List<CommandHandleInfo> handles,
                                  ExecutionOptions options)

drfsRoots

public Map<String,Path> drfsRoots()