Module datarush.commons
Package com.pervasive.datarush.cal

Interface DistributedExecutorService

All Known Subinterfaces:
PseudoDistributedExecutorService
All Known Implementing Classes:
com.pervasive.datarush.cal.dr.AbstractDRClusterExecutor, YarnClusterExecutor
public interface DistributedExecutorService
Distributed equivalent of the ExecutorService concept. To obtain a handle to an executor service, use JobExecutorServiceFactory.

  • Method Details

    • ensureInitialNodeAllocation

      NodeAllocationPlan ensureInitialNodeAllocation()
      Returns the original node allocation (that which was allocated before any nodes failed). This method forces allocation of nodes if not already allocated

    • getMasterContext

      DistributedCallableContext getMasterContext()
      Returns the job master context. This is the context that nodes will be handled when submitted using ExecutionOptions.MASTER.
      Returns:
      the non-parallel/non-distributed context

    • submitMaster

      <T> MonitoredFuture<T> submitMaster(MonitoredCallable<T> callable)
      Asynchronous, local, and non-parallel submission of the given callable.
      Type Parameters:
      T - the result type
      Parameters:
      callable - the callable to run
      Returns:
      an execution handle

    • submit

      <T> MonitoredFuture<T> submit(DistributedCallable<T> callable, ExecutionOptions options)
      Asynchronously invokes the given callable. The command will be invoked once per VirtualNodeID and results/status/exceptions merged together.

      NOTE: Implementations are advised to implement this in terms of DistributedCallableBatch. This has the advantage of minimizing communication overhead. It also has the advantage of deserializing the command once-per-JVM which provides command implementations a convenient way of sharing (larger) objects within the same JVM. (DR relies on the assumption for certain performance optimizations).
      Type Parameters:
      T - the type of results/status
      Parameters:
      callable - the callable to execute
      options - miscellaneous execution options
      Returns:
      an execution handle

    • submit

      <T> MonitoredFuture<T> submit(DistributedCallableBatch<T> callable, ExecutionOptions options)
      Asynchronously invokes the given callable. The command will be invoked once per physicalNodeID and results/status/exceptions merged together.

      Type Parameters:
      T - the type of results/status
      Parameters:
      callable - the callable to execute
      options - miscellaneous execution options
      Returns:
      an execution handle

    • execute

      <T> T execute(DistributedCallable<T> command, ExecutionOptions options) throws DistributedExecutionException, InterruptedException
      Synchronously invokes the given callable. The command will be invoked once per VirtualNodeID and results/status/exceptions merged together.

      NOTE: Implementations are advised to implement this in terms of DistributedCallableBatch. This has the advantage of minimizing communication overhead. It also has the advantage of deserializing the command once-per-JVM which provides command implementations a convenient way of sharing (larger) objects within the same JVM. (DR relies on the assumption for certain performance optimizations).
      Type Parameters:
      T - the type of results/status
      Parameters:
      command - the command to execute
      options - miscellaneous execution options
      Returns:
      the result
      Throws:
      DistributedExecutionException - if an exception occurs during execution
      InterruptedException - if interrupted while waiting for results

    • execute

      <T> T execute(DistributedCallableBatch<T> command, ExecutionOptions options) throws DistributedExecutionException, InterruptedException
      Synchronously invokes the given callable. The command will be invoked once per physicalNodeID and results/status/exceptions merged together.

      Type Parameters:
      T - the type of results/status
      Parameters:
      command - the command to execute
      options - miscellaneous execution options
      Returns:
      an execution handle
      Throws:
      DistributedExecutionException - if an exception occurs during execution
      InterruptedException - if interrupted while waiting for results