Module datarush.hadoop.core
Package com.actian.dataflow.cal.yarn.server

Class YarnClusterExecutor

java.lang.Object
com.pervasive.datarush.cal.dr.AbstractDRClusterExecutor
com.actian.dataflow.cal.yarn.server.YarnClusterExecutor
All Implemented Interfaces:
DistributedExecutorService
public class YarnClusterExecutor extends com.pervasive.datarush.cal.dr.AbstractDRClusterExecutor
The executor responsible for starting Dataflow worker containers within YARN. An executor is started by the cluster execution framework for every phase of a dataflow job. It is the executor's job to start and manage a container for every partition of a job. The executor is started within the application master and is invoked by the cluster execution framework as needed to manage containers.

  • Nested Class Summary

    Nested classes/interfaces inherited from class com.pervasive.datarush.cal.dr.AbstractDRClusterExecutor

    com.pervasive.datarush.cal.dr.AbstractDRClusterExecutor.CommandServiceHelper, com.pervasive.datarush.cal.dr.AbstractDRClusterExecutor.CommandServiceInfo

  • Field Summary

    Fields inherited from class com.pervasive.datarush.cal.dr.AbstractDRClusterExecutor

    allocationProvider, DIST_MUTEX, distExecutors, fileClient, jobSpec

  • Constructor Summary

    Constructors
    Constructor
    Description
    YarnClusterExecutor(FileClient fileClient, JobSpecifier jobSpec, com.pervasive.datarush.hadoop.shims.yarn.ContainerManager containerMgr, com.pervasive.datarush.cluster.ExecutorOptions execOptions, Path cacheLocation, Path libArchiveLocation, com.pervasive.datarush.cluster.preferences.ClusterPreferences clusterPrefs, Map<String,ApplicationResource> appResources)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    protected <R> MonitoredFuture<R>
    createFuture(DistributedCallableBatch<R> batch, List<com.pervasive.datarush.cal.dr.CommandHandleInfo> handles, ExecutionOptions options)
     
    Map<String,Path>
    drfsRoots()
     
    protected ResourceAllocation
    getMasterResources()
     
    protected final com.pervasive.datarush.cal.dr.ParallelStartupHelper
    parallelStartup()
     
    protected List<com.pervasive.datarush.cal.dr.AbstractDRClusterExecutor.CommandServiceInfo>
    start()
     
    protected <R> List<com.pervasive.datarush.cal.dr.CommandHandleInfo>
    submit(DistributedCallableBatch<R> command, List<com.pervasive.datarush.cal.dr.AbstractDRClusterExecutor.CommandServiceHelper> services)
    Overridden to either do parallel submit (for dist) or non-parallel (for pseudo-dist)

    Methods inherited from class com.pervasive.datarush.cal.dr.AbstractDRClusterExecutor

    ensureInitialNodeAllocation, execute, execute, fullPartitions, getMasterContext, getNonParallelStatistics, startShutdown, submit, submit, submitMaster

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • YarnClusterExecutor

      public YarnClusterExecutor(FileClient fileClient, JobSpecifier jobSpec, com.pervasive.datarush.hadoop.shims.yarn.ContainerManager containerMgr, com.pervasive.datarush.cluster.ExecutorOptions execOptions, Path cacheLocation, Path libArchiveLocation, com.pervasive.datarush.cluster.preferences.ClusterPreferences clusterPrefs, Map<String,ApplicationResource> appResources)

  • Method Details

    • start

      protected List<com.pervasive.datarush.cal.dr.AbstractDRClusterExecutor.CommandServiceInfo> start()
      Specified by:
      start in class com.pervasive.datarush.cal.dr.AbstractDRClusterExecutor

    • submit

      protected <R> List<com.pervasive.datarush.cal.dr.CommandHandleInfo> submit(DistributedCallableBatch<R> command, List<com.pervasive.datarush.cal.dr.AbstractDRClusterExecutor.CommandServiceHelper> services)
      Description copied from class: com.pervasive.datarush.cal.dr.AbstractDRClusterExecutor
      Overridden to either do parallel submit (for dist) or non-parallel (for pseudo-dist)
      Specified by:
      submit in class com.pervasive.datarush.cal.dr.AbstractDRClusterExecutor
      Returns:

    • createFuture

      protected <R> MonitoredFuture<R> createFuture(DistributedCallableBatch<R> batch, List<com.pervasive.datarush.cal.dr.CommandHandleInfo> handles, ExecutionOptions options)
      Specified by:
      createFuture in class com.pervasive.datarush.cal.dr.AbstractDRClusterExecutor

    • getMasterResources

      protected ResourceAllocation getMasterResources()
      Overrides:
      getMasterResources in class com.pervasive.datarush.cal.dr.AbstractDRClusterExecutor

    • drfsRoots

      public Map<String,Path> drfsRoots()
      Overrides:
      drfsRoots in class com.pervasive.datarush.cal.dr.AbstractDRClusterExecutor

    • parallelStartup

      protected final com.pervasive.datarush.cal.dr.ParallelStartupHelper parallelStartup()