Package org.apache.hadoop.yarn.server.nodemanager

Class ContainerExecutor

java.lang.Object
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor
All Implemented Interfaces:
org.apache.hadoop.conf.Configurable
Direct Known Subclasses:
DefaultContainerExecutor, LinuxContainerExecutor
public abstract class ContainerExecutor extends Object implements org.apache.hadoop.conf.Configurable
This class is abstraction of the mechanism used to launch a container on the underlying OS. All executor implementations must extend ContainerExecutor.

  • Field Details

  • Constructor Details

    • ContainerExecutor

      public ContainerExecutor()

  • Method Details

    • setConf

      public void setConf(org.apache.hadoop.conf.Configuration conf)
      Specified by:
      setConf in interface org.apache.hadoop.conf.Configurable

    • getConf

      public org.apache.hadoop.conf.Configuration getConf()
      Specified by:
      getConf in interface org.apache.hadoop.conf.Configurable

    • init

      public abstract void init(Context nmContext) throws IOException
      Run the executor initialization steps. Verify that the necessary configs and permissions are in place.
      Parameters:
      nmContext - Context of NM
      Throws:
      IOException - if initialization fails

    • start

      public void start()

    • stop

      public void stop()

    • localizeClasspathJar

      public org.apache.hadoop.fs.Path localizeClasspathJar(org.apache.hadoop.fs.Path jarPath, org.apache.hadoop.fs.Path target, String owner) throws IOException
      This function localizes the JAR file on-demand. On Windows the ContainerLaunch creates a temporary special JAR manifest of other JARs to workaround the CLASSPATH length. In a secure cluster this JAR must be localized so that the container has access to it. The default implementation returns the classpath passed to it, which is expected to have been created in the node manager's fprivate folder, which will not work with secure Windows clusters.
      Parameters:
      jarPath - the path to the JAR to localize
      target - the directory where the JAR file should be localized
      owner - the name of the user who should own the localized file
      Returns:
      the path to the localized JAR file
      Throws:
      IOException - if localization fails

    • startLocalizer

      public abstract void startLocalizer(LocalizerStartContext ctx) throws IOException, InterruptedException, org.apache.hadoop.yarn.exceptions.ConfigurationException
      Prepare the environment for containers in this application to execute. 
       For $x in local.dirs
   create $x/$user/$appId
 Copy $nmLocal/appTokens -> $N/$user/$appId
 For $rsrc in private resources
   Copy $rsrc -> $N/$user/filecache/[idef]
 For $rsrc in job resources
   Copy $rsrc -> $N/$user/$appId/filecache/idef
      Parameters:
      ctx - LocalizerStartContext that encapsulates necessary information for starting a localizer.
      Throws:
      IOException - for most application init failures
      InterruptedException - if application init thread is halted by NM
      org.apache.hadoop.yarn.exceptions.ConfigurationException - if config error was found

    • prepareContainer

      public void prepareContainer(ContainerPrepareContext ctx) throws IOException
      Prepare the container prior to the launch environment being written.
      Parameters:
      ctx - Encapsulates information necessary for launching containers.
      Throws:
      IOException - if errors occur during container preparation

    • launchContainer

      public abstract int launchContainer(ContainerStartContext ctx) throws IOException, org.apache.hadoop.yarn.exceptions.ConfigurationException
      Launch the container on the node. This is a blocking call and returns only when the container exits.
      Parameters:
      ctx - Encapsulates information necessary for launching containers.
      Returns:
      the return status of the launch
      Throws:
      IOException - if the container launch fails
      org.apache.hadoop.yarn.exceptions.ConfigurationException - if config error was found

    • relaunchContainer

      public abstract int relaunchContainer(ContainerStartContext ctx) throws IOException, org.apache.hadoop.yarn.exceptions.ConfigurationException
      Relaunch the container on the node. This is a blocking call and returns only when the container exits.
      Parameters:
      ctx - Encapsulates information necessary for relaunching containers.
      Returns:
      the return status of the relaunch
      Throws:
      IOException - if the container relaunch fails
      org.apache.hadoop.yarn.exceptions.ConfigurationException - if config error was found

    • signalContainer

      public abstract boolean signalContainer(ContainerSignalContext ctx) throws IOException
      Signal container with the specified signal.
      Parameters:
      ctx - Encapsulates information necessary for signaling containers.
      Returns:
      returns true if the operation succeeded
      Throws:
      IOException - if signaling the container fails

    • reapContainer

      public abstract boolean reapContainer(ContainerReapContext ctx) throws IOException
      Perform the steps necessary to reap the container.
      Parameters:
      ctx - Encapsulates information necessary for reaping containers.
      Returns:
      returns true if the operation succeeded.
      Throws:
      IOException - if reaping the container fails.

    • execContainer

      public abstract org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair execContainer(ContainerExecContext ctx) throws ContainerExecutionException
      Perform interactive docker command into running container.
      Parameters:
      ctx - Encapsulates information necessary for exec containers.
      Returns:
      return input/output stream if the operation succeeded.
      Throws:
      ContainerExecutionException - if container exec fails.

    • deleteAsUser

      public abstract void deleteAsUser(DeletionAsUserContext ctx) throws IOException, InterruptedException
      Delete specified directories as a given user.
      Parameters:
      ctx - Encapsulates information necessary for deletion.
      Throws:
      IOException - if delete fails
      InterruptedException - if interrupted while waiting for the deletion operation to complete

    • symLink

      public abstract void symLink(String target, String symlink) throws IOException
      Create a symlink file which points to the target.
      Parameters:
      target - The target for symlink
      symlink - the symlink file
      Throws:
      IOException - Error when creating symlinks

    • isContainerAlive

      public abstract boolean isContainerAlive(ContainerLivenessContext ctx) throws IOException
      Check if a container is alive.
      Parameters:
      ctx - Encapsulates information necessary for container liveness check.
      Returns:
      true if container is still alive
      Throws:
      IOException - if there is a failure while checking the container status

    • getLocalResources

      public Map<String,org.apache.hadoop.yarn.api.records.LocalResource> getLocalResources(Container container) throws IOException
      Throws:
      IOException

    • updateYarnSysFS

      public abstract void updateYarnSysFS(Context ctx, String user, String appId, String spec) throws IOException
      Update cluster information inside container.
      Parameters:
      ctx - ContainerRuntimeContext
      user - Owner of application
      appId - YARN application ID
      spec - Service Specification
      Throws:
      IOException - if there is a failure while writing spec to disk

    • reacquireContainer

      public int reacquireContainer(ContainerReacquisitionContext ctx) throws IOException, InterruptedException
      Recover an already existing container. This is a blocking call and returns only when the container exits. Note that the container must have been activated prior to this call.
      Parameters:
      ctx - encapsulates information necessary to reacquire container
      Returns:
      The exit code of the pre-existing container
      Throws:
      IOException - if there is a failure while reacquiring the container
      InterruptedException - if interrupted while waiting to reacquire the container

    • writeLaunchEnv

      public void writeLaunchEnv(OutputStream out, Map<String,String> environment, Map<org.apache.hadoop.fs.Path,List<String>> resources, List<String> command, org.apache.hadoop.fs.Path logDir, String user, LinkedHashSet<String> nmVars) throws IOException
      This method writes out the launch environment of a container to the default container launch script. For the default container script path see ContainerLaunch.CONTAINER_SCRIPT.
      Parameters:
      out - the output stream to which the environment is written (usually a script file which will be executed by the Launcher)
      environment - the environment variables and their values
      resources - the resources which have been localized for this container. Symlinks will be created to these localized resources
      command - the command that will be run
      logDir - the log dir to which to copy debugging information
      user - the username of the job owner
      nmVars - the set of environment vars that are explicitly set by NM
      Throws:
      IOException - if any errors happened writing to the OutputStream, while creating symlinks

    • writeLaunchEnv

      @VisibleForTesting public void writeLaunchEnv(OutputStream out, Map<String,String> environment, Map<org.apache.hadoop.fs.Path,List<String>> resources, List<String> command, org.apache.hadoop.fs.Path logDir, String user, String outFilename, LinkedHashSet<String> nmVars) throws IOException
      This method writes out the launch environment of a container to a specified path.
      Parameters:
      out - the output stream to which the environment is written (usually a script file which will be executed by the Launcher)
      environment - the environment variables and their values
      resources - the resources which have been localized for this container. Symlinks will be created to these localized resources
      command - the command that will be run
      logDir - the log dir to which to copy debugging information
      user - the username of the job owner
      outFilename - the path to which to write the launch environment
      nmVars - the set of environment vars that are explicitly set by NM
      Throws:
      IOException - if any errors happened writing to the OutputStream, while creating symlinks

    • readDirAsUser

      protected File[] readDirAsUser(String user, org.apache.hadoop.fs.Path dir)
      Return the files in the target directory. If retrieving the list of files requires specific access rights, that access will happen as the specified user. The list will not include entries for "." or "..".
      Parameters:
      user - the user as whom to access the target directory
      dir - the target directory
      Returns:
      a list of files in the target directory

    • logOutput

      protected void logOutput(String output)
      Log each line of the output string as INFO level log messages.
      Parameters:
      output - the output string to log

    • getPidFilePath

      protected org.apache.hadoop.fs.Path getPidFilePath(org.apache.hadoop.yarn.api.records.ContainerId containerId)
      Get the pidFile of the container.
      Parameters:
      containerId - the container ID
      Returns:
      the path of the pid-file for the given containerId.

    • getRunCommand

      protected String[] getRunCommand(String command, String groupId, String userName, org.apache.hadoop.fs.Path pidFile, org.apache.hadoop.conf.Configuration config)
      Return a command line to execute the given command in the OS shell. On Windows, the {code}groupId{code} parameter can be used to launch and associate the given GID with a process group. On non-Windows hosts, the {code}groupId{code} parameter is ignored.
      Parameters:
      command - the command to execute
      groupId - the job owner's GID
      userName - the job owner's username
      pidFile - the path to the container's PID file
      config - the configuration
      Returns:
      the command line to execute

    • getRunCommand

      protected String[] getRunCommand(String command, String groupId, String userName, org.apache.hadoop.fs.Path pidFile, org.apache.hadoop.conf.Configuration config, org.apache.hadoop.yarn.api.records.Resource resource)
      Return a command line to execute the given command in the OS shell. On Windows, the {code}groupId{code} parameter can be used to launch and associate the given GID with a process group. On non-Windows hosts, the {code}groupId{code} parameter is ignored.
      Parameters:
      command - the command to execute
      groupId - the job owner's GID for Windows. On other operating systems it is ignored.
      userName - the job owner's username for Windows. On other operating systems it is ignored.
      pidFile - the path to the container's PID file on Windows. On other operating systems it is ignored.
      config - the configuration
      resource - on Windows this parameter controls memory and CPU limits. If null, no limits are set. On other operating systems it is ignored.
      Returns:
      the command line to execute

    • getRunCommandForWindows

      protected String[] getRunCommandForWindows(String command, String groupId, String userName, org.apache.hadoop.fs.Path pidFile, org.apache.hadoop.conf.Configuration config, org.apache.hadoop.yarn.api.records.Resource resource)
      Return a command line to execute the given command in the OS shell. The {code}groupId{code} parameter can be used to launch and associate the given GID with a process group.
      Parameters:
      command - the command to execute
      groupId - the job owner's GID
      userName - the job owner's username
      pidFile - the path to the container's PID file
      config - the configuration
      resource - this parameter controls memory and CPU limits. If null, no limits are set.
      Returns:
      the command line to execute

    • getRunCommandForOther

      protected String[] getRunCommandForOther(String command, org.apache.hadoop.conf.Configuration config)
      Return a command line to execute the given command in the OS shell.
      Parameters:
      command - the command to execute
      config - the configuration
      Returns:
      the command line to execute

    • isContainerActive

      protected boolean isContainerActive(org.apache.hadoop.yarn.api.records.ContainerId containerId)
      Return whether the container is still active.
      Parameters:
      containerId - the target container's ID
      Returns:
      true if the container is active

    • getNMEnvVar

      @VisibleForTesting protected String getNMEnvVar(String varname)

    • activateContainer

      public void activateContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.fs.Path pidFilePath)
      Mark the container as active.
      Parameters:
      containerId - the container ID
      pidFilePath - the path where the executor should write the PID of the launched process

    • getIpAndHost

      public String[] getIpAndHost(Container container) throws ContainerExecutionException
      Throws:
      ContainerExecutionException

    • getLocalIpAndHost

      public static String[] getLocalIpAndHost(Container container)

    • deactivateContainer

      public void deactivateContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)
      Mark the container as inactive. For inactive containers this method has no effect.
      Parameters:
      containerId - the container ID

    • pauseContainer

      public void pauseContainer(Container container)
      Pause the container. The default implementation is to raise a kill event. Specific executor implementations can override this behavior.
      Parameters:
      container - the Container

    • resumeContainer

      public void resumeContainer(Container container)
      Resume the container from pause state. The default implementation ignores this event. Specific implementations can override this behavior.
      Parameters:
      container - the Container

    • cleanupBeforeRelaunch

      public void cleanupBeforeRelaunch(Container container) throws IOException, InterruptedException
      Perform any cleanup before the next launch of the container.
      Parameters:
      container - container
      Throws:
      IOException
      InterruptedException

    • getProcessId

      public String getProcessId(org.apache.hadoop.yarn.api.records.ContainerId containerID)
      Get the process-identifier for the container.
      Parameters:
      containerID - the container ID
      Returns:
      the process ID of the container if it has already launched, or null otherwise

    • getExposedPorts

      public String getExposedPorts(Container container) throws ContainerExecutionException
      Throws:
      ContainerExecutionException