Package org.apache.hadoop.yarn.server.resourcemanager.scheduler

Interface YarnScheduler

All Superinterfaces:
org.apache.hadoop.yarn.event.EventHandler<SchedulerEvent>
All Known Subinterfaces:
MutableConfScheduler, PreemptableResourceScheduler, ResourceScheduler
All Known Implementing Classes:
AbstractYarnScheduler, CapacityScheduler, FairScheduler, FifoScheduler
public interface YarnScheduler extends org.apache.hadoop.yarn.event.EventHandler<SchedulerEvent>
This interface is used by the components to talk to the scheduler for allocating of resources, cleaning up resources.

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    addQueue(Queue newQueue)
    Add to the scheduler a new Queue.
    Allocation
    allocate(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId, List<org.apache.hadoop.yarn.api.records.ResourceRequest> ask, List<org.apache.hadoop.yarn.api.records.SchedulingRequest> schedulingRequests, List<org.apache.hadoop.yarn.api.records.ContainerId> release, List<String> blacklistAdditions, List<String> blacklistRemovals, ContainerUpdates updateRequests)
    The main API between the ApplicationMaster and the Scheduler.
    boolean
    checkAccess(org.apache.hadoop.security.UserGroupInformation callerUGI, org.apache.hadoop.yarn.api.records.QueueACL acl, String queueName)
    Check if the user has permission to perform the operation.
    long
    checkAndGetApplicationLifetime(String queueName, long lifetime, RMAppImpl app)
    Verify whether a submitted application lifetime is valid as per configured Queue lifetime.
    org.apache.hadoop.yarn.api.records.Priority
    checkAndGetApplicationPriority(org.apache.hadoop.yarn.api.records.Priority priorityRequestedByApp, org.apache.hadoop.security.UserGroupInformation user, String queuePath, org.apache.hadoop.yarn.api.records.ApplicationId applicationId)
    Verify whether a submitted application priority is valid as per configured Queue
    org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport
    getAppResourceUsageReport(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)
    Get a resource usage report from a given app attempt ID.
    List<org.apache.hadoop.yarn.api.records.ApplicationAttemptId>
    getAppsInQueue(String queueName)
    Gets the apps under a given queue
    org.apache.hadoop.yarn.api.records.Resource
    getClusterResource()
    Get the whole resource capacity of the cluster.
    org.apache.hadoop.yarn.api.records.Priority
    getMaxClusterLevelAppPriority()
    Get cluster max priority.
    long
    getMaximumApplicationLifetime(String queueName)
    Get maximum lifetime for a queue.
    org.apache.hadoop.yarn.api.records.Resource
    getMaximumResourceCapability()
    Get maximum allocatable Resource at the cluster level.
    org.apache.hadoop.yarn.api.records.Resource
    getMaximumResourceCapability(String queueName)
    Get maximum allocatable Resource for the queue specified.
    org.apache.hadoop.yarn.api.records.Resource
    getMinimumResourceCapability()
    Get minimum allocatable Resource.
    SchedulerNodeReport
    getNodeReport(org.apache.hadoop.yarn.api.records.NodeId nodeId)
    Get node resource usage report.
    org.apache.hadoop.yarn.api.records.Resource
    getNormalizedResource(org.apache.hadoop.yarn.api.records.Resource requestedResource, org.apache.hadoop.yarn.api.records.Resource maxResourceCapability)
    Normalize a resource request using scheduler level maximum resource or queue based maximum resource.
    int
    getNumClusterNodes()
    Get the number of nodes available in the cluster.
    List<org.apache.hadoop.yarn.api.records.ResourceRequest>
    getPendingResourceRequestsForAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attemptId)
    Get pending resource request for specified application attempt.
    List<org.apache.hadoop.yarn.api.records.SchedulingRequest>
    getPendingSchedulingRequestsForAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attemptId)
    Get pending scheduling request for specified application attempt.
    Set<String>
    getPlanQueues()
    Gets the list of names for queues managed by the Reservation System.
    org.apache.hadoop.yarn.api.records.QueueInfo
    getQueueInfo(String queueName, boolean includeChildQueues, boolean recursive)
    Get queue information.
    List<org.apache.hadoop.yarn.api.records.QueueUserACLInfo>
    getQueueUserAclInfo()
    Get acls for queues for current user.
    org.apache.hadoop.yarn.util.resource.ResourceCalculator
    getResourceCalculator()
     
    RMContainer
    getRMContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)
    Get the container for the given containerId.
    QueueMetrics
    getRootQueueMetrics()
    Get the root queue for the scheduler.
    SchedulerAppReport
    getSchedulerAppInfo(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)
    Get the Scheduler app for a given app attempt Id.
    SchedulerNode
    getSchedulerNode(org.apache.hadoop.yarn.api.records.NodeId nodeId)
    Get SchedulerNode corresponds to nodeId.
    EnumSet<org.apache.hadoop.yarn.proto.YarnServiceProtos.SchedulerResourceTypes>
    getSchedulingResourceTypes()
    Return a collection of the resource types that are considered when scheduling
    List<org.apache.hadoop.yarn.api.records.Container>
    getTransferredContainers(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)
    Get previous attempts' live containers for work-preserving AM restart.
    void
    killAllAppsInQueue(String queueName)
    Terminate all applications in the specified queue.
    void
    moveAllApps(String sourceQueue, String destQueue)
    Completely drain sourceQueue of applications, by moving all of them to destQueue.
    String
    moveApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId, String newQueue)
    Moves the given application to the given queue.
    void
    preValidateMoveApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId, String newQueue)
     
    void
    removeQueue(String queueName)
    Remove an existing queue.
    void
    setClusterMaxPriority(org.apache.hadoop.conf.Configuration conf)
    Set the cluster max priority.
    void
    setEntitlement(String queue, QueueEntitlement entitlement)
    This method increase the entitlement for current queue (must respect invariants, e.g., no overcommit of parents, non negative, etc.).
    org.apache.hadoop.yarn.api.records.Priority
    updateApplicationPriority(org.apache.hadoop.yarn.api.records.Priority newPriority, org.apache.hadoop.yarn.api.records.ApplicationId applicationId, org.apache.hadoop.thirdparty.com.google.common.util.concurrent.SettableFuture<Object> future, org.apache.hadoop.security.UserGroupInformation user)
    Change application priority of a submitted application at runtime

    Methods inherited from interface org.apache.hadoop.yarn.event.EventHandler

    handle

  • Method Details

    • getQueueInfo

      @Public @Stable org.apache.hadoop.yarn.api.records.QueueInfo getQueueInfo(String queueName, boolean includeChildQueues, boolean recursive) throws IOException
      Get queue information.
      Parameters:
      queueName - queue name
      includeChildQueues - include child queues?
      recursive - get children queues?
      Returns:
      queue information
      Throws:
      IOException - an I/O exception has occurred.

    • getQueueUserAclInfo

      @Public @Stable List<org.apache.hadoop.yarn.api.records.QueueUserACLInfo> getQueueUserAclInfo()
      Get acls for queues for current user.
      Returns:
      acls for queues for current user

    • getClusterResource

      @LimitedPrivate("yarn") @Unstable org.apache.hadoop.yarn.api.records.Resource getClusterResource()
      Get the whole resource capacity of the cluster.
      Returns:
      the whole resource capacity of the cluster.

    • getMinimumResourceCapability

      @Public @Stable org.apache.hadoop.yarn.api.records.Resource getMinimumResourceCapability()
      Get minimum allocatable Resource.
      Returns:
      minimum allocatable resource

    • getMaximumResourceCapability

      @Public @Stable org.apache.hadoop.yarn.api.records.Resource getMaximumResourceCapability()
      Get maximum allocatable Resource at the cluster level.
      Returns:
      maximum allocatable resource

    • getMaximumResourceCapability

      @Public @Stable org.apache.hadoop.yarn.api.records.Resource getMaximumResourceCapability(String queueName)
      Get maximum allocatable Resource for the queue specified.
      Parameters:
      queueName - queue name
      Returns:
      maximum allocatable resource

    • getResourceCalculator

      @LimitedPrivate("yarn") @Evolving org.apache.hadoop.yarn.util.resource.ResourceCalculator getResourceCalculator()

    • getNumClusterNodes

      @Public @Stable int getNumClusterNodes()
      Get the number of nodes available in the cluster.
      Returns:
      the number of available nodes.

    • allocate

      @Public @Stable Allocation allocate(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId, List<org.apache.hadoop.yarn.api.records.ResourceRequest> ask, List<org.apache.hadoop.yarn.api.records.SchedulingRequest> schedulingRequests, List<org.apache.hadoop.yarn.api.records.ContainerId> release, List<String> blacklistAdditions, List<String> blacklistRemovals, ContainerUpdates updateRequests)
      The main API between the ApplicationMaster and the Scheduler. The ApplicationMaster may request/update container resources, number of containers, node/rack preference for allocations etc. to the Scheduler.
      Parameters:
      appAttemptId - the id of the application attempt.
      ask - the request made by an application to obtain various allocations like host/rack, resource, number of containers, relaxLocality etc., see ResourceRequest.
      schedulingRequests - similar to ask, but with added ability to specify allocation tags etc., see SchedulingRequest.
      release - the list of containers to be released.
      blacklistAdditions - places (node/rack) to be added to the blacklist.
      blacklistRemovals - places (node/rack) to be removed from the blacklist.
      updateRequests - container promotion/demotion updates.
      Returns:
      the Allocation for the application.

    • getNodeReport

      @LimitedPrivate("yarn") @Stable SchedulerNodeReport getNodeReport(org.apache.hadoop.yarn.api.records.NodeId nodeId)
      Get node resource usage report.
      Parameters:
      nodeId - nodeId.
      Returns:
      the SchedulerNodeReport for the node or null if nodeId does not point to a defined node.

    • getSchedulerAppInfo

      @LimitedPrivate("yarn") @Stable SchedulerAppReport getSchedulerAppInfo(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)
      Get the Scheduler app for a given app attempt Id.
      Parameters:
      appAttemptId - the id of the application attempt
      Returns:
      SchedulerApp for this given attempt.

    • getAppResourceUsageReport

      @LimitedPrivate("yarn") @Evolving org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport getAppResourceUsageReport(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)
      Get a resource usage report from a given app attempt ID.
      Parameters:
      appAttemptId - the id of the application attempt
      Returns:
      resource usage report for this given attempt

    • getRootQueueMetrics

      @LimitedPrivate("yarn") @Evolving QueueMetrics getRootQueueMetrics()
      Get the root queue for the scheduler.
      Returns:
      the root queue for the scheduler.

    • checkAccess

      boolean checkAccess(org.apache.hadoop.security.UserGroupInformation callerUGI, org.apache.hadoop.yarn.api.records.QueueACL acl, String queueName)
      Check if the user has permission to perform the operation. If the user has QueueACL.ADMINISTER_QUEUE permission, this user can view/modify the applications in this queue.
      Parameters:
      callerUGI - caller UserGroupInformation.
      acl - queue ACL.
      queueName - queue Name.
      Returns:
      true if the user has the permission, false otherwise

    • getAppsInQueue

      @LimitedPrivate("yarn") @Stable List<org.apache.hadoop.yarn.api.records.ApplicationAttemptId> getAppsInQueue(String queueName)
      Gets the apps under a given queue
      Parameters:
      queueName - the name of the queue.
      Returns:
      a collection of app attempt ids in the given queue.

    • getRMContainer

      @LimitedPrivate("yarn") @Unstable RMContainer getRMContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)
      Get the container for the given containerId.
      Parameters:
      containerId - the given containerId.
      Returns:
      the container for the given containerId.

    • moveApplication

      @LimitedPrivate("yarn") @Evolving String moveApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId, String newQueue) throws org.apache.hadoop.yarn.exceptions.YarnException
      Moves the given application to the given queue.
      Parameters:
      appId - application Id
      newQueue - the given queue.
      Returns:
      the name of the queue the application was placed into
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the move cannot be carried out

    • preValidateMoveApplication

      @LimitedPrivate("yarn") @Evolving void preValidateMoveApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId, String newQueue) throws org.apache.hadoop.yarn.exceptions.YarnException
      Parameters:
      appId - Application ID
      newQueue - Target QueueName
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the pre-validation for move cannot be carried out

    • moveAllApps

      void moveAllApps(String sourceQueue, String destQueue) throws org.apache.hadoop.yarn.exceptions.YarnException
      Completely drain sourceQueue of applications, by moving all of them to destQueue.
      Parameters:
      sourceQueue - sourceQueue.
      destQueue - destQueue.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

    • killAllAppsInQueue

      void killAllAppsInQueue(String queueName) throws org.apache.hadoop.yarn.exceptions.YarnException
      Terminate all applications in the specified queue.
      Parameters:
      queueName - the name of queue to be drained
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

    • removeQueue

      void removeQueue(String queueName) throws org.apache.hadoop.yarn.exceptions.YarnException
      Remove an existing queue. Implementations might limit when a queue could be removed (e.g., must have zero entitlement, and no applications running, or must be a leaf, etc..).
      Parameters:
      queueName - name of the queue to remove
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

    • addQueue

      void addQueue(Queue newQueue) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Add to the scheduler a new Queue. Implementations might limit what type of queues can be dynamically added (e.g., Queue must be a leaf, must be attached to existing parent, must have zero entitlement).
      Parameters:
      newQueue - the queue being added.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.
      IOException - when io exception occur.

    • setEntitlement

      void setEntitlement(String queue, QueueEntitlement entitlement) throws org.apache.hadoop.yarn.exceptions.YarnException
      This method increase the entitlement for current queue (must respect invariants, e.g., no overcommit of parents, non negative, etc.). Entitlement is a general term for weights in FairScheduler, capacity for the CapacityScheduler, etc.
      Parameters:
      queue - the queue for which we change entitlement
      entitlement - the new entitlement for the queue (capacity, maxCapacity, etc..)
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

    • getPlanQueues

      Set<String> getPlanQueues() throws org.apache.hadoop.yarn.exceptions.YarnException
      Gets the list of names for queues managed by the Reservation System.
      Returns:
      the list of queues which support reservations
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

    • getSchedulingResourceTypes

      EnumSet<org.apache.hadoop.yarn.proto.YarnServiceProtos.SchedulerResourceTypes> getSchedulingResourceTypes()
      Return a collection of the resource types that are considered when scheduling
      Returns:
      an EnumSet containing the resource types

    • checkAndGetApplicationPriority

      org.apache.hadoop.yarn.api.records.Priority checkAndGetApplicationPriority(org.apache.hadoop.yarn.api.records.Priority priorityRequestedByApp, org.apache.hadoop.security.UserGroupInformation user, String queuePath, org.apache.hadoop.yarn.api.records.ApplicationId applicationId) throws org.apache.hadoop.yarn.exceptions.YarnException
      Verify whether a submitted application priority is valid as per configured Queue
      Parameters:
      priorityRequestedByApp - Submitted Application priority.
      user - User who submitted the Application
      queuePath - Name of the Queue
      applicationId - Application ID
      Returns:
      Updated Priority from scheduler
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

    • updateApplicationPriority

      org.apache.hadoop.yarn.api.records.Priority updateApplicationPriority(org.apache.hadoop.yarn.api.records.Priority newPriority, org.apache.hadoop.yarn.api.records.ApplicationId applicationId, org.apache.hadoop.thirdparty.com.google.common.util.concurrent.SettableFuture<Object> future, org.apache.hadoop.security.UserGroupInformation user) throws org.apache.hadoop.yarn.exceptions.YarnException
      Change application priority of a submitted application at runtime
      Parameters:
      newPriority - Submitted Application priority.
      applicationId - Application ID
      future - Sets any type of exception happened from StateStore
      user - who submitted the application
      Returns:
      updated priority
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

    • getTransferredContainers

      List<org.apache.hadoop.yarn.api.records.Container> getTransferredContainers(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)
      Get previous attempts' live containers for work-preserving AM restart.
      Parameters:
      appAttemptId - the id of the application attempt
      Returns:
      list of live containers for the given attempt

    • setClusterMaxPriority

      void setClusterMaxPriority(org.apache.hadoop.conf.Configuration conf) throws org.apache.hadoop.yarn.exceptions.YarnException
      Set the cluster max priority.
      Parameters:
      conf - Configuration.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

    • getPendingResourceRequestsForAttempt

      List<org.apache.hadoop.yarn.api.records.ResourceRequest> getPendingResourceRequestsForAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attemptId)
      Get pending resource request for specified application attempt.
      Parameters:
      attemptId - the id of the application attempt
      Returns:
      pending resource requests.

    • getPendingSchedulingRequestsForAttempt

      List<org.apache.hadoop.yarn.api.records.SchedulingRequest> getPendingSchedulingRequestsForAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attemptId)
      Get pending scheduling request for specified application attempt.
      Parameters:
      attemptId - the id of the application attempt
      Returns:
      pending scheduling requests

    • getMaxClusterLevelAppPriority

      org.apache.hadoop.yarn.api.records.Priority getMaxClusterLevelAppPriority()
      Get cluster max priority.
      Returns:
      maximum priority of cluster

    • getSchedulerNode

      SchedulerNode getSchedulerNode(org.apache.hadoop.yarn.api.records.NodeId nodeId)
      Get SchedulerNode corresponds to nodeId.
      Parameters:
      nodeId - the node id of RMNode
      Returns:
      SchedulerNode corresponds to nodeId

    • getNormalizedResource

      org.apache.hadoop.yarn.api.records.Resource getNormalizedResource(org.apache.hadoop.yarn.api.records.Resource requestedResource, org.apache.hadoop.yarn.api.records.Resource maxResourceCapability)
      Normalize a resource request using scheduler level maximum resource or queue based maximum resource.
      Parameters:
      requestedResource - the resource to be normalized
      maxResourceCapability - Maximum container allocation value, if null or empty scheduler level maximum container allocation value will be used
      Returns:
      the normalized resource

    • checkAndGetApplicationLifetime

      @Public @Evolving long checkAndGetApplicationLifetime(String queueName, long lifetime, RMAppImpl app)
      Verify whether a submitted application lifetime is valid as per configured Queue lifetime.
      Parameters:
      queueName - Name of the Queue
      lifetime - configured application lifetime
      app - details of app
      Returns:
      valid lifetime as per queue

    • getMaximumApplicationLifetime

      @Public @Evolving long getMaximumApplicationLifetime(String queueName)
      Get maximum lifetime for a queue.
      Parameters:
      queueName - to get lifetime
      Returns:
      maximum lifetime in seconds