Package org.apache.hadoop.yarn.server.nodemanager.util

Class NodeManagerHardwareUtils

java.lang.Object
org.apache.hadoop.yarn.server.nodemanager.util.NodeManagerHardwareUtils
@Private @Unstable public class NodeManagerHardwareUtils extends Object
Helper class to determine hardware related characteristics such as the number of processors and the amount of memory on the node.

  • Constructor Summary

    Constructors
    Constructor
    Description
    NodeManagerHardwareUtils()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    static long
    getContainerMemoryMB(org.apache.hadoop.conf.Configuration conf)
    Function to return how much memory we should set aside for YARN containers.
    static long
    getContainerMemoryMB(org.apache.hadoop.yarn.util.ResourceCalculatorPlugin plugin, org.apache.hadoop.conf.Configuration conf)
    Function to return how much memory we should set aside for YARN containers.
    static float
    getContainersCPUs(org.apache.hadoop.conf.Configuration conf)
    Returns the fraction of CPUs that should be used for YARN containers.
    static float
    getContainersCPUs(org.apache.hadoop.yarn.util.ResourceCalculatorPlugin plugin, org.apache.hadoop.conf.Configuration conf)
    Returns the fraction of CPUs that should be used for YARN containers.
    static int
    getNodeCpuPercentage(org.apache.hadoop.conf.Configuration conf)
    Gets the percentage of physical CPU that is configured for YARN containers.
    static int
    getNodeCPUs(org.apache.hadoop.conf.Configuration conf)
    Returns the number of CPUs on the node.
    static int
    getNodeCPUs(org.apache.hadoop.yarn.util.ResourceCalculatorPlugin plugin, org.apache.hadoop.conf.Configuration conf)
    Returns the number of CPUs on the node.
    static org.apache.hadoop.yarn.api.records.Resource
    getNodeResources(org.apache.hadoop.conf.Configuration configuration)
    Get the resources for the node.
    static int
    getVCores(org.apache.hadoop.conf.Configuration conf)
    Function to return the number of vcores on the system that can be used for YARN containers.
    static int
    getVCores(org.apache.hadoop.yarn.util.ResourceCalculatorPlugin plugin, org.apache.hadoop.conf.Configuration conf)
    Function to return the number of vcores on the system that can be used for YARN containers.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • NodeManagerHardwareUtils

      public NodeManagerHardwareUtils()

  • Method Details

    • getNodeCPUs

      public static int getNodeCPUs(org.apache.hadoop.conf.Configuration conf)
      Returns the number of CPUs on the node. This value depends on the configuration setting which decides whether to count logical processors (such as hyperthreads) as cores or not.
      Parameters:
      conf - - Configuration object
      Returns:
      Number of CPUs

    • getNodeCPUs

      public static int getNodeCPUs(org.apache.hadoop.yarn.util.ResourceCalculatorPlugin plugin, org.apache.hadoop.conf.Configuration conf)
      Returns the number of CPUs on the node. This value depends on the configuration setting which decides whether to count logical processors (such as hyperthreads) as cores or not.
      Parameters:
      plugin - - ResourceCalculatorPlugin object to determine hardware specs
      conf - - Configuration object
      Returns:
      Number of CPU cores on the node.

    • getContainersCPUs

      public static float getContainersCPUs(org.apache.hadoop.conf.Configuration conf)
      Returns the fraction of CPUs that should be used for YARN containers. The number is derived based on various configuration params such as YarnConfiguration.NM_RESOURCE_PERCENTAGE_PHYSICAL_CPU_LIMIT
      Parameters:
      conf - - Configuration object
      Returns:
      Fraction of CPUs to be used for YARN containers

    • getContainersCPUs

      public static float getContainersCPUs(org.apache.hadoop.yarn.util.ResourceCalculatorPlugin plugin, org.apache.hadoop.conf.Configuration conf)
      Returns the fraction of CPUs that should be used for YARN containers. The number is derived based on various configuration params such as YarnConfiguration.NM_RESOURCE_PERCENTAGE_PHYSICAL_CPU_LIMIT
      Parameters:
      plugin - - ResourceCalculatorPlugin object to determine hardware specs
      conf - - Configuration object
      Returns:
      Fraction of CPUs to be used for YARN containers

    • getNodeCpuPercentage

      public static int getNodeCpuPercentage(org.apache.hadoop.conf.Configuration conf)
      Gets the percentage of physical CPU that is configured for YARN containers. This is percent > 0 and <= 100 based on YarnConfiguration.NM_RESOURCE_PERCENTAGE_PHYSICAL_CPU_LIMIT
      Parameters:
      conf - Configuration object
      Returns:
      percent > 0 and <= 100

    • getVCores

      public static int getVCores(org.apache.hadoop.conf.Configuration conf)
      Function to return the number of vcores on the system that can be used for YARN containers. If a number is specified in the configuration file, then that number is returned. If nothing is specified - 1. If the OS is an "unknown" OS(one for which we don't have ResourceCalculatorPlugin implemented), return the default NodeManager cores. 2. If the config variable yarn.nodemanager.cpu.use_logical_processors is set to true, it returns the logical processor count(count hyperthreads as cores), else it returns the physical cores count.
      Parameters:
      conf - - the configuration for the NodeManager
      Returns:
      the number of cores to be used for YARN containers

    • getVCores

      public static int getVCores(org.apache.hadoop.yarn.util.ResourceCalculatorPlugin plugin, org.apache.hadoop.conf.Configuration conf)
      Function to return the number of vcores on the system that can be used for YARN containers. If a number is specified in the configuration file, then that number is returned. If nothing is specified - 1. If the OS is an "unknown" OS(one for which we don't have ResourceCalculatorPlugin implemented), return the default NodeManager cores. 2. If the config variable yarn.nodemanager.cpu.use_logical_processors is set to true, it returns the logical processor count(count hyperthreads as cores), else it returns the physical cores count.
      Parameters:
      plugin - - ResourceCalculatorPlugin object to determine hardware specs
      conf - - the configuration for the NodeManager
      Returns:
      the number of cores to be used for YARN containers

    • getContainerMemoryMB

      public static long getContainerMemoryMB(org.apache.hadoop.conf.Configuration conf)
      Function to return how much memory we should set aside for YARN containers. If a number is specified in the configuration file, then that number is returned. If nothing is specified - 1. If the OS is an "unknown" OS(one for which we don't have ResourceCalculatorPlugin implemented), return the default NodeManager physical memory. 2. If the OS has a ResourceCalculatorPlugin implemented, the calculation is 0.8 * (RAM - 2 * JVM-memory) i.e. use 80% of the memory after accounting for memory used by the DataNode and the NodeManager. If the number is less than 1GB, log a warning message.
      Parameters:
      conf - - the configuration for the NodeManager
      Returns:
      the amount of memory that will be used for YARN containers in MB.

    • getContainerMemoryMB

      public static long getContainerMemoryMB(org.apache.hadoop.yarn.util.ResourceCalculatorPlugin plugin, org.apache.hadoop.conf.Configuration conf)
      Function to return how much memory we should set aside for YARN containers. If a number is specified in the configuration file, then that number is returned. If nothing is specified - 1. If the OS is an "unknown" OS(one for which we don't have ResourceCalculatorPlugin implemented), return the default NodeManager physical memory. 2. If the OS has a ResourceCalculatorPlugin implemented, the calculation is 0.8 * (RAM - 2 * JVM-memory) i.e. use 80% of the memory after accounting for memory used by the DataNode and the NodeManager. If the number is less than 1GB, log a warning message.
      Parameters:
      plugin - - ResourceCalculatorPlugin object to determine hardware specs
      conf - - the configuration for the NodeManager
      Returns:
      the amount of memory that will be used for YARN containers in MB.

    • getNodeResources

      public static org.apache.hadoop.yarn.api.records.Resource getNodeResources(org.apache.hadoop.conf.Configuration configuration)
      Get the resources for the node.
      Parameters:
      configuration - configuration file
      Returns:
      the resources for the node