Package org.apache.hadoop.yarn.service.utils

Class ServiceUtils

java.lang.Object
org.apache.hadoop.yarn.service.utils.ServiceUtils
public final class ServiceUtils extends Object
These are slider-specific Util methods

  • Method Details

    • isUnset

      public static boolean isUnset(String s)
      Implementation of set-ness, groovy definition of true/false for a string
      Parameters:
      s - string
      Returns:
      true iff the string is neither null nor empty

    • isSet

      public static boolean isSet(String s)

    • isEmpty

      public static boolean isEmpty(Collection l)

    • findContainingJarOrFail

      public static File findContainingJarOrFail(Class clazz) throws IOException
      Find a containing JAR
      Parameters:
      clazz - class to find
      Returns:
      the file
      Throws:
      IOException - any IO problem, including the class not having a classloader
      FileNotFoundException - if the class did not resolve to a file

    • findContainingJar

      public static File findContainingJar(Class my_class) throws IOException
      Find a containing JAR
      Parameters:
      my_class - class to find
      Returns:
      the file or null if it is not found
      Throws:
      IOException - any IO problem, including the class not having a classloader

    • copy

      public static void copy(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path srcFile, org.apache.hadoop.fs.Path destFile) throws IOException, BadClusterStateException
      Copy a file to a new FS -both paths must be qualified.
      Parameters:
      conf - conf file
      srcFile - src file
      destFile - dest file
      Throws:
      IOException
      BadClusterStateException

    • collectionToStringList

      public static List<String> collectionToStringList(Collection c)
      Take a collection, return a list containing the string value of every element in the collection.
      Parameters:
      c - collection
      Returns:
      a stringified list

    • join

      public static String join(Collection collection, String separator)
      Join an collection of objects with a separator that appears after every instance in the list -including at the end
      Parameters:
      collection - collection to call toString() on each element
      separator - separator string
      Returns:
      the joined entries

    • join

      public static String join(Collection collection, String separator, boolean trailing)
      Join an collection of objects with a separator that appears after every instance in the list -optionally at the end
      Parameters:
      collection - collection to call toString() on each element
      separator - separator string
      trailing - add a trailing entry or not
      Returns:
      the joined entries

    • join

      public static String join(String[] collection, String separator, boolean trailing)
      Join an array of strings with a separator that appears after every instance in the list -optionally at the end
      Parameters:
      collection - strings
      separator - separator string
      trailing - add a trailing entry or not
      Returns:
      the joined entries

    • mandatoryEnvVariable

      public static String mandatoryEnvVariable(String key) throws BadClusterStateException
      Resolve a mandatory environment variable
      Parameters:
      key - env var
      Returns:
      the resolved value
      Throws:
      BadClusterStateException

    • mergeMapsIgnoreDuplicateKeys

      public static <T1, T2> Map<T1,T2> mergeMapsIgnoreDuplicateKeys(Map<T1,T2> first, Map<T1,T2> second)
      Generic map merge logic
      Type Parameters:
      T1 - key type
      T2 - value type
      Parameters:
      first - first map
      second - second map
      Returns:
      'first' merged with the second

    • stringifyMap

      public static String stringifyMap(Map<String,String> map)
      Convert a map to a multi-line string for printing
      Parameters:
      map - map to stringify
      Returns:
      a string representation of the map

    • stringify

      public static String stringify(org.apache.hadoop.yarn.api.records.URL url)
      Convert a YARN URL into a string value of a normal URL
      Parameters:
      url - URL
      Returns:
      string representatin

    • getOpenPort

      public static int getOpenPort() throws IOException
      Get a random open port
      Returns:
      true if the port was available for listening on
      Throws:
      IOException

    • isPortAvailable

      public static boolean isPortAvailable(int port)
      See if a port is available for listening on by trying to listen on it and seeing if that works or fails.
      Parameters:
      port - port to listen to
      Returns:
      true if the port was available for listening on

    • buildEnvMap

      public static Map<String,String> buildEnvMap(Configuration conf, Map<String,String> tokenMap)

    • getLibDirs

      public static String[] getLibDirs()

    • putJar

      public static org.apache.hadoop.yarn.api.records.LocalResource putJar(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> providerResources, SliderFileSystem sliderFileSystem, Class clazz, org.apache.hadoop.fs.Path tempPath, String libdir, String jarName) throws IOException, SliderException
      Submit a JAR containing a specific class and map it
      Parameters:
      providerResources - provider map to build up
      sliderFileSystem - remote fs
      clazz - class to look for
      libdir - lib directory
      jarName - At the destination
      Returns:
      the local resource ref
      Throws:
      IOException - trouble copying to HDFS
      SliderException

    • putAllJars

      public static void putAllJars(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> providerResources, SliderFileSystem sliderFileSystem, org.apache.hadoop.fs.Path tempPath, String libDir, String srcPath) throws IOException, SliderException
      Submit a JAR containing and map it
      Parameters:
      providerResources - provider map to build up
      sliderFileSystem - remote fs
      libDir - lib directory
      srcPath - copy jars from
      Throws:
      IOException
      SliderException

    • createJarFilter

      public static FilenameFilter createJarFilter()
      Accept all filenames ending with .jar
      Returns:
      a filename filter

    • createLocalPath

      public static org.apache.hadoop.fs.Path createLocalPath(File file)
      Create a file:// path from a local file
      Parameters:
      file - file to point the path
      Returns:
      a new Path

    • buildClasspath

      public static ClasspathConstructor buildClasspath(String sliderConfDir, String libdir, SliderFileSystem sliderFileSystem, String configClassPath, boolean usingMiniMRCluster)
      Build up the classpath for execution -behaves very differently on a mini test cluster vs a production production one.
      Parameters:
      sliderConfDir - relative path to the dir containing slider config options to put on the classpath -or null
      libdir - directory containing the JAR files
      configClassPath - extra class path configured in yarn-site.xml
      usingMiniMRCluster - flag to indicate the MiniMR cluster is in use (and hence the current classpath should be used, not anything built up)
      Returns:
      a classpath

    • tarGzipFolder

      public static void tarGzipFolder(String[] libDirs, File tarGzipFile, FilenameFilter filter) throws IOException
      Given a source folder create a tar.gz file
      Parameters:
      libDirs -
      tarGzipFile -
      Throws:
      IOException

    • createNameTag

      public static String createNameTag(String name)

    • createVersionTag

      public static String createVersionTag(String version)

    • createDescriptionTag

      public static String createDescriptionTag(String description)

    • getLocalHostName

      public static String getLocalHostName(@Nullable org.apache.hadoop.conf.Configuration conf) throws UnknownHostException
      Throws:
      UnknownHostException