Package org.apache.hadoop.yarn.service.utils

Class CoreFileSystem

java.lang.Object
org.apache.hadoop.yarn.service.utils.CoreFileSystem
Direct Known Subclasses:
SliderFileSystem
public class CoreFileSystem extends Object

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected final org.apache.hadoop.conf.Configuration
    configuration
     
    protected final org.apache.hadoop.fs.FileSystem
    fileSystem
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    CoreFileSystem(org.apache.hadoop.conf.Configuration configuration)
     
    CoreFileSystem(org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.conf.Configuration configuration)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.hadoop.fs.Path
    buildClusterDirPath(String clustername)
    Build up the path string for a cluster instance -no attempt to create the directory is made
    org.apache.hadoop.fs.Path
    buildClusterUpgradeDirPath(String clusterName, String version)
    Build up the upgrade path string for a cluster.
    org.apache.hadoop.fs.Path
    buildKeytabInstallationDirPath(String keytabFolder)
    Build up the path string for keytab install location -no attempt to create the directory is made
    org.apache.hadoop.fs.Path
    buildKeytabPath(String keytabDir, String keytabName, String clusterName)
    Build up the path string for keytab install location -no attempt to create the directory is made
    org.apache.hadoop.fs.Path
    buildResourcePath(String resourceFolder)
    Build up the path string for resource install location -no attempt to create the directory is made
    org.apache.hadoop.fs.Path
    buildResourcePath(String dirName, String fileName)
    Build up the path string for resource install location -no attempt to create the directory is made
    String
    cat(org.apache.hadoop.fs.Path path)
     
    void
    copyHdfsFileToLocal(org.apache.hadoop.fs.Path hdfsPath, File destFile)
     
    void
    copyLocalFileToHdfs(File localPath, org.apache.hadoop.fs.Path destPath, org.apache.hadoop.fs.permission.FsPermission fp)
     
    org.apache.hadoop.yarn.api.records.LocalResource
    createAmResource(org.apache.hadoop.fs.Path destPath, org.apache.hadoop.yarn.api.records.LocalResourceType resourceType, org.apache.hadoop.yarn.api.records.LocalResourceVisibility visibility)
    Create an AM resource from the
    void
    createWithPermissions(org.apache.hadoop.fs.Path dir, org.apache.hadoop.fs.permission.FsPermission clusterPerms)
    Create a directory with the given permissions.
    void
    deleteClusterUpgradeDir(String clusterName, String version)
    Delete the upgrade cluster directory.
    org.apache.hadoop.fs.Path
    getBaseApplicationPath()
    Get the base path
    org.apache.hadoop.fs.Path
    getDependencyTarGzip()
    Get service dependency absolute filepath in HDFS used for application submission.
    org.apache.hadoop.fs.FileSystem
    getFileSystem()
    Returns the underlying FileSystem for this object.
    org.apache.hadoop.fs.Path
    getHomeDirectory()
     
    org.apache.hadoop.fs.Path
    getTempPathForCluster(String clustername)
    Get the temp path for this cluster
    boolean
    isFile(org.apache.hadoop.fs.Path path)
    Given a path, check if it exists and is a file
    String
    listFSDir(org.apache.hadoop.fs.Path path)
    list entries in a filesystem directory
    Map<String,org.apache.hadoop.yarn.api.records.LocalResource>
    submitDirectory(org.apache.hadoop.fs.Path srcDir, String destRelativeDir)
    Register all files under a fs path as a directory to push out
    org.apache.hadoop.yarn.api.records.LocalResource
    submitFile(File localFile, org.apache.hadoop.fs.Path tempPath, String subdir, String destFileName)
    Submit a local file to the filesystem references by the instance's cluster filesystem
    org.apache.hadoop.yarn.api.records.LocalResource
    submitJarWithClass(Class clazz, org.apache.hadoop.fs.Path tempPath, String subdir, String jarName)
    Submit a JAR containing a specific class, returning the resource to be mapped in
    void
    submitTarGzipAndUpdate(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> providerResources)
    Submit the AM tar.gz resource referenced by the instance's cluster filesystem.
    String
    toString()
     
    void
    verifyClusterDirectoryNonexistent(String clustername, org.apache.hadoop.fs.Path clusterDirectory)
    Verify that the cluster directory is not present
    void
    verifyDirectoryNonexistent(org.apache.hadoop.fs.Path clusterDirectory)
    Verify that the given directory is not present
    void
    verifyDirectoryWriteAccess(org.apache.hadoop.fs.Path dirPath)
    Verify that a user has write access to a directory.
    void
    verifyFileExists(org.apache.hadoop.fs.Path path)
    Verify that a path exists
    void
    verifyPathExists(org.apache.hadoop.fs.Path path)
    Verify that a path exists

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

  • Field Details

    • fileSystem

      protected final org.apache.hadoop.fs.FileSystem fileSystem

    • configuration

      protected final org.apache.hadoop.conf.Configuration configuration

  • Constructor Details

    • CoreFileSystem

      public CoreFileSystem(org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.conf.Configuration configuration)

    • CoreFileSystem

      public CoreFileSystem(org.apache.hadoop.conf.Configuration configuration) throws IOException
      Throws:
      IOException

  • Method Details

    • getTempPathForCluster

      public org.apache.hadoop.fs.Path getTempPathForCluster(String clustername)
      Get the temp path for this cluster
      Parameters:
      clustername - name of the cluster
      Returns:
      path for temp files (is not purged)

    • getFileSystem

      public org.apache.hadoop.fs.FileSystem getFileSystem()
      Returns the underlying FileSystem for this object.
      Returns:
      filesystem

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • buildClusterDirPath

      public org.apache.hadoop.fs.Path buildClusterDirPath(String clustername)
      Build up the path string for a cluster instance -no attempt to create the directory is made
      Parameters:
      clustername - name of the cluster
      Returns:
      the path for persistent data

    • buildClusterUpgradeDirPath

      public org.apache.hadoop.fs.Path buildClusterUpgradeDirPath(String clusterName, String version)
      Build up the upgrade path string for a cluster. No attempt to create the directory is made.
      Parameters:
      clusterName - name of the cluster
      version - version of the cluster
      Returns:
      the upgrade path to the cluster

    • deleteClusterUpgradeDir

      public void deleteClusterUpgradeDir(String clusterName, String version) throws IOException
      Delete the upgrade cluster directory.
      Parameters:
      clusterName - name of the cluster
      version - version of the cluster
      Throws:
      IOException

    • buildKeytabInstallationDirPath

      public org.apache.hadoop.fs.Path buildKeytabInstallationDirPath(String keytabFolder)
      Build up the path string for keytab install location -no attempt to create the directory is made
      Returns:
      the path for keytab

    • buildKeytabPath

      public org.apache.hadoop.fs.Path buildKeytabPath(String keytabDir, String keytabName, String clusterName)
      Build up the path string for keytab install location -no attempt to create the directory is made
      Returns:
      the path for keytab installation location

    • buildResourcePath

      public org.apache.hadoop.fs.Path buildResourcePath(String resourceFolder)
      Build up the path string for resource install location -no attempt to create the directory is made
      Returns:
      the path for resource

    • buildResourcePath

      public org.apache.hadoop.fs.Path buildResourcePath(String dirName, String fileName)
      Build up the path string for resource install location -no attempt to create the directory is made
      Returns:
      the path for resource

    • createWithPermissions

      public void createWithPermissions(org.apache.hadoop.fs.Path dir, org.apache.hadoop.fs.permission.FsPermission clusterPerms) throws IOException, BadClusterStateException
      Create a directory with the given permissions.
      Parameters:
      dir - directory
      clusterPerms - cluster permissions
      Throws:
      IOException - IO problem
      BadClusterStateException - any cluster state problem

    • verifyClusterDirectoryNonexistent

      public void verifyClusterDirectoryNonexistent(String clustername, org.apache.hadoop.fs.Path clusterDirectory) throws IOException, SliderException
      Verify that the cluster directory is not present
      Parameters:
      clustername - name of the cluster
      clusterDirectory - actual directory to look for
      Throws:
      IOException - trouble with FS
      SliderException - If the directory exists

    • verifyDirectoryNonexistent

      public void verifyDirectoryNonexistent(org.apache.hadoop.fs.Path clusterDirectory) throws IOException, SliderException
      Verify that the given directory is not present
      Parameters:
      clusterDirectory - actual directory to look for
      Throws:
      IOException - trouble with FS
      SliderException - If the directory exists

    • verifyDirectoryWriteAccess

      public void verifyDirectoryWriteAccess(org.apache.hadoop.fs.Path dirPath) throws IOException, SliderException
      Verify that a user has write access to a directory. It does this by creating then deleting a temp file
      Parameters:
      dirPath - actual directory to look for
      Throws:
      FileNotFoundException - file not found
      IOException - trouble with FS
      BadClusterStateException - if the directory is not writeable
      SliderException

    • verifyPathExists

      public void verifyPathExists(org.apache.hadoop.fs.Path path) throws IOException
      Verify that a path exists
      Parameters:
      path - path to check
      Throws:
      FileNotFoundException - file not found
      IOException - trouble with FS

    • verifyFileExists

      public void verifyFileExists(org.apache.hadoop.fs.Path path) throws IOException
      Verify that a path exists
      Parameters:
      path - path to check
      Throws:
      FileNotFoundException - file not found or is not a file
      IOException - trouble with FS

    • isFile

      public boolean isFile(org.apache.hadoop.fs.Path path)
      Given a path, check if it exists and is a file
      Parameters:
      path - absolute path to the file to check
      Returns:
      true if and only if path exists and is a file, false for all other reasons including if file check throws IOException

    • getBaseApplicationPath

      public org.apache.hadoop.fs.Path getBaseApplicationPath()
      Get the base path
      Returns:
      the base path optionally configured by YarnServiceConf.YARN_SERVICE_BASE_PATH

    • getDependencyTarGzip

      public org.apache.hadoop.fs.Path getDependencyTarGzip()
      Get service dependency absolute filepath in HDFS used for application submission.
      Returns:
      the absolute path to service dependency tarball in HDFS

    • getHomeDirectory

      public org.apache.hadoop.fs.Path getHomeDirectory()

    • createAmResource

      public org.apache.hadoop.yarn.api.records.LocalResource createAmResource(org.apache.hadoop.fs.Path destPath, org.apache.hadoop.yarn.api.records.LocalResourceType resourceType, org.apache.hadoop.yarn.api.records.LocalResourceVisibility visibility) throws IOException
      Create an AM resource from the
      Parameters:
      destPath - dest path in filesystem
      resourceType - resource type
      Returns:
      the local resource for AM
      Throws:
      IOException

    • submitDirectory

      public Map<String,org.apache.hadoop.yarn.api.records.LocalResource> submitDirectory(org.apache.hadoop.fs.Path srcDir, String destRelativeDir) throws IOException
      Register all files under a fs path as a directory to push out
      Parameters:
      srcDir - src dir
      destRelativeDir - dest dir (no trailing /)
      Returns:
      the map of entries
      Throws:
      IOException

    • submitJarWithClass

      public org.apache.hadoop.yarn.api.records.LocalResource submitJarWithClass(Class clazz, org.apache.hadoop.fs.Path tempPath, String subdir, String jarName) throws IOException, SliderException
      Submit a JAR containing a specific class, returning the resource to be mapped in
      Parameters:
      clazz - class to look for
      subdir - subdirectory (expected to end in a "/")
      jarName - At the destination
      Returns:
      the local resource ref
      Throws:
      IOException - trouble copying to HDFS
      SliderException

    • submitFile

      public org.apache.hadoop.yarn.api.records.LocalResource submitFile(File localFile, org.apache.hadoop.fs.Path tempPath, String subdir, String destFileName) throws IOException
      Submit a local file to the filesystem references by the instance's cluster filesystem
      Parameters:
      localFile - filename
      subdir - subdirectory (expected to end in a "/")
      destFileName - destination filename
      Returns:
      the local resource ref
      Throws:
      IOException - trouble copying to HDFS

    • submitTarGzipAndUpdate

      public void submitTarGzipAndUpdate(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> providerResources) throws IOException, BadClusterStateException
      Submit the AM tar.gz resource referenced by the instance's cluster filesystem. Also, update the providerResources object with the new resource.
      Parameters:
      providerResources - the provider resource map to be updated
      Throws:
      IOException - trouble copying to HDFS
      BadClusterStateException

    • copyLocalFileToHdfs

      public void copyLocalFileToHdfs(File localPath, org.apache.hadoop.fs.Path destPath, org.apache.hadoop.fs.permission.FsPermission fp) throws IOException
      Throws:
      IOException

    • copyHdfsFileToLocal

      public void copyHdfsFileToLocal(org.apache.hadoop.fs.Path hdfsPath, File destFile) throws IOException
      Throws:
      IOException

    • listFSDir

      public String listFSDir(org.apache.hadoop.fs.Path path) throws IOException
      list entries in a filesystem directory
      Parameters:
      path - directory
      Returns:
      a listing, one to a line
      Throws:
      IOException

    • cat

      public String cat(org.apache.hadoop.fs.Path path) throws IOException
      Throws:
      IOException