Package org.apache.hadoop.yarn.server.sharedcachemanager.store

Class SCMStore

java.lang.Object
org.apache.hadoop.service.AbstractService
org.apache.hadoop.service.CompositeService
org.apache.hadoop.yarn.server.sharedcachemanager.store.SCMStore
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.hadoop.service.Service
Direct Known Subclasses:
InMemorySCMStore
@Private @Evolving public abstract class SCMStore extends org.apache.hadoop.service.CompositeService
An abstract class for the data store used by the shared cache manager service. All implementations of methods in this interface need to be thread safe and atomic.

  • Nested Class Summary

    Nested classes/interfaces inherited from class org.apache.hadoop.service.CompositeService

    org.apache.hadoop.service.CompositeService.CompositeServiceShutdownHook

    Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

    org.apache.hadoop.service.Service.STATE

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected AppChecker
    appChecker
     

    Fields inherited from class org.apache.hadoop.service.CompositeService

    STOP_ONLY_STARTED_SERVICES

  • Constructor Summary

    Constructors
    Modifier
    Constructor
    Description
    protected
    SCMStore(String name)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    abstract String
    addResource(String key, String fileName)
    Add a resource to the shared cache and it's associated filename.
    abstract String
    addResourceReference(String key, SharedCacheResourceReference ref)
    Add a SharedCacheResourceReference to a resource and update the resource access time.
    void
    cleanResourceReferences(String key)
    Clean all resource references to a cache resource that contain application ids pointing to finished applications.
    static AppChecker
    createAppCheckerService(org.apache.hadoop.conf.Configuration conf)
    Create an instance of the AppChecker service via reflection based on the YarnConfiguration.SCM_APP_CHECKER_CLASS parameter.
    abstract Collection<SharedCacheResourceReference>
    getResourceReferences(String key)
    Get the SharedCacheResourceReference(s) associated with the resource.
    abstract boolean
    isResourceEvictable(String key, org.apache.hadoop.fs.FileStatus file)
    Check if a specific resource is evictable according to the store's enabled cache eviction policies.
    abstract boolean
    removeResource(String key)
    Remove a resource from the shared cache.
    abstract boolean
    removeResourceReference(String key, SharedCacheResourceReference ref, boolean updateAccessTime)
    Remove a SharedCacheResourceReference from a resource.
    abstract void
    removeResourceReferences(String key, Collection<SharedCacheResourceReference> refs, boolean updateAccessTime)
    Remove a collection of SharedCacheResourceReferences from a resource.
    protected void
    serviceInit(org.apache.hadoop.conf.Configuration conf)
     

    Methods inherited from class org.apache.hadoop.service.CompositeService

    addIfService, addService, getServices, removeService, serviceStart, serviceStop

    Methods inherited from class org.apache.hadoop.service.AbstractService

    close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

  • Field Details

  • Constructor Details

    • SCMStore

      protected SCMStore(String name)

  • Method Details

    • serviceInit

      protected void serviceInit(org.apache.hadoop.conf.Configuration conf) throws Exception
      Overrides:
      serviceInit in class org.apache.hadoop.service.CompositeService
      Throws:
      Exception

    • addResource

      @Private public abstract String addResource(String key, String fileName)
      Add a resource to the shared cache and it's associated filename. The resource is identified by a unique key. If the key already exists no action is taken and the filename of the existing resource is returned. If the key does not exist, the resource is added, it's access time is set, and the filename of the resource is returned.
      Parameters:
      key - a unique identifier for a resource
      fileName - the filename of the resource
      Returns:
      the filename of the resource as represented by the cache

    • removeResource

      @Private public abstract boolean removeResource(String key)
      Remove a resource from the shared cache.
      Parameters:
      key - a unique identifier for a resource
      Returns:
      true if the resource was removed or did not exist, false if the resource existed, contained at least one SharedCacheResourceReference and was not removed.

    • addResourceReference

      @Private public abstract String addResourceReference(String key, SharedCacheResourceReference ref)
      Add a SharedCacheResourceReference to a resource and update the resource access time.
      Parameters:
      key - a unique identifier for a resource
      ref - the SharedCacheResourceReference to add
      Returns:
      String the filename of the resource if the SharedCacheResourceReference was added or already existed. null if the resource did not exist

    • getResourceReferences

      @Private public abstract Collection<SharedCacheResourceReference> getResourceReferences(String key)
      Get the SharedCacheResourceReference(s) associated with the resource.
      Parameters:
      key - a unique identifier for a resource
      Returns:
      an unmodifiable collection of SharedCacheResourceReferences. If the resource does not exist, an empty set is returned.

    • removeResourceReference

      @Private public abstract boolean removeResourceReference(String key, SharedCacheResourceReference ref, boolean updateAccessTime)
      Remove a SharedCacheResourceReference from a resource.
      Parameters:
      key - a unique identifier for a resource
      ref - the SharedCacheResourceReference to remove
      updateAccessTime - true if the call should update the access time for the resource
      Returns:
      true if the reference was removed, false otherwise

    • removeResourceReferences

      @Private public abstract void removeResourceReferences(String key, Collection<SharedCacheResourceReference> refs, boolean updateAccessTime)
      Remove a collection of SharedCacheResourceReferences from a resource.
      Parameters:
      key - a unique identifier for a resource
      refs - the collection of SharedCacheResourceReferences to remove
      updateAccessTime - true if the call should update the access time for the resource

    • cleanResourceReferences

      @Private public void cleanResourceReferences(String key) throws org.apache.hadoop.yarn.exceptions.YarnException
      Clean all resource references to a cache resource that contain application ids pointing to finished applications. If the resource key does not exist, do nothing.
      Parameters:
      key - a unique identifier for a resource
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException

    • isResourceEvictable

      @Private public abstract boolean isResourceEvictable(String key, org.apache.hadoop.fs.FileStatus file)
      Check if a specific resource is evictable according to the store's enabled cache eviction policies.
      Parameters:
      key - a unique identifier for a resource
      file - the FileStatus object for the resource file in the file system.
      Returns:
      true if the resource is evicatble, false otherwise

    • createAppCheckerService

      @Private public static AppChecker createAppCheckerService(org.apache.hadoop.conf.Configuration conf)
      Create an instance of the AppChecker service via reflection based on the YarnConfiguration.SCM_APP_CHECKER_CLASS parameter.
      Parameters:
      conf -
      Returns:
      an instance of the AppChecker class