Package org.apache.hadoop.fs.shell

Class CopyCommandWithMultiThread

java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.hadoop.fs.shell.Command
org.apache.hadoop.fs.shell.FsCommand
org.apache.hadoop.fs.shell.CopyCommandWithMultiThread
All Implemented Interfaces:
Configurable
public abstract class CopyCommandWithMultiThread extends FsCommand
Abstract command to enable sub copy commands run with multi-thread.

  • Field Details

    • DEFAULT_QUEUE_SIZE

      public static final int DEFAULT_QUEUE_SIZE
      See Also:

    • LOG

      protected static final org.slf4j.Logger LOG

    • dst

      protected PathData dst

  • Constructor Details

    • CopyCommandWithMultiThread

      public CopyCommandWithMultiThread()

  • Method Details

    • setThreadCount

      protected void setThreadCount(String optValue)
      set thread count by option value, if the value less than 1, use 1 instead.
      Parameters:
      optValue - option value

    • setThreadPoolQueueSize

      protected void setThreadPoolQueueSize(String optValue)
      set thread pool queue size by option value, if the value less than 1, use DEFAULT_QUEUE_SIZE instead.
      Parameters:
      optValue - option value

    • getThreadCount

      @VisibleForTesting protected int getThreadCount()

    • getThreadPoolQueueSize

      @VisibleForTesting protected int getThreadPoolQueueSize()

    • getExecutor

      @VisibleForTesting protected ThreadPoolExecutor getExecutor()

    • processArguments

      protected void processArguments(LinkedList<PathData> args) throws IOException
      Description copied from class: Command
      Processes the command's list of expanded arguments. Command.processArgument(PathData) will be invoked with each item in the list. The loop catches IOExceptions, increments the error count, and displays the exception.
      Parameters:
      args - a list of PathData to process
      Throws:
      IOException - if anything goes wrong...

    • isMultiThreadNecessary

      @VisibleForTesting protected boolean isMultiThreadNecessary(LinkedList<PathData> args) throws IOException
      Throws:
      IOException

    • copyFileToTarget

      protected void copyFileToTarget(PathData src, PathData target) throws IOException
      Copies the source file to the target.
      Parameters:
      src - item to copy
      target - where to copy the item
      Throws:
      IOException - if copy fails

    • setOverwrite

      protected void setOverwrite(boolean flag)
      This method is used to enable the force(-f) option while copying the files.
      Parameters:
      flag - true/false

    • setLazyPersist

      protected void setLazyPersist(boolean flag)

    • setVerifyChecksum

      protected void setVerifyChecksum(boolean flag)

    • setWriteChecksum

      protected void setWriteChecksum(boolean flag)

    • setDirectWrite

      protected void setDirectWrite(boolean flag)

    • setPreserve

      protected void setPreserve(boolean preserve)
      If true, the last modified time, last access time, owner, group and permission information of the source file will be preserved as far as target FileSystem implementation allows.
      Parameters:
      preserve - preserve.

    • preserve

      protected void preserve(org.apache.hadoop.fs.shell.CommandWithDestination.FileAttribute fileAttribute)
      Add file attributes that need to be preserved. This method may be called multiple times to add attributes.
      Parameters:
      fileAttribute - - Attribute to add, one at a time

    • getLocalDestination

      protected void getLocalDestination(LinkedList<String> args) throws IOException
      The last arg is expected to be a local path, if only one argument is given then the destination will be the current directory
      Parameters:
      args - is the list of arguments
      Throws:
      IOException - raised on errors performing I/O.

    • getRemoteDestination

      protected void getRemoteDestination(LinkedList<String> args) throws IOException
      The last arg is expected to be a remote path, if only one argument is given then the destination will be the remote user's directory
      Parameters:
      args - is the list of arguments
      Throws:
      PathIOException - if path doesn't exist or matches too many times
      IOException

    • processPathArgument

      protected void processPathArgument(PathData src) throws IOException
      Description copied from class: Command
      This is the last chance to modify an argument before going into the (possibly) recursive Command.processPaths(PathData, PathData...) -> Command.processPath(PathData) loop. Ex. ls and du use this to expand out directories.
      Overrides:
      processPathArgument in class Command
      Parameters:
      src - a PathData representing a path which exists
      Throws:
      IOException - if anything goes wrong...

    • processPath

      protected void processPath(PathData src) throws IOException
      Description copied from class: Command
      Hook for commands to implement an operation to be applied on each path for the command. Note implementation of this method is optional if earlier methods in the chain handle the operation.
      Overrides:
      processPath in class Command
      Parameters:
      src - a PathData object
      Throws:
      IOException - if anything else goes wrong in the user-implementation

    • processPath

      protected void processPath(PathData src, PathData dst) throws IOException
      Called with a source and target destination pair
      Parameters:
      src - for the operation
      dst - for the operation
      Throws:
      IOException - if anything goes wrong

    • recursePath

      protected void recursePath(PathData src) throws IOException
      Description copied from class: Command
      Gets the directory listing for a path and invokes Command.processPaths(PathData, PathData...)
      Overrides:
      recursePath in class Command
      Parameters:
      src - PathData for directory to recurse into
      Throws:
      IOException - if anything goes wrong...

    • getTargetPath

      protected PathData getTargetPath(PathData src) throws IOException
      Throws:
      IOException

    • copyStreamToTarget

      protected void copyStreamToTarget(InputStream in, PathData target) throws IOException
      If direct write is disabled ,copies the stream contents to a temporary file "target._COPYING_". If the copy is successful, the temporary file will be renamed to the real path, else the temporary file will be deleted. if direct write is enabled , then creation temporary file is skipped.
      Parameters:
      in - the input stream for the copy
      target - where to store the contents of the stream
      Throws:
      IOException - if copy fails

    • preserveAttributes

      protected void preserveAttributes(PathData src, PathData target, boolean preserveRawXAttrs) throws IOException
      Preserve the attributes of the source to the target. The method calls shouldPreserve(FileAttribute) to check what attribute to preserve.
      Parameters:
      src - source to preserve
      target - where to preserve attributes
      preserveRawXAttrs - true if raw.* xattrs should be preserved
      Throws:
      IOException - if fails to preserve attributes