Package org.apache.hadoop.util

Class GenericOptionsParser

java.lang.Object
org.apache.hadoop.util.GenericOptionsParser
Direct Known Subclasses:
ServiceLauncher.MinimalGenericOptionsParser
@Private @Evolving public class GenericOptionsParser extends Object
GenericOptionsParser is a utility to parse command line arguments generic to the Hadoop framework. GenericOptionsParser recognizes several standard command line arguments, enabling applications to easily specify a namenode, a ResourceManager, additional configuration resources etc.

Generic Options

The supported generic options are:

     -conf <configuration file>     specify a configuration file
     -D <property=value>            use value for given property
     -fs <local|namenode:port>      specify a namenode
     -jt <local|resourcemanager:port>    specify a ResourceManager
     -files <comma separated list of files>    specify comma separated
                            files to be copied to the map reduce cluster
     -libjars <comma separated list of jars>   specify comma separated
                            jar files to include in the classpath.
     -archives <comma separated list of archives>    specify comma
             separated archives to be unarchived on the compute machines.

The general command line syntax is:

 
 bin/hadoop command [genericOptions] [commandOptions]

Generic command line arguments might modify Configuration objects, given to constructors.

The functionality is implemented using Commons CLI.

Examples:

 $ bin/hadoop dfs -fs darwin:8020 -ls /data
 list /data directory in dfs with namenode darwin:8020
 
 $ bin/hadoop dfs -D fs.default.name=darwin:8020 -ls /data
 list /data directory in dfs with namenode darwin:8020
     
 $ bin/hadoop dfs -conf core-site.xml -conf hdfs-site.xml -ls /data
 list /data directory in dfs with multiple conf files specified.

 $ bin/hadoop job -D yarn.resourcemanager.address=darwin:8032 -submit job.xml
 submit a job to ResourceManager darwin:8032

 $ bin/hadoop job -jt darwin:8032 -submit job.xml
 submit a job to ResourceManager darwin:8032

 $ bin/hadoop job -jt local -submit job.xml
 submit a job to local runner
 
 $ bin/hadoop jar -libjars testlib.jar 
 -archives test.tgz -files file.txt inputjar args
 job submission with libjars, files and archives
See Also:

  • Constructor Details

    • GenericOptionsParser

      public GenericOptionsParser(org.apache.commons.cli.Options opts, String[] args) throws IOException
      Create an options parser with the given options to parse the args.
      Parameters:
      opts - the options
      args - the command line arguments
      Throws:
      IOException - raised on errors performing I/O.

    • GenericOptionsParser

      public GenericOptionsParser(String[] args) throws IOException
      Create an options parser to parse the args.
      Parameters:
      args - the command line arguments
      Throws:
      IOException - raised on errors performing I/O.

    • GenericOptionsParser

      public GenericOptionsParser(Configuration conf, String[] args) throws IOException
      Create a GenericOptionsParser to parse only the generic Hadoop arguments. The array of string arguments other than the generic arguments can be obtained by getRemainingArgs().
      Parameters:
      conf - the Configuration to modify.
      args - command-line arguments.
      Throws:
      IOException - raised on errors performing I/O.

    • GenericOptionsParser

      public GenericOptionsParser(Configuration conf, org.apache.commons.cli.Options options, String[] args) throws IOException
      Create a GenericOptionsParser to parse given options as well as generic Hadoop options. The resulting CommandLine object can be obtained by getCommandLine().
      Parameters:
      conf - the configuration to modify
      options - options built by the caller
      args - User-specified arguments
      Throws:
      IOException - raised on errors performing I/O.

  • Method Details

    • getRemainingArgs

      public String[] getRemainingArgs()
      Returns an array of Strings containing only application-specific arguments.
      Returns:
      array of Strings containing the un-parsed arguments or empty array if commandLine was not defined.

    • getConfiguration

      public Configuration getConfiguration()
      Get the modified configuration
      Returns:
      the configuration that has the modified parameters.

    • getCommandLine

      public org.apache.commons.cli.CommandLine getCommandLine()
      Returns the commons-cli CommandLine object to process the parsed arguments. Note: If the object is created with GenericOptionsParser(Configuration, String[]), then returned object will only contain parsed generic options.
      Returns:
      CommandLine representing list of arguments parsed against Options descriptor.

    • isParseSuccessful

      public boolean isParseSuccessful()
      Query for the parse operation succeeding.
      Returns:
      true if parsing the CLI was successful

    • buildGeneralOptions

      protected org.apache.commons.cli.Options buildGeneralOptions(org.apache.commons.cli.Options opts)
      Parameters:
      opts - input opts.
      Returns:
      Specify properties of each generic option. Important: as Option is not thread safe, subclasses must synchronize use on Option.class

    • getLibJars

      public static URL[] getLibJars(Configuration conf) throws IOException
      If libjars are set in the conf, parse the libjars.
      Parameters:
      conf - input Configuration.
      Returns:
      libjar urls
      Throws:
      IOException - raised on errors performing I/O.

    • printGenericCommandUsage

      public static void printGenericCommandUsage(PrintStream out)
      Print the usage message for generic command-line options supported.
      Parameters:
      out - stream to print the usage message to.