Package org.apache.hadoop.net

Class NetUtils

java.lang.Object
org.apache.hadoop.net.NetUtils
@LimitedPrivate({"HDFS","MapReduce"}) @Unstable public class NetUtils extends Object

  • Field Details

    • UNKNOWN_HOST

      public static final String UNKNOWN_HOST
      text included in wrapped exceptions if the host is null: "(unknown)"
      See Also:

    • HADOOP_WIKI

      public static final String HADOOP_WIKI
      Base URL of the Hadoop Wiki: "http://wiki.apache.org/hadoop/"
      See Also:

  • Constructor Details

    • NetUtils

      public NetUtils()

  • Method Details

    • getSocketFactory

      public static SocketFactory getSocketFactory(Configuration conf, Class<?> clazz)
      Get the socket factory for the given class according to its configuration parameter hadoop.rpc.socket.factory.class.<ClassName>. When no such parameter exists then fall back on the default socket factory as configured by hadoop.rpc.socket.factory.class.default. If this default socket factory is not configured, then fall back on the JVM default socket factory.
      Parameters:
      conf - the configuration
      clazz - the class (usually a VersionedProtocol)
      Returns:
      a socket factory

    • getDefaultSocketFactory

      public static SocketFactory getDefaultSocketFactory(Configuration conf)
      Get the default socket factory as specified by the configuration parameter hadoop.rpc.socket.factory.default
      Parameters:
      conf - the configuration
      Returns:
      the default socket factory as specified in the configuration or the JVM default socket factory if the configuration does not contain a default socket factory property.

    • getSocketFactoryFromProperty

      public static SocketFactory getSocketFactoryFromProperty(Configuration conf, String propValue)
      Get the socket factory corresponding to the given proxy URI. If the given proxy URI corresponds to an absence of configuration parameter, returns null. If the URI is malformed raises an exception.
      Parameters:
      conf - configuration.
      propValue - the property which is the class name of the SocketFactory to instantiate; assumed non null and non empty.
      Returns:
      a socket factory as defined in the property value.

    • createSocketAddr

      public static InetSocketAddress createSocketAddr(String target)
      Util method to build socket addr from either. <host>:<port> <fs>://<host>:<port>/<path>
      Parameters:
      target - target.
      Returns:
      socket addr.

    • createSocketAddrUnresolved

      public static InetSocketAddress createSocketAddrUnresolved(String target)

    • createSocketAddr

      public static InetSocketAddress createSocketAddr(String target, int defaultPort)
      Util method to build socket addr from either. <host> <host>:<port> <fs>://<host>:<port>/<path>
      Parameters:
      target - target.
      defaultPort - default port.
      Returns:
      socket addr.

    • createSocketAddr

      public static InetSocketAddress createSocketAddr(String target, int defaultPort, String configName)
      Create an InetSocketAddress from the given target string and default port. If the string cannot be parsed correctly, the configName parameter is used as part of the exception message, allowing the user to better diagnose the misconfiguration.
      Parameters:
      target - a string of either "host" or "host:port"
      defaultPort - the default port if target does not include a port number
      configName - the name of the configuration from which target was loaded. This is used in the exception message in the case that parsing fails.
      Returns:
      socket addr.

    • createSocketAddr

      public static InetSocketAddress createSocketAddr(String target, int defaultPort, String configName, boolean useCacheIfPresent)
      Create an InetSocketAddress from the given target string and default port. If the string cannot be parsed correctly, the configName parameter is used as part of the exception message, allowing the user to better diagnose the misconfiguration.
      Parameters:
      target - a string of either "host" or "host:port"
      defaultPort - the default port if target does not include a port number
      configName - the name of the configuration from which target was loaded. This is used in the exception message in the case that parsing fails.
      useCacheIfPresent - Whether use cache when create URI
      Returns:
      socket addr

    • createSocketAddr

      public static InetSocketAddress createSocketAddr(String target, int defaultPort, String configName, boolean useCacheIfPresent, boolean isResolved)

    • createSocketAddrForHost

      public static InetSocketAddress createSocketAddrForHost(String host, int port)
      Create a socket address with the given host and port. The hostname might be replaced with another host that was set via addStaticResolution(String, String). The value of hadoop.security.token.service.use_ip will determine whether the standard java host resolver is used, or if the fully qualified resolver is used.
      Parameters:
      host - the hostname or IP use to instantiate the object
      port - the port number
      Returns:
      InetSocketAddress

    • getCanonicalUri

      public static URI getCanonicalUri(URI uri, int defaultPort)
      Resolve the uri's hostname and add the default port if not in the uri
      Parameters:
      uri - to resolve
      defaultPort - if none is given
      Returns:
      URI

    • addStaticResolution

      public static void addStaticResolution(String host, String resolvedName)
      Adds a static resolution for host. This can be used for setting up hostnames with names that are fake to point to a well known host. For e.g. in some testcases we require to have daemons with different hostnames running on the same machine. In order to create connections to these daemons, one can set up mappings from those hostnames to "localhost". getStaticResolution(String) can be used to query for the actual hostname.
      Parameters:
      host - the hostname or IP use to instantiate the object.
      resolvedName - resolved name.

    • getStaticResolution

      public static String getStaticResolution(String host)
      Retrieves the resolved name for the passed host. The resolved name must have been set earlier using addStaticResolution(String, String)
      Parameters:
      host - the hostname or IP use to instantiate the object.
      Returns:
      the resolution

    • getAllStaticResolutions

      public static List<String[]> getAllStaticResolutions()
      This is used to get all the resolutions that were added using addStaticResolution(String, String). The return value is a List each element of which contains an array of String of the form String[0]=hostname, String[1]=resolved-hostname
      Returns:
      the list of resolutions

    • getConnectAddress

      public static InetSocketAddress getConnectAddress(Server server)
      Returns InetSocketAddress that a client can use to connect to the server. Server.getListenerAddress() is not correct when the server binds to "0.0.0.0". This returns "hostname:port" of the server, or "127.0.0.1:port" when the getListenerAddress() returns "0.0.0.0:port".
      Parameters:
      server - server.
      Returns:
      socket address that a client can use to connect to the server.

    • getConnectAddress

      public static InetSocketAddress getConnectAddress(InetSocketAddress addr)
      Returns an InetSocketAddress that a client can use to connect to the given listening address.
      Parameters:
      addr - of a listener
      Returns:
      socket address that a client can use to connect to the server.

    • getInputStream

      public static SocketInputWrapper getInputStream(Socket socket) throws IOException
      Same as getInputStream(socket, socket.getSoTimeout()).
      Parameters:
      socket - socket.
      Returns:
      SocketInputWrapper for reading from the socket.
      Throws:
      IOException - raised on errors performing I/O.
      See Also:

    • getInputStream

      public static SocketInputWrapper getInputStream(Socket socket, long timeout) throws IOException
      Return a SocketInputWrapper for the socket and set the given timeout. If the socket does not have an associated channel, then its socket timeout will be set to the specified value. Otherwise, a SocketInputStream will be created which reads with the configured timeout. Any socket created using socket factories returned by NetUtils(), must use this interface instead of Socket.getInputStream(). In general, this should be called only once on each socket: see the note in SocketInputWrapper.setTimeout(long) for more information.
      Parameters:
      socket - socket.
      timeout - timeout in milliseconds. zero for waiting as long as necessary.
      Returns:
      SocketInputWrapper for reading from the socket.
      Throws:
      IOException - raised on errors performing I/O.
      See Also:

    • getOutputStream

      public static OutputStream getOutputStream(Socket socket) throws IOException
      Same as getOutputStream(socket, 0). Timeout of zero implies write will wait until data is available.

      From documentation for getOutputStream(Socket, long) :
      Returns OutputStream for the socket. If the socket has an associated SocketChannel then it returns a SocketOutputStream with the given timeout. If the socket does not have a channel, Socket.getOutputStream() is returned. In the later case, the timeout argument is ignored and the write will wait until data is available.

      Any socket created using socket factories returned by NetUtils, must use this interface instead of Socket.getOutputStream().
      Parameters:
      socket - socket.
      Returns:
      OutputStream for writing to the socket.
      Throws:
      IOException - raised on errors performing I/O.
      See Also:

    • getOutputStream

      public static OutputStream getOutputStream(Socket socket, long timeout) throws IOException
      Returns OutputStream for the socket. If the socket has an associated SocketChannel then it returns a SocketOutputStream with the given timeout. If the socket does not have a channel, Socket.getOutputStream() is returned. In the later case, the timeout argument is ignored and the write will wait until data is available.

      Any socket created using socket factories returned by NetUtils, must use this interface instead of Socket.getOutputStream().
      Parameters:
      socket - socket.
      timeout - timeout in milliseconds. This may not always apply. zero for waiting as long as necessary.
      Returns:
      OutputStream for writing to the socket.
      Throws:
      IOException - raised on errors performing I/O.
      See Also:

    • connect

      public static void connect(Socket socket, SocketAddress address, int timeout) throws IOException
      This is a drop-in replacement for Socket.connect(SocketAddress, int). In the case of normal sockets that don't have associated channels, this just invokes socket.connect(endpoint, timeout). If socket.getChannel() returns a non-null channel, connect is implemented using Hadoop's selectors. This is done mainly to avoid Sun's connect implementation from creating thread-local selectors, since Hadoop does not have control on when these are closed and could end up taking all the available file descriptors.
      Parameters:
      socket - socket.
      address - the remote address
      timeout - timeout in milliseconds
      Throws:
      IOException - raised on errors performing I/O.
      See Also:

    • connect

      public static void connect(Socket socket, SocketAddress endpoint, SocketAddress localAddr, int timeout) throws IOException
      Like connect(Socket, SocketAddress, int) but also takes a local address and port to bind the socket to.
      Parameters:
      socket - socket.
      endpoint - the remote address
      localAddr - the local address to bind the socket to
      timeout - timeout in milliseconds
      Throws:
      IOException - raised on errors performing I/O.

    • normalizeHostName

      public static String normalizeHostName(String name)
      Given a string representation of a host, return its ip address in textual presentation.
      Parameters:
      name - a string representation of a host: either a textual representation its IP address or its host name
      Returns:
      its IP address in the string format

    • normalizeHostNames

      public static List<String> normalizeHostNames(Collection<String> names)
      Given a collection of string representation of hosts, return a list of corresponding IP addresses in the textual representation.
      Parameters:
      names - a collection of string representations of hosts
      Returns:
      a list of corresponding IP addresses in the string format
      See Also:

    • verifyHostnames

      public static void verifyHostnames(String[] names) throws UnknownHostException
      Performs a sanity check on the list of hostnames/IPs to verify they at least appear to be valid.
      Parameters:
      names - - List of hostnames/IPs
      Throws:
      UnknownHostException - Unknown Host Exception.

    • getHostNameOfIP

      public static String getHostNameOfIP(String ipPort)
      Attempt to obtain the host name of the given string which contains an IP address and an optional port.
      Parameters:
      ipPort - string of form ip[:port]
      Returns:
      Host name or null if the name can not be determined

    • normalizeIP2HostName

      public static String normalizeIP2HostName(String ipPort)
      Attempt to normalize the given string to "host:port" if it like "ip:port".
      Parameters:
      ipPort - maybe lik ip:port or host:port.
      Returns:
      host:port

    • getLocalHostname

      public static String getLocalHostname()
      Return hostname without throwing exception. The returned hostname String format is "hostname".
      Returns:
      hostname

    • getHostname

      public static String getHostname()
      Return hostname without throwing exception. The returned hostname String format is "hostname/ip address".
      Returns:
      hostname

    • getHostPortString

      public static String getHostPortString(InetSocketAddress addr)
      Compose a "host:port" string from the address.
      Parameters:
      addr - address.
      Returns:
      hort port string.

    • getPortFromHostPortString

      public static int getPortFromHostPortString(String addr) throws IllegalArgumentException
      Get port as integer from host port string like host:port.
      Parameters:
      addr - host + port string like host:port.
      Returns:
      an integer value representing the port.
      Throws:
      IllegalArgumentException - if the input is not in the correct format.

    • getLocalInetAddress

      public static InetAddress getLocalInetAddress(String host) throws SocketException
      Checks if host is a local host name and return InetAddress corresponding to that address.
      Parameters:
      host - the specified host
      Returns:
      a valid local InetAddress or null
      Throws:
      SocketException - if an I/O error occurs

    • isLocalAddress

      public static boolean isLocalAddress(InetAddress addr)
      Given an InetAddress, checks to see if the address is a local address, by comparing the address with all the interfaces on the node.
      Parameters:
      addr - address to check if it is local node's address
      Returns:
      true if the address corresponds to the local node

    • wrapException

      public static IOException wrapException(String destHost, int destPort, String localHost, int localPort, IOException exception)
      Take an IOException , the local host port and remote host port details and return an IOException with the input exception as the cause and also include the host details. The new exception provides the stack trace of the place where the exception is thrown and some extra diagnostics information. If the exception is of type BindException, ConnectException, UnknownHostException, SocketTimeoutException or has a String constructor, return a new one of the same type; Otherwise return an IOException.
      Parameters:
      destHost - target host (nullable)
      destPort - target port
      localHost - local host (nullable)
      localPort - local port
      exception - the caught exception.
      Returns:
      an exception to throw

    • addNodeNameToIOException

      public static IOException addNodeNameToIOException(IOException ioe, String nodeName)
      Return an @IOException of the same type as the input exception but with a modified exception message that includes the node name.
      Parameters:
      ioe - existing exception.
      nodeName - name of the node.
      Returns:
      IOException

    • isValidSubnet

      public static boolean isValidSubnet(String subnet)
      isValidSubnet.
      Parameters:
      subnet - subnet.
      Returns:
      true if the given string is a subnet specified using CIDR notation, false otherwise

    • getIPs

      public static List<InetAddress> getIPs(String subnet, boolean returnSubinterfaces)
      Return an InetAddress for each interface that matches the given subnet specified using CIDR notation.
      Parameters:
      subnet - subnet specified using CIDR notation
      returnSubinterfaces - whether to return IPs associated with subinterfaces
      Returns:
      ips.
      Throws:
      IllegalArgumentException - if subnet is invalid

    • getFreeSocketPort

      public static int getFreeSocketPort()
      Return a free port number. There is no guarantee it will remain free, so it should be used immediately.
      Returns:
      A free port for binding a local socket

    • getFreeSocketPorts

      public static Set<Integer> getFreeSocketPorts(int numOfPorts)
      Return free ports. There is no guarantee they will remain free, so ports should be used immediately. The number of free ports returned by this method should match argument numOfPorts. Num of ports provided in the argument should not exceed 25.
      Parameters:
      numOfPorts - Number of free ports to acquire.
      Returns:
      Free ports for binding a local socket.

    • bindToLocalAddress

      public static InetAddress bindToLocalAddress(InetAddress localAddr, boolean bindWildCardAddress)
      Return an @InetAddress to bind to. If bindWildCardAddress is true then returns null.
      Parameters:
      localAddr - local addr.
      bindWildCardAddress - bind wildcard address.
      Returns:
      InetAddress