Package org.apache.hadoop.hdfs.client.impl

Class BlockReaderRemote

java.lang.Object
org.apache.hadoop.hdfs.client.impl.BlockReaderRemote
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.hadoop.fs.ByteBufferReadable, BlockReader
@Private public class BlockReaderRemote extends Object implements BlockReader
This is a wrapper around connection to datanode and understands checksum, offset etc. Terminology:
block
The hdfs block, typically large (~64MB).
chunk
A block is divided into chunks, each comes with a checksum. We want transfers to be chunk-aligned, to be able to verify checksums.
packet
A grouping of chunks used for transport. It contains a header, followed by checksum data, followed by real data.
Please see DataNode for the RPC specification. This is a new implementation introduced in Hadoop 0.23 which is more efficient and simpler than the older BlockReader implementation. It is renamed to BlockReaderRemote from BlockReaderRemote2.

  • Constructor Details

    • BlockReaderRemote

      protected BlockReaderRemote(String file, long blockId, org.apache.hadoop.util.DataChecksum checksum, boolean verifyChecksum, long startOffset, long firstChunkOffset, long bytesToRead, Peer peer, DatanodeID datanodeID, PeerCache peerCache, int networkDistance)

  • Method Details

    • getPeer

      @VisibleForTesting public Peer getPeer()

    • read

      public int read(byte[] buf, int off, int len) throws IOException
      Specified by:
      read in interface BlockReader
      Throws:
      IOException

    • read

      public int read(ByteBuffer buf) throws IOException
      Specified by:
      read in interface org.apache.hadoop.fs.ByteBufferReadable
      Throws:
      IOException

    • skip

      public long skip(long n) throws IOException
      Description copied from interface: BlockReader
      Skip the given number of bytes
      Specified by:
      skip in interface BlockReader
      Throws:
      IOException

    • close

      public void close() throws IOException
      Description copied from interface: BlockReader
      Close the block reader.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface BlockReader
      Specified by:
      close in interface Closeable
      Throws:
      IOException

    • getFileName

      public static String getFileName(InetSocketAddress s, String poolId, long blockId)
      File name to print when accessing a block directly (from servlets)
      Parameters:
      s - Address of the block location
      poolId - Block pool ID of the block
      blockId - Block ID of the block
      Returns:
      string that has a file name for debug purposes

    • readAll

      public int readAll(byte[] buf, int offset, int len) throws IOException
      Description copied from interface: BlockReader
      Similar to BlockReader.readFully(byte[], int, int) except that it will not throw an exception on EOF. However, it differs from the simple BlockReader.read(byte[], int, int) call in that it is guaranteed to read the data if it is available. In other words, if this call does not throw an exception, then either the buffer has been filled or the next call will return EOF.
      Specified by:
      readAll in interface BlockReader
      Throws:
      IOException

    • readFully

      public void readFully(byte[] buf, int off, int len) throws IOException
      Description copied from interface: BlockReader
      Read exactly the given amount of data, throwing an exception if EOF is reached before that amount
      Specified by:
      readFully in interface BlockReader
      Throws:
      IOException

    • newBlockReader

      public static BlockReader newBlockReader(String file, ExtendedBlock block, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, long startOffset, long len, boolean verifyChecksum, String clientName, Peer peer, DatanodeID datanodeID, PeerCache peerCache, CachingStrategy cachingStrategy, int networkDistance, org.apache.hadoop.conf.Configuration configuration) throws IOException
      Create a new BlockReader specifically to satisfy a read. This method also sends the OP_READ_BLOCK request.
      Parameters:
      file - File location.
      block - The block object.
      blockToken - The block token for security.
      startOffset - The read offset, relative to block head.
      len - The number of bytes to read.
      verifyChecksum - Whether to verify checksum.
      clientName - Client name.
      peer - The Peer to use.
      datanodeID - The DatanodeID this peer is connected to.
      peerCache - Caches TCP and UNIX domain sockets for reuse.
      cachingStrategy - Caching strategy to use when reading the block.
      networkDistance - Return the distance between two nodes by comparing their network paths.
      configuration - Configuration of client.
      Returns:
      New BlockReader instance, or null on error.
      Throws:
      IOException

    • available

      public int available()
      Description copied from interface: BlockReader
      Returns an estimate of the number of bytes that can be read (or skipped over) from this input stream without performing network I/O. This may return more than what is actually present in the block.
      Specified by:
      available in interface BlockReader

    • isShortCircuit

      public boolean isShortCircuit()
      Specified by:
      isShortCircuit in interface BlockReader
      Returns:
      true only if this is a short-circuit read. All short-circuit reads are also local.

    • getClientMmap

      public ClientMmap getClientMmap(EnumSet<org.apache.hadoop.fs.ReadOption> opts)
      Description copied from interface: BlockReader
      Get a ClientMmap object for this BlockReader.
      Specified by:
      getClientMmap in interface BlockReader
      Parameters:
      opts - The read options to use.
      Returns:
      The ClientMmap object, or null if mmap is not supported.

    • getDataChecksum

      public org.apache.hadoop.util.DataChecksum getDataChecksum()
      Specified by:
      getDataChecksum in interface BlockReader
      Returns:
      The DataChecksum used by the read block

    • getNetworkDistance

      public int getNetworkDistance()
      Description copied from interface: BlockReader
      Return the network distance between local machine and the remote machine.
      Specified by:
      getNetworkDistance in interface BlockReader