Package org.apache.hadoop.hdfs.protocol.datatransfer

Interface DataTransferProtocol

All Known Implementing Classes:
Sender
@Private @Evolving public interface DataTransferProtocol
Transfer data to/from datanode using a streaming protocol.

  • Field Details

    • LOG

      static final org.slf4j.Logger LOG

    • DATA_TRANSFER_VERSION

      static final int DATA_TRANSFER_VERSION
      Version for data transfers between clients and datanodes This should change when serialization of DatanodeInfo, not just when protocol changes. It is not very obvious.
      See Also:

  • Method Details

    • readBlock

      void readBlock(ExtendedBlock blk, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, String clientName, long blockOffset, long length, boolean sendChecksum, CachingStrategy cachingStrategy) throws IOException
      Read a block.
      Parameters:
      blk - the block being read.
      blockToken - security token for accessing the block.
      clientName - client's name.
      blockOffset - offset of the block.
      length - maximum number of bytes for this read.
      sendChecksum - if false, the DN should skip reading and sending checksums
      cachingStrategy - The caching strategy to use.
      Throws:
      IOException

    • writeBlock

      void writeBlock(ExtendedBlock blk, org.apache.hadoop.fs.StorageType storageType, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, String clientName, DatanodeInfo[] targets, org.apache.hadoop.fs.StorageType[] targetStorageTypes, DatanodeInfo source, BlockConstructionStage stage, int pipelineSize, long minBytesRcvd, long maxBytesRcvd, long latestGenerationStamp, org.apache.hadoop.util.DataChecksum requestedChecksum, CachingStrategy cachingStrategy, boolean allowLazyPersist, boolean pinning, boolean[] targetPinnings, String storageID, String[] targetStorageIDs) throws IOException
      Write a block to a datanode pipeline. The receiver datanode of this call is the next datanode in the pipeline. The other downstream datanodes are specified by the targets parameter. Note that the receiver DatanodeInfo is not required in the parameter list since the receiver datanode knows its info. However, the StorageType for storing the replica in the receiver datanode is a parameter since the receiver datanode may support multiple storage types.
      Parameters:
      blk - the block being written.
      storageType - for storing the replica in the receiver datanode.
      blockToken - security token for accessing the block.
      clientName - client's name.
      targets - other downstream datanodes in the pipeline.
      targetStorageTypes - target StorageTypes corresponding to the target datanodes.
      source - source datanode.
      stage - pipeline stage.
      pipelineSize - the size of the pipeline.
      minBytesRcvd - minimum number of bytes received.
      maxBytesRcvd - maximum number of bytes received.
      latestGenerationStamp - the latest generation stamp of the block.
      requestedChecksum - the requested checksum mechanism
      cachingStrategy - the caching strategy
      allowLazyPersist - hint to the DataNode that the block can be allocated on transient storage i.e. memory and written to disk lazily
      pinning - whether to pin the block, so Balancer won't move it.
      targetPinnings - whether to pin the block on target datanode
      storageID - optional StorageIDs designating where to write the block. An empty String or null indicates that this has not been provided.
      targetStorageIDs - target StorageIDs corresponding to the target datanodes.
      Throws:
      IOException

    • transferBlock

      void transferBlock(ExtendedBlock blk, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, String clientName, DatanodeInfo[] targets, org.apache.hadoop.fs.StorageType[] targetStorageTypes, String[] targetStorageIDs) throws IOException
      Transfer a block to another datanode. The block stage must be either BlockConstructionStage.TRANSFER_RBW or BlockConstructionStage.TRANSFER_FINALIZED.
      Parameters:
      blk - the block being transferred.
      blockToken - security token for accessing the block.
      clientName - client's name.
      targets - target datanodes.
      targetStorageIDs - StorageID designating where to write the block.
      Throws:
      IOException

    • requestShortCircuitFds

      void requestShortCircuitFds(ExtendedBlock blk, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, ShortCircuitShm.SlotId slotId, int maxVersion, boolean supportsReceiptVerification) throws IOException
      Request short circuit access file descriptors from a DataNode.
      Parameters:
      blk - The block to get file descriptors for.
      blockToken - Security token for accessing the block.
      slotId - The shared memory slot id to use, or null to use no slot id.
      maxVersion - Maximum version of the block data the client can understand.
      supportsReceiptVerification - True if the client supports receipt verification.
      Throws:
      IOException

    • releaseShortCircuitFds

      void releaseShortCircuitFds(ShortCircuitShm.SlotId slotId) throws IOException
      Release a pair of short-circuit FDs requested earlier.
      Parameters:
      slotId - SlotID used by the earlier file descriptors.
      Throws:
      IOException

    • requestShortCircuitShm

      void requestShortCircuitShm(String clientName) throws IOException
      Request a short circuit shared memory area from a DataNode.
      Parameters:
      clientName - The name of the client.
      Throws:
      IOException

    • replaceBlock

      void replaceBlock(ExtendedBlock blk, org.apache.hadoop.fs.StorageType storageType, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, String delHint, DatanodeInfo source, String storageId) throws IOException
      Receive a block from a source datanode and then notifies the namenode to remove the copy from the original datanode. Note that the source datanode and the original datanode can be different. It is used for balancing purpose.
      Parameters:
      blk - the block being replaced.
      storageType - the StorageType for storing the block.
      blockToken - security token for accessing the block.
      delHint - the hint for deleting the block in the original datanode.
      source - the source datanode for receiving the block.
      storageId - an optional storage ID to designate where the block is replaced to.
      Throws:
      IOException

    • copyBlock

      void copyBlock(ExtendedBlock blk, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken) throws IOException
      Copy a block. It is used for balancing purpose.
      Parameters:
      blk - the block being copied.
      blockToken - security token for accessing the block.
      Throws:
      IOException

    • blockChecksum

      void blockChecksum(ExtendedBlock blk, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, BlockChecksumOptions blockChecksumOptions) throws IOException
      Get block checksum (MD5 of CRC32).
      Parameters:
      blk - a block.
      blockToken - security token for accessing the block.
      blockChecksumOptions - determines how the block-level checksum is computed from underlying block metadata.
      Throws:
      IOException

    • blockGroupChecksum

      void blockGroupChecksum(StripedBlockInfo stripedBlockInfo, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, long requestedNumBytes, BlockChecksumOptions blockChecksumOptions) throws IOException
      Get striped block group checksum (MD5 of CRC32).
      Parameters:
      stripedBlockInfo - a striped block info.
      blockToken - security token for accessing the block.
      requestedNumBytes - requested number of bytes in the block group to compute the checksum.
      blockChecksumOptions - determines how the block-level checksum is computed from underlying block metadata.
      Throws:
      IOException