Package org.apache.hadoop.hdfs.util

Class StripedBlockUtil

java.lang.Object
org.apache.hadoop.hdfs.util.StripedBlockUtil
@Private public class StripedBlockUtil extends Object
When accessing a file in striped layout, operations on logical byte ranges in the file need to be mapped to physical byte ranges on block files stored on DataNodes. This utility class facilities this mapping by defining and exposing a number of striping-related concepts. The most basic ones are illustrated in the following diagram. Unless otherwise specified, all range-related calculations are inclusive (the end offset of the previous range should be 1 byte lower than the start offset of the next one).

  • Field Details

    • LOG

      public static final org.slf4j.Logger LOG

  • Constructor Details

    • StripedBlockUtil

      public StripedBlockUtil()

  • Method Details

    • parseStripedBlockGroup

      public static LocatedBlock[] parseStripedBlockGroup(LocatedStripedBlock bg, int cellSize, int dataBlkNum, int parityBlkNum)
      This method parses a striped block group into individual blocks.
      Parameters:
      bg - The striped block group
      cellSize - The size of a striping cell
      dataBlkNum - The number of data blocks
      Returns:
      An array containing the blocks in the group

    • constructInternalBlock

      public static LocatedBlock constructInternalBlock(LocatedStripedBlock bg, int idxInReturnedLocs, int cellSize, int dataBlkNum, int idxInBlockGroup)
      This method creates an internal block at the given index of a block group.
      Parameters:
      idxInReturnedLocs - The index in the stored locations in the LocatedStripedBlock object
      idxInBlockGroup - The logical index in the striped block group
      Returns:
      The constructed internal block

    • constructInternalBlock

      public static ExtendedBlock constructInternalBlock(ExtendedBlock blockGroup, ErasureCodingPolicy ecPolicy, int idxInBlockGroup)

    • constructInternalBlock

      public static ExtendedBlock constructInternalBlock(ExtendedBlock blockGroup, int cellSize, int dataBlkNum, int idxInBlockGroup)
      This method creates an internal ExtendedBlock at the given index of a block group.

    • getInternalBlockLength

      public static long getInternalBlockLength(long dataSize, ErasureCodingPolicy ecPolicy, int idxInBlockGroup)

    • getInternalBlockLength

      public static long getInternalBlockLength(long dataSize, int cellSize, int numDataBlocks, int idxInBlockGroup)
      Get the size of an internal block at the given index of a block group.
      Parameters:
      dataSize - Size of the block group only counting data blocks
      cellSize - The size of a striping cell
      numDataBlocks - The number of data blocks
      idxInBlockGroup - The logical index in the striped block group
      Returns:
      The size of the internal block at the specified index

    • getSafeLength

      public static long getSafeLength(ErasureCodingPolicy ecPolicy, long[] blockLens)
      Compute the safe length given the internal block lengths.
      Parameters:
      ecPolicy - The EC policy used for the block group
      blockLens - The lengths of internal blocks
      Returns:
      The safe length

    • offsetInBlkToOffsetInBG

      public static long offsetInBlkToOffsetInBG(int cellSize, int dataBlkNum, long offsetInBlk, int idxInBlockGroup)
      Given a byte's offset in an internal block, calculate the offset in the block group.

    • getNextCompletedStripedRead

      public static StripedBlockUtil.StripingChunkReadResult getNextCompletedStripedRead(CompletionService<StripedBlockUtil.BlockReadStats> readService, Map<Future<StripedBlockUtil.BlockReadStats>,Integer> futures, long timeoutMillis) throws InterruptedException
      Get the next completed striped read task.
      Returns:
      StripedBlockUtil.StripingChunkReadResult indicating the status of the read task succeeded, and the block index of the task. If the method times out without getting any completed read tasks, -1 is returned as block index.
      Throws:
      InterruptedException

    • spaceConsumedByStripedBlock

      public static long spaceConsumedByStripedBlock(long numDataBlkBytes, int dataBlkNum, int parityBlkNum, int cellSize)
      Get the total usage of the striped blocks, which is the total of data blocks and parity blocks.
      Parameters:
      numDataBlkBytes - Size of the block group only counting data blocks
      dataBlkNum - The number of data blocks
      parityBlkNum - The number of parity blocks
      cellSize - The size of a striping cell
      Returns:
      The total usage of data blocks and parity blocks

    • divideOneStripe

      public static StripedBlockUtil.AlignedStripe[] divideOneStripe(ErasureCodingPolicy ecPolicy, int cellSize, LocatedStripedBlock blockGroup, long rangeStartInBlockGroup, long rangeEndInBlockGroup, ByteBuffer buf)
      Similar functionality with divideByteRangeIntoStripes(org.apache.hadoop.hdfs.protocol.ErasureCodingPolicy, int, org.apache.hadoop.hdfs.protocol.LocatedStripedBlock, long, long, java.nio.ByteBuffer), but is used by stateful read and uses ByteBuffer as reading target buffer. Besides the read range is within a single stripe thus the calculation logic is simpler.

    • divideByteRangeIntoStripes

      public static StripedBlockUtil.AlignedStripe[] divideByteRangeIntoStripes(ErasureCodingPolicy ecPolicy, int cellSize, LocatedStripedBlock blockGroup, long rangeStartInBlockGroup, long rangeEndInBlockGroup, ByteBuffer buf)
      This method divides a requested byte range into an array of inclusive StripedBlockUtil.AlignedStripe.
      Parameters:
      ecPolicy - The codec policy for the file, which carries the numbers of data / parity blocks
      cellSize - Cell size of stripe
      blockGroup - The striped block group
      rangeStartInBlockGroup - The byte range's start offset in block group
      rangeEndInBlockGroup - The byte range's end offset in block group
      buf - Destination buffer of the read operation for the byte range At most 5 stripes will be generated from each logical range, as demonstrated in the header of StripedBlockUtil.AlignedStripe.

    • checkBlocks

      public static void checkBlocks(ExtendedBlock blockGroup, int i, ExtendedBlock blocki) throws IOException
      Check if the information such as IDs and generation stamps in block-i match the block group.
      Throws:
      IOException

    • getBlockIndex

      public static int getBlockIndex(Block reportedBlock)