Package org.apache.hadoop.io.erasurecode.rawcoder

Class RawErasureDecoder

java.lang.Object
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder
Direct Known Subclasses:
DummyRawDecoder, NativeRSRawDecoder, NativeXORRawDecoder, RSLegacyRawDecoder, RSRawDecoder, XORRawDecoder
@Private public abstract class RawErasureDecoder extends Object
An abstract raw erasure decoder that's to be inherited by new decoders. Raw erasure coder is part of erasure codec framework, where erasure coder is used to encode/decode a group of blocks (BlockGroup) according to the codec specific BlockGroup layout and logic. An erasure coder extracts chunks of data from the blocks and can employ various low level raw erasure coders to perform encoding/decoding against the chunks. To distinguish from erasure coder, here raw erasure coder is used to mean the low level constructs, since it only takes care of the math calculation with a group of byte buffers. Note it mainly provides decode() calls, which should be stateless and may be made thread-safe in future.

  • Constructor Summary

    Constructors
    Constructor
    Description
    RawErasureDecoder(ErasureCoderOptions coderOptions)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    boolean
    allowChangeInputs()
    Allow change into input buffers or not while perform encoding/decoding.
    boolean
    allowVerboseDump()
    Allow to dump verbose info during encoding/decoding.
    void
    decode(byte[][] inputs, int[] erasedIndexes, byte[][] outputs)
    Decode with inputs and erasedIndexes, generates outputs.
    void
    decode(ByteBuffer[] inputs, int[] erasedIndexes, ByteBuffer[] outputs)
    Decode with inputs and erasedIndexes, generates outputs.
    void
    decode(ECChunk[] inputs, int[] erasedIndexes, ECChunk[] outputs)
    Decode with inputs and erasedIndexes, generates outputs.
    protected abstract void
    doDecode(org.apache.hadoop.io.erasurecode.rawcoder.ByteArrayDecodingState decodingState)
    Perform the real decoding using bytes array, supporting offsets and lengths.
    protected abstract void
    doDecode(org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState decodingState)
    Perform the real decoding using Direct ByteBuffer.
    protected int
    getNumAllUnits()
     
    int
    getNumDataUnits()
     
    int
    getNumParityUnits()
     
    boolean
    preferDirectBuffer()
    Tell if direct buffer is preferred or not.
    void
    release()
    Should be called when release this coder.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

  • Method Details

    • decode

      public void decode(ByteBuffer[] inputs, int[] erasedIndexes, ByteBuffer[] outputs) throws IOException
      Decode with inputs and erasedIndexes, generates outputs. How to prepare for inputs: 1. Create an array containing data units + parity units. Please note the data units should be first or before the parity units. 2. Set null in the array locations specified via erasedIndexes to indicate they're erased and no data are to read from; 3. Set null in the array locations for extra redundant items, as they're not necessary to read when decoding. For example in RS-6-3, if only 1 unit is really erased, then we have 2 extra items as redundant. They can be set as null to indicate no data will be used from them. For an example using RS (6, 3), assuming sources (d0, d1, d2, d3, d4, d5) and parities (p0, p1, p2), d2 being erased. We can and may want to use only 6 units like (d1, d3, d4, d5, p0, p2) to recover d2. We will have: inputs = [null(d0), d1, null(d2), d3, d4, d5, p0, null(p1), p2] erasedIndexes = [2] // index of d2 into inputs array outputs = [a-writable-buffer] Note, for both inputs and outputs, no mixing of on-heap buffers and direct buffers are allowed. If the coder option ALLOW_CHANGE_INPUTS is set true (false by default), the content of input buffers may change after the call, subject to concrete implementation.
      Parameters:
      inputs - input buffers to read data from. The buffers' remaining will be 0 after decoding
      erasedIndexes - indexes of erased units in the inputs array
      outputs - output buffers to put decoded data into according to erasedIndexes, ready for read after the call
      Throws:
      IOException - raised on errors performing I/O.

    • doDecode

      protected abstract void doDecode(org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState decodingState) throws IOException
      Perform the real decoding using Direct ByteBuffer.
      Parameters:
      decodingState - the decoding state
      Throws:
      IOException - raised on errors performing I/O.

    • decode

      public void decode(byte[][] inputs, int[] erasedIndexes, byte[][] outputs) throws IOException
      Decode with inputs and erasedIndexes, generates outputs. More see above.
      Parameters:
      inputs - input buffers to read data from
      erasedIndexes - indexes of erased units in the inputs array
      outputs - output buffers to put decoded data into according to erasedIndexes, ready for read after the call
      Throws:
      IOException - if the decoder is closed.

    • doDecode

      protected abstract void doDecode(org.apache.hadoop.io.erasurecode.rawcoder.ByteArrayDecodingState decodingState) throws IOException
      Perform the real decoding using bytes array, supporting offsets and lengths.
      Parameters:
      decodingState - the decoding state
      Throws:
      IOException - if the decoder is closed.

    • decode

      public void decode(ECChunk[] inputs, int[] erasedIndexes, ECChunk[] outputs) throws IOException
      Decode with inputs and erasedIndexes, generates outputs. More see above. Note, for both input and output ECChunks, no mixing of on-heap buffers and direct buffers are allowed.
      Parameters:
      inputs - input buffers to read data from
      erasedIndexes - indexes of erased units in the inputs array
      outputs - output buffers to put decoded data into according to erasedIndexes, ready for read after the call
      Throws:
      IOException - if the decoder is closed

    • getNumDataUnits

      public int getNumDataUnits()

    • getNumParityUnits

      public int getNumParityUnits()

    • getNumAllUnits

      protected int getNumAllUnits()

    • preferDirectBuffer

      public boolean preferDirectBuffer()
      Tell if direct buffer is preferred or not. It's for callers to decide how to allocate coding chunk buffers, using DirectByteBuffer or bytes array. It will return false by default.
      Returns:
      true if native buffer is preferred for performance consideration, otherwise false.

    • allowChangeInputs

      public boolean allowChangeInputs()
      Allow change into input buffers or not while perform encoding/decoding.
      Returns:
      true if it's allowed to change inputs, false otherwise

    • allowVerboseDump

      public boolean allowVerboseDump()
      Allow to dump verbose info during encoding/decoding.
      Returns:
      true if it's allowed to do verbose dump, false otherwise.

    • release

      public void release()
      Should be called when release this coder. Good chance to release encoding or decoding buffers