Package org.apache.hadoop.hdfs

Class DFSStripedInputStream

java.lang.Object

java.io.InputStream

org.apache.hadoop.fs.FSInputStream

org.apache.hadoop.hdfs.DFSInputStream

org.apache.hadoop.hdfs.DFSStripedInputStream

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.fs.ByteBufferPositionedReadable, org.apache.hadoop.fs.ByteBufferReadable, org.apache.hadoop.fs.CanSetDropBehind, org.apache.hadoop.fs.CanSetReadahead, org.apache.hadoop.fs.CanUnbuffer, org.apache.hadoop.fs.HasEnhancedByteBufferAccess, org.apache.hadoop.fs.PositionedReadable, org.apache.hadoop.fs.Seekable, org.apache.hadoop.fs.StreamCapabilities

@Private
public class DFSStripedInputStream
extends DFSInputStream

DFSStripedInputStream reads from striped block groups.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities

org.apache.hadoop.fs.StreamCapabilities.StreamCapability

Field Summary

Fields

Modifier and Type	Field	Description
protected ByteBuffer	parityBuf

Fields inherited from class org.apache.hadoop.hdfs.DFSInputStream

blockEnd, cachingStrategy, closed, currentLocatedBlock, dfsClient, failures, infoLock, locatedBlocks, pos, readStatistics, src, tcpReadsDisabledForTesting, verifyChecksum

Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities

ABORTABLE_STREAM, DROPBEHIND, HFLUSH, HSYNC, IOSTATISTICS, IOSTATISTICS_CONTEXT, PREADBYTEBUFFER, READAHEAD, READBYTEBUFFER, UNBUFFER, VECTOREDIO, VECTOREDIO_BUFFERS_SLICED

Method Summary

Modifier and Type	Method	Description
void	close()	Close it down!
protected void	closeCurrentBlockReaders()	Extend the super method with the logic of switching between cells.
protected void	closeReader(org.apache.hadoop.hdfs.StripeReader.BlockReaderInfo readerInfo)
protected void	fetchBlockByteRange(LocatedBlock block, long start, long end, ByteBuffer buf, DFSUtilClient.CorruptedBlocks corruptedBlocks, Map<InetSocketAddress,List<IOException>> exceptionMap)	Real implementation of pread.
protected org.apache.hadoop.io.ByteBufferPool	getBufferPool()
protected ByteBuffer	getCurStripeBuf()
protected ByteBuffer	getParityBuffer()
protected ThreadPoolExecutor	getStripedReadsThreadPool()
ByteBuffer	read(org.apache.hadoop.io.ByteBufferPool bufferPool, int maxLength, EnumSet<org.apache.hadoop.fs.ReadOption> opts)	May need online read recovery, zero-copy read doesn't make sense, so don't support it.
protected int	readWithStrategy(org.apache.hadoop.hdfs.ReaderStrategy strategy)
protected LocatedBlock	refreshLocatedBlock(LocatedBlock block)	The super method DFSInputStream.refreshLocatedBlock(org.apache.hadoop.hdfs.protocol.LocatedBlock) refreshes cached LocatedBlock by executing DFSInputStream.getBlockAt(long) again.
void	releaseBuffer(ByteBuffer buffer)
protected void	reportLostBlock(LocatedBlock lostBlock, Collection<DatanodeInfo> ignoredNodes)	Warn the user of a lost block
void	seek(long targetPos)	Seek to a new arbitrary location.
boolean	seekToNewSource(long targetPos)	Seek to given position on a node other than the current node.
void	unbuffer()

Methods inherited from class org.apache.hadoop.hdfs.DFSInputStream

addToLocalDeadNodes, available, clearReadStatistics, fetchBlockAt, getAllBlocks, getBestNodeDNAddrPair, getBlockAt, getBlockReader, getCurrentBlock, getCurrentBlockLocationsLength, getCurrentDatanode, getDFSClient, getFileEncryptionInfo, getFileLength, getHedgedReadOpsLoopNumForTesting, getLocalDeadNodes, getLocatedBlocks, getPos, getReadStatistics, getSrc, hasCapability, mark, markSupported, maybeRegisterBlockRefresh, read, read, read, read, read, readFully, removeFromLocalDeadNodes, reportCheckSumFailure, reset, setDropBehind, setReadahead, skip, tokenRefetchNeeded

Methods inherited from class org.apache.hadoop.fs.FSInputStream

readFully, readFully, toString, validatePositionedReadArgs

Methods inherited from class java.io.InputStream

nullInputStream, read, readAllBytes, readNBytes, readNBytes, skipNBytes, transferTo

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.fs.PositionedReadable

maxReadSizeForVectorReads, minSeekForVectorReads, readVectored, readVectored

Field Details

parityBuf

@VisibleForTesting
protected ByteBuffer parityBuf

Method Details

getParityBuffer

protected ByteBuffer getParityBuffer()

getCurStripeBuf

protected ByteBuffer getCurStripeBuf()

getBufferPool

protected org.apache.hadoop.io.ByteBufferPool getBufferPool()

getStripedReadsThreadPool

protected ThreadPoolExecutor getStripedReadsThreadPool()

close

public void close()
           throws IOException

Description copied from class: DFSInputStream

Close it down!

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Overrides:

close in class DFSInputStream

Throws:

IOException

closeCurrentBlockReaders

protected void closeCurrentBlockReaders()

Extend the super method with the logic of switching between cells. When reaching the end of a cell, proceed to the next cell and read it with the next blockReader.

Overrides:

closeCurrentBlockReaders in class DFSInputStream

closeReader

protected void closeReader(org.apache.hadoop.hdfs.StripeReader.BlockReaderInfo readerInfo)

seek

public void seek(long targetPos)
          throws IOException

Seek to a new arbitrary location.

Specified by:

seek in interface org.apache.hadoop.fs.Seekable

Overrides:

seek in class DFSInputStream

Throws:

IOException

seekToNewSource

public boolean seekToNewSource(long targetPos)
                        throws IOException

Description copied from class: DFSInputStream

Seek to given position on a node other than the current node. If a node other than the current node is found, then returns true. If another node could not be found, then returns false.

Specified by:

seekToNewSource in interface org.apache.hadoop.fs.Seekable

Overrides:

seekToNewSource in class DFSInputStream

Throws:

IOException

readWithStrategy

protected int readWithStrategy(org.apache.hadoop.hdfs.ReaderStrategy strategy)
                        throws IOException

Overrides:

readWithStrategy in class DFSInputStream

Throws:

IOException

refreshLocatedBlock

protected LocatedBlock refreshLocatedBlock(LocatedBlock block)
                                    throws IOException

The super method DFSInputStream.refreshLocatedBlock(org.apache.hadoop.hdfs.protocol.LocatedBlock) refreshes cached LocatedBlock by executing DFSInputStream.getBlockAt(long) again. This method extends the logic by first remembering the index of the internal block, and re-parsing the refreshed block group with the same index.

Overrides:

refreshLocatedBlock in class DFSInputStream

Parameters:

block - The currently cached block locations

Returns:

Refreshed block locations

Throws:

IOException

fetchBlockByteRange

protected void fetchBlockByteRange(LocatedBlock block,
 long start,
 long end,
 ByteBuffer buf,
 DFSUtilClient.CorruptedBlocks corruptedBlocks,
 Map<InetSocketAddress,List<IOException>> exceptionMap)
                            throws IOException

Real implementation of pread.

Note: exceptionMap is not populated with ioExceptions as what we added for DFSInputStream. If you need this function, please implement it.

Overrides:

fetchBlockByteRange in class DFSInputStream

Throws:

IOException

reportLostBlock

protected void reportLostBlock(LocatedBlock lostBlock,
 Collection<DatanodeInfo> ignoredNodes)

Description copied from class: DFSInputStream

Warn the user of a lost block

Overrides:

reportLostBlock in class DFSInputStream

read

public ByteBuffer read(org.apache.hadoop.io.ByteBufferPool bufferPool,
 int maxLength,
 EnumSet<org.apache.hadoop.fs.ReadOption> opts)
                throws IOException,
UnsupportedOperationException

May need online read recovery, zero-copy read doesn't make sense, so don't support it.

Specified by:

read in interface org.apache.hadoop.fs.HasEnhancedByteBufferAccess

Overrides:

read in class DFSInputStream

Throws:

IOException

UnsupportedOperationException

releaseBuffer

public void releaseBuffer(ByteBuffer buffer)

Specified by:

releaseBuffer in interface org.apache.hadoop.fs.HasEnhancedByteBufferAccess

Overrides:

releaseBuffer in class DFSInputStream

unbuffer

public void unbuffer()

Specified by:

unbuffer in interface org.apache.hadoop.fs.CanUnbuffer

Overrides:

unbuffer in class DFSInputStream