Package org.apache.hadoop.hdfs.server.namenode

Class FileJournalManager

java.lang.Object
org.apache.hadoop.hdfs.server.namenode.FileJournalManager
All Implemented Interfaces:
Closeable, AutoCloseable, Storage.FormatConfirmable, JournalManager
@Private public class FileJournalManager extends Object implements JournalManager
Journal manager for the common case of edits files being written to a storage directory. Note: this class is not thread-safe and should be externally synchronized.

  • Constructor Details

  • Method Details

    • close

      public void close() throws IOException
      Description copied from interface: JournalManager
      Close the journal manager, freeing any resources it may hold.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Specified by:
      close in interface JournalManager
      Throws:
      IOException

    • format

      public void format(NamespaceInfo ns, boolean force) throws IOException
      Description copied from interface: JournalManager
      Format the underlying storage, removing any previously stored data.
      Specified by:
      format in interface JournalManager
      Throws:
      IOException

    • hasSomeData

      public boolean hasSomeData()
      Specified by:
      hasSomeData in interface Storage.FormatConfirmable
      Returns:
      true if the storage seems to have some valid data in it, and the user should be required to confirm the format. Otherwise, false.

    • startLogSegment

      public EditLogOutputStream startLogSegment(long txid, int layoutVersion) throws IOException
      Description copied from interface: JournalManager
      Begin writing to a new segment of the log stream, which starts at the given transaction ID.
      Specified by:
      startLogSegment in interface JournalManager
      Throws:
      IOException

    • finalizeLogSegment

      public void finalizeLogSegment(long firstTxId, long lastTxId) throws IOException
      Description copied from interface: JournalManager
      Mark the log segment that spans from firstTxId to lastTxId as finalized and complete.
      Specified by:
      finalizeLogSegment in interface JournalManager
      Throws:
      IOException

    • getStorageDirectory

      @VisibleForTesting public Storage.StorageDirectory getStorageDirectory()

    • setOutputBufferCapacity

      public void setOutputBufferCapacity(int size)
      Description copied from interface: JournalManager
      Set the amount of memory that this stream should use to buffer edits
      Specified by:
      setOutputBufferCapacity in interface JournalManager

    • getLastReadableTxId

      public long getLastReadableTxId()

    • setLastReadableTxId

      public void setLastReadableTxId(long id)

    • purgeLogsOlderThan

      public void purgeLogsOlderThan(long minTxIdToKeep) throws IOException
      Purges the unnecessary edits and edits_inprogress files. Edits files that are ending before the minTxIdToKeep are purged. Edits in progress files that are starting before minTxIdToKeep are purged. Edits in progress files that are marked as empty, trash, corrupted or stale by file extension and starting before minTxIdToKeep are purged. Edits in progress files that are after minTxIdToKeep, but before the current edits in progress files are marked as stale for clarity. In case file removal or rename is failing a warning is logged, but that does not fail the operation.
      Parameters:
      minTxIdToKeep - the lowest transaction ID that should be retained
      Throws:
      IOException - if listing the storage directory fails.

    • getRemoteEditLogs

      public List<RemoteEditLog> getRemoteEditLogs(long firstTxId, boolean inProgressOk) throws IOException
      Find all editlog segments starting at or above the given txid.
      Parameters:
      firstTxId - the txnid which to start looking
      inProgressOk - whether or not to include the in-progress edit log segment
      Returns:
      a list of remote edit logs
      Throws:
      IOException - if edit logs cannot be listed.

    • matchEditLogs

      public static List<FileJournalManager.EditLogFile> matchEditLogs(File logDir) throws IOException
      returns matching edit logs via the log directory. Simple helper function that lists the files in the logDir and calls matchEditLogs(File[])
      Parameters:
      logDir - directory to match edit logs in
      Returns:
      matched edit logs
      Throws:
      IOException - IOException thrown for invalid logDir

    • selectInputStreams

      public void selectInputStreams(Collection<EditLogInputStream> streams, long fromTxnId, boolean inProgressOk) throws IOException
      Throws:
      IOException

    • selectInputStreams

      public void selectInputStreams(Collection<EditLogInputStream> streams, long fromTxId, boolean inProgressOk, boolean onlyDurableTxns) throws IOException
      Throws:
      IOException

    • recoverUnfinalizedSegments

      public void recoverUnfinalizedSegments() throws IOException
      Description copied from interface: JournalManager
      Recover segments which have not been finalized.
      Specified by:
      recoverUnfinalizedSegments in interface JournalManager
      Throws:
      IOException

    • getLogFiles

      public List<FileJournalManager.EditLogFile> getLogFiles(long fromTxId) throws IOException
      Throws:
      IOException

    • getLogFile

      public FileJournalManager.EditLogFile getLogFile(long startTxId) throws IOException
      Throws:
      IOException

    • getLogFile

      public FileJournalManager.EditLogFile getLogFile(long startTxId, boolean inProgressOk) throws IOException
      Throws:
      IOException

    • getLogFile

      public static FileJournalManager.EditLogFile getLogFile(File dir, long startTxId) throws IOException
      Throws:
      IOException

    • getLogFile

      public static FileJournalManager.EditLogFile getLogFile(File dir, long startTxId, boolean inProgressOk) throws IOException
      Throws:
      IOException

    • toString

      public String toString()
      Specified by:
      toString in interface Storage.FormatConfirmable
      Overrides:
      toString in class Object
      Returns:
      a string representation of the formattable item, suitable for display to the user inside a prompt

    • doPreUpgrade

      public void doPreUpgrade() throws IOException
      Description copied from interface: JournalManager
      Perform any steps that must succeed across all JournalManagers involved in an upgrade before proceeding onto the actual upgrade stage. If a call to any JM's doPreUpgrade method fails, then doUpgrade will not be called for any JM.
      Specified by:
      doPreUpgrade in interface JournalManager
      Throws:
      IOException

    • doUpgrade

      public void doUpgrade(Storage storage) throws IOException
      This method assumes that the fields of the Storage object have already been updated to the appropriate new values for the upgrade.
      Specified by:
      doUpgrade in interface JournalManager
      Parameters:
      storage - info about the new upgraded versions.
      Throws:
      IOException

    • doFinalize

      public void doFinalize() throws IOException
      Description copied from interface: JournalManager
      Finalize the upgrade. JMs should purge any state that they had been keeping around during the upgrade process. After this is completed, rollback is no longer allowed.
      Specified by:
      doFinalize in interface JournalManager
      Throws:
      IOException

    • canRollBack

      public boolean canRollBack(StorageInfo storage, StorageInfo prevStorage, int targetLayoutVersion) throws IOException
      Description copied from interface: JournalManager
      Return true if this JM can roll back to the previous storage state, false otherwise. The NN will refuse to run the rollback operation unless at least one JM or fsimage storage directory can roll back.
      Specified by:
      canRollBack in interface JournalManager
      Parameters:
      storage - the storage info for the current state
      prevStorage - the storage info for the previous (unupgraded) state
      targetLayoutVersion - the layout version we intend to roll back to
      Returns:
      true if this JM can roll back, false otherwise.
      Throws:
      IOException

    • doRollback

      public void doRollback() throws IOException
      Description copied from interface: JournalManager
      Perform the rollback to the previous FS state. JMs which do not need to roll back their state should just return without error.
      Specified by:
      doRollback in interface JournalManager
      Throws:
      IOException

    • discardSegments

      public void discardSegments(long startTxid) throws IOException
      Description copied from interface: JournalManager
      Discard the segments whose first txid is >= the given txid.
      Specified by:
      discardSegments in interface JournalManager
      Parameters:
      startTxid - The given txid should be right at the segment boundary, i.e., it should be the first txid of some segment, if segment corresponding to the txid exists.
      Throws:
      IOException

    • getJournalCTime

      public long getJournalCTime() throws IOException
      Specified by:
      getJournalCTime in interface JournalManager
      Returns:
      the CTime of the journal manager.
      Throws:
      IOException