Package org.apache.hadoop.fs.impl

Class FileSystemMultipartUploader

java.lang.Object

org.apache.hadoop.fs.impl.AbstractMultipartUploader

org.apache.hadoop.fs.impl.FileSystemMultipartUploader

All Implemented Interfaces:

Closeable, AutoCloseable, MultipartUploader, IOStatisticsSource

@Private
@Unstable
public class FileSystemMultipartUploader
extends AbstractMultipartUploader

A MultipartUploader that uses the basic FileSystem commands. This is done in three stages:

• Init - create a temp _multipart directory.
• PutPart - copying the individual parts of the file to the temp directory.
• Complete - use FileSystem.concat(org.apache.hadoop.fs.Path, org.apache.hadoop.fs.Path[]) to merge the files; and then delete the temp directory.

Constructor Summary

Constructors

Constructor	Description
FileSystemMultipartUploader(FileSystemMultipartUploaderBuilder builder, FileSystem fs)

Method Summary

Modifier and Type	Method	Description
CompletableFuture<Void>	abort(UploadHandle uploadId, Path filePath)	Aborts a multipart upload.
CompletableFuture<PathHandle>	complete(UploadHandle uploadId, Path filePath, Map<Integer,PartHandle> handleMap)	Complete a multipart upload.
CompletableFuture<PartHandle>	putPart(UploadHandle uploadId, int partNumber, boolean isLastPart, Path filePath, InputStream inputStream, long lengthInBytes)	Put part as part of a multipart upload.
CompletableFuture<UploadHandle>	startUpload(Path filePath)	Initialize a multipart upload.

Methods inherited from class org.apache.hadoop.fs.impl.AbstractMultipartUploader

abortUploadsUnderPath, checkPartHandles, checkPath, checkPutArguments, checkUploadId, close, getBasePath

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.fs.statistics.IOStatisticsSource

getIOStatistics

Constructor Details

FileSystemMultipartUploader

public FileSystemMultipartUploader(FileSystemMultipartUploaderBuilder builder,
 FileSystem fs)

Method Details

startUpload

public CompletableFuture<UploadHandle> startUpload(Path filePath)
                                            throws IOException

Description copied from interface: MultipartUploader

Initialize a multipart upload.

Parameters:

filePath - Target path for upload.

Returns:

unique identifier associating part uploads.

Throws:

IOException - IO failure

putPart

public CompletableFuture<PartHandle> putPart(UploadHandle uploadId,
 int partNumber,
 boolean isLastPart,
 Path filePath,
 InputStream inputStream,
 long lengthInBytes)
                                      throws IOException

Description copied from interface: MultipartUploader

Put part as part of a multipart upload. It is possible to have parts uploaded in any order (or in parallel).

Parameters:

uploadId - Identifier from MultipartUploader.startUpload(Path).

partNumber - Index of the part relative to others.

isLastPart - is the part the last part of the upload?

filePath - Target path for upload (as MultipartUploader.startUpload(Path)).

inputStream - Data for this part. Implementations MUST close this stream after reading in the data.

lengthInBytes - Target length to read from the stream.

Returns:

unique PartHandle identifier for the uploaded part.

Throws:

IOException - IO failure

complete

public CompletableFuture<PathHandle> complete(UploadHandle uploadId,
 Path filePath,
 Map<Integer,PartHandle> handleMap)
                                       throws IOException

Description copied from interface: MultipartUploader

Complete a multipart upload.

Parameters:

uploadId - Identifier from MultipartUploader.startUpload(Path).

filePath - Target path for upload (as MultipartUploader.startUpload(Path).

handleMap - non-empty map of part number to part handle. from MultipartUploader.putPart(UploadHandle, int, boolean, Path, InputStream, long).

Returns:

unique PathHandle identifier for the uploaded file.

Throws:

IOException - IO failure

abort

public CompletableFuture<Void> abort(UploadHandle uploadId,
 Path filePath)
                              throws IOException

Description copied from interface: MultipartUploader

Aborts a multipart upload.

Parameters:

uploadId - Identifier from MultipartUploader.startUpload(Path).

filePath - Target path for upload (same as MultipartUploader.startUpload(Path).

Returns:

a future; the operation will have completed

Throws:

IOException - IO failure