[
https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15715389#comment-15715389
]
Steve Loughran commented on HADOOP-13786:
-----------------------------------------
Update: the way s3a does commits via mergeOrUpdate() isn't really that atomic.
It's atomic per destination rename, but as it treewalks doing a merge, there
are actual potentials for races/inconsistent outcomes. What s3guard is at least
guarantee that the view of the FS during that walk is consistent across clients.
I'm checking in/pushing up an update, where I've been slowly moving up to some
s3a internals, more for performance than consistency.
Examples:
* switch to {{innerDelete(FileStatus)}} for the delete, adding a parameter
there to allow the check for maybe creating a parent dir to be skipped. As we
know there's about be a new child, skipping that avoids 1-3 HEAD calls and a
PUT which will soon be deleted
* switch to an implementation of {{FileSystem.rename/3}}. This raises
exceptions and lets us choose overwrite policy (good), but it still does two
getFileStatusCalls, one for the src and one for the dest. As we have the source
and have just deleted the test, we don't need them again. Having an @Privat
method which took the source and dest values would strip out another 2-6 HTTP
requests. Admittedly, on a large commit the cost of his preamble is low —the
COPY becomes the expense.
> add output committer which uses s3guard for consistent O(1) commits to S3
> -------------------------------------------------------------------------
>
> Key: HADOOP-13786
> URL: https://issues.apache.org/jira/browse/HADOOP-13786
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0-alpha2
> Reporter: Steve Loughran
> Assignee: Steve Loughran
>
> A goal of this code is "support O(1) commits to S3 repositories in the
> presence of failures". Implement it, including whatever is needed to
> demonstrate the correctness of the algorithm. (that is, assuming that s3guard
> provides a consistent view of the presence/absence of blobs, show that we can
> commit directly).
> I consider ourselves free to expose the blobstore-ness of the s3 output
> streams (ie. not visible until the close()), if we need to use that to allow
> us to abort commit operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
