[
https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13786:
------------------------------------
Attachment: HADOOP-13786-HADOOP-13345-010.patch
Patch 010; work on the mock tests. Overall, ~50% success rate.
These tests are failing for two main reasons
# I've broken the code
# In changing the code, I've broken the test
A fair few of the tests are failing in that the mock calls don't follow the
path they've expected; that's the problem with mocks...you are asserting about
the internal operations, rather than the final observed state of the SUT.
Sometimes that's good for looking into the interior, but it's very, very
brittle.
These tests are all doing something ugly to get a mock s3a FS set up for code
to get when they ask for an FS. I plan to remove that wrapper mock and inject
whatever mock FS is created straight into the FileSystem.get() cache. That's
the proper way to do it.
After that, I'll look at why the tests are failing, focusing on ones where
results are not what is expected, rather than just mock counter mismatch. I'll
assume those are false alarms for now, and only worry about the details once
the more functional tests have passed
> Add S3Guard committer for zero-rename commits to consistent S3 endpoints
> ------------------------------------------------------------------------
>
> Key: HADOOP-13786
> URL: https://issues.apache.org/jira/browse/HADOOP-13786
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-13786-HADOOP-13345-001.patch,
> HADOOP-13786-HADOOP-13345-002.patch, HADOOP-13786-HADOOP-13345-003.patch,
> HADOOP-13786-HADOOP-13345-004.patch, HADOOP-13786-HADOOP-13345-005.patch,
> HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-006.patch,
> HADOOP-13786-HADOOP-13345-007.patch, HADOOP-13786-HADOOP-13345-009.patch,
> HADOOP-13786-HADOOP-13345-010.patch, s3committer-master.zip
>
>
> A goal of this code is "support O(1) commits to S3 repositories in the
> presence of failures". Implement it, including whatever is needed to
> demonstrate the correctness of the algorithm. (that is, assuming that s3guard
> provides a consistent view of the presence/absence of blobs, show that we can
> commit directly).
> I consider ourselves free to expose the blobstore-ness of the s3 output
> streams (ie. not visible until the close()), if we need to use that to allow
> us to abort commit operations.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
