[
https://issues.apache.org/jira/browse/HADOOP-13793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-13793:
----------------------------------
Attachment: HADOOP-13793-HADOOP-13345.001.patch
Attaching v1 patch.
This adds a new integration test, {{ITestS3GuardListConsistency}} which sets up
the failure (inconsistency) injection and enables the in-memory MetadataStore,
then asserts that listStatus() is consistent even though visibility of one of
the paths it creates has been delayed.
To accomplish the inconsistency injection, I
- Promoted DefaultS3ClientFactory to a top-level class (it was an internal
class in S3ClientFactory interface)
- Pulled out the one-line call to construct a AmazonS3Client into an
overridable function.
- Create new class InconsistentS3ClientFactory which just subclases
DefaultS3ClientFactory and creates InconsistentS3Client instead of
AmazonS3Client.
- Created new class InconsistentS3Client which wraps AmazonS3Client, and delays
listObjects() visibility of any keys which match a given substring by 500 msec
after they are created.
I confirmed that, in ITestS3GuardListConsistency, if you comment out this line,
the test will fail:
{code}
+ conf.setClass(S3Guard.S3_METADATA_STORE_IMPL, LocalMetadataStore.class,
+ MetadataStore.class);
{code}
> s3guard: add inconsistency injection, integration tests
> -------------------------------------------------------
>
> Key: HADOOP-13793
> URL: https://issues.apache.org/jira/browse/HADOOP-13793
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Aaron Fabbri
> Assignee: Aaron Fabbri
> Attachments: HADOOP-13793-HADOOP-13345.001.patch
>
>
> Many of us share concerns that testing the consistency features of S3Guard
> will be difficult if we depend on the rare and unpredictable occurrence of
> actual inconsistency in S3 to exercise those code paths.
> I think we should have a mechanism for injecting failure to force exercising
> of the consistency codepaths in S3Guard.
> Requirements:
> - Integration tests that cause S3A to see the types of inconsistency we
> address with S3Guard.
> - These are deterministic integration tests.
> Unit tests are possible as well, if we were to stub out the S3Client. That
> may be less bang for the buck, though.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
