[
https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16157209#comment-16157209
]
Steve Loughran commented on HADOOP-14220:
-----------------------------------------
HADOOP-14220 patch 006
* Fix Aaron's points
* tune how ExitUtils is invoked
* try to make some intermittent test failures which only surface in parallel
runs
Testing, s3 frankfurt. Saw some intermittent failures
{code}
Tests run: 52, Failures: 0, Errors: 0, Skipped: 52, Time elapsed: 0.537 sec -
in org.apache.hadoop.fs.s3native.ITestInMemoryNativeS3FileSystemContract
Running org.apache.hadoop.fs.s3native.ITestJets3tNativeFileSystemStore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.273 sec - in
org.apache.hadoop.fs.s3native.ITestJets3tNativeFileSystemStore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.147 sec - in
org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.514 sec - in
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardConcurrentOps
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.793 sec - in
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB
Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 54.233 sec -
in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
Tests run: 43, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 98.268 sec -
in org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 118.548 sec <<<
FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
testConsistentListFiles(org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency)
Time elapsed: 46.552 sec <<< FAILURE!
java.lang.AssertionError: Unexpected number of files returned by listFiles()
call expected:<3> but was:<2>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.doTestListFiles(ITestS3GuardListConsistency.java:438)
at
org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testConsistentListFiles(ITestS3GuardListConsistency.java:359)
{code}
swapped some of the checks around, got a bit more meaningful failure
{code}
Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.388 sec -
in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 100.955 sec <<<
FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
testConsistentListFiles(org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency)
Time elapsed: 27.809 sec <<< FAILURE!
java.lang.AssertionError:
s3a://hwdev-steve-frankfurt-new/fork-7/test/doTestListFiles-2-0-1-true/file-0-DELAY_LISTING_ME
should have been listed
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.verifyFileIsListed(ITestS3GuardListConsistency.java:455)
at
org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.doTestListFiles(ITestS3GuardListConsistency.java:443)
at
org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testConsistentListFiles(ITestS3GuardListConsistency.java:359)
{code}
and a new test
{code}
unning org.apache.hadoop.fs.s3native.ITestJets3tNativeFileSystemStore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.314 sec - in
org.apache.hadoop.fs.s3native.ITestJets3tNativeFileSystemStore
Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 9.27 sec - in
org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.04 sec - in
org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.634 sec -
in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 46.742 sec <<<
FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB
testDynamoDBInitDestroyCycle(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
Time elapsed: 12.374 sec <<< FAILURE!
java.lang.AssertionError: No Dynamo diagnostics in output Filesystem
s3a://hwdev-steve-frankfurt-new
Location: eu-central-1
Filesystem s3a://hwdev-steve-frankfurt-new is using S3Guard with store
LocalMetadataStore{, uriHost='hwdev-steve-frankfurt-new'}
Authoritative S3Guard: fs.s3a.metadatastore.authoritative=false
Metadata Store Diagnostics:
name=local://metadata
description=Local in-VM metadata store for testing
uriHost=hwdev-steve-frankfurt-new
S3A Client
Endpoint: fs.s3a.endpoint=s3.eu-central-1.amazonaws.com
Encryption: fs.s3a.server-side-encryption-algorithm=none
Input seek policy: fs.s3a.experimental.input.fadvise=normal
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB.testDynamoDBInitDestroyCycle(ITestS3GuardToolDynamoDB.java:126)
Tests run: 43, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 75.652 sec -
in org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 83.129 sec - in
org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
Tests run: 62, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 147.703 sec -
in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
Results :
Failed tests:
ITestS3GuardToolDynamoDB.testDynamoDBInitDestroyCycle:126->Assert.assertTrue:41->Assert.fail:88
No Dynamo diagnostics in output Filesystem s3a://hwdev-steve-frankfurt-new
Location: eu-central-1
Filesystem s3a://hwdev-steve-frankfurt-new is using S3Guard with store
LocalMetadataStore{, uriHost='hwdev-steve-frankfurt-new'}
Authoritative S3Guard: fs.s3a.metadatastore.authoritative=false
Metadata Store Diagnostics:
name=local://metadata
description=Local in-VM metadata store for testing
uriHost=hwdev-steve-frankfurt-new
S3A Client
Endpoint: fs.s3a.endpoint=s3.eu-central-1.amazonaws.com
Encryption: fs.s3a.server-side-encryption-algorithm=none
Input seek policy: fs.s3a.experimental.input.fadvise=normal
{code}
Went away later. Looks like there's a cached entry in the JVM. Fix: disable
caching in all s3guard tests
Note also: "LocalMetadataStore{, uriHost='hwdev-steve-frankfurt-new'}"...fixed
that
> Enhance S3GuardTool with bucket-info and set-capacity commands, tests
> ---------------------------------------------------------------------
>
> Key: HADOOP-14220
> URL: https://issues.apache.org/jira/browse/HADOOP-14220
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0-beta1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-14220-006.patch, HADOOP-14220-007.patch,
> HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch,
> HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch,
> HADOOP-14220-HADOOP-13345-005.patch
>
>
> Add a diagnostics command to s3guard which does whatever we need to diagnose
> problems for a specific (named) s3a url. This is something which can be
> attached to bug reports as well as used by developers.
> * Properties to log (with provenance attribute, which can track bucket
> overrides: s3guard metastore setup, autocreate, capacity,....
> * table present/absent
> * # of keys in DDB table for that bucket?
> * any other stats?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
