[
https://issues.apache.org/jira/browse/HADOOP-10714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-10714:
------------------------------------
Attachment: HADOOP-10714-007.patch
Updated patch. I haven't run the scale tests themselves; got into issues with
the core tests and some code review work.
# {{testRenameWithNonEmptySubDir() }} sticks its data straight into the
hadoop-common dir
{{hadoop-common-project/hadoop-common/testRenameWithNonEmptySubDir/}}. The base
class {{path()}} method
sets up directories properly. (fixed)
# {{BasicAWSCredentialsProvider}} and credential setup gets trimmed config
option, checks for "" as well as null.
# downgraded logs of FS operations from INFO to DEBUG
# fixed "livetest" probes to compare scheme in test URI; existing comparator
wouldn't have worked.
# renamed {{TestDeleteManyFiles}} to {{TestS3ADeleteManyFiles}}
# saw intermittent failures of {{TestS3AContractRename}}; after which test
wouldn't restart.
{code}
testRenameWithNonEmptySubDir(org.apache.hadoop.fs.contract.s3a.TestS3AContractRename)
Time elapsed: 17.563 sec <<< FAILURE!
java.lang.AssertionError: not deleted: unexpectedly found
testRenameWithNonEmptySubDir/src1/source.txt as
S3AFileStatus{path=s3a://tests3neu/user/stevel/testRenameWithNonEmptySubDir/src1/source.txt;
isDirectory=false; length=27; replication=1; blocksize=0;
modification_time=1411936074000; access_time=0; owner=; group=;
permission=rw-rw-rw-; isSymlink=false}
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.fs.contract.ContractTestUtils.assertPathDoesNotExist(ContractTestUtils.java:702)
at
org.apache.hadoop.fs.contract.AbstractContractRenameTest.testRenameWithNonEmptySubDir(AbstractContractRenameTest.java:223)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
The next run failed as the source dirs were still there:
{code}
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 48.168 sec <<<
FAILURE! - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
testRenameWithNonEmptySubDir(org.apache.hadoop.fs.contract.s3a.TestS3AContractRename)
Time elapsed: 6.49 sec <<< ERROR!
org.apache.hadoop.fs.FileAlreadyExistsException:
testRenameWithNonEmptySubDir/src1/source.txt already exists
at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:252)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
at
org.apache.hadoop.fs.contract.ContractTestUtils.createFile(ContractTestUtils.java:507)
at
org.apache.hadoop.fs.contract.ContractTestUtils.writeTextFile(ContractTestUtils.java:491)
at
org.apache.hadoop.fs.contract.AbstractContractRenameTest.testRenameWithNonEmptySubDir(AbstractContractRenameTest.java:198)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
Fix: make sure dirs are cleaned before test runs. applied same change to scale
test
> AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call
> --------------------------------------------------------------------------
>
> Key: HADOOP-10714
> URL: https://issues.apache.org/jira/browse/HADOOP-10714
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.5.0
> Reporter: David S. Wang
> Assignee: Juan Yu
> Priority: Critical
> Labels: s3
> Attachments: HADOOP-10714-007.patch, HADOOP-10714-1.patch,
> HADOOP-10714.001.patch, HADOOP-10714.002.patch, HADOOP-10714.003.patch,
> HADOOP-10714.004.patch, HADOOP-10714.005.patch, HADOOP-10714.006.patch
>
>
> In the patch for HADOOP-10400, calls to AmazonS3Client.deleteObjects() need
> to have the number of entries at 1000 or below. Otherwise we get a Malformed
> XML error similar to:
> com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS
> Service: Amazon S3, AWS Request ID: 6626AD56A3C76F5B, AWS Error Code:
> MalformedXML, AWS Error Message: The XML you provided was not well-formed or
> did not validate against our published schema, S3 Extended Request ID:
> DOt6C+Y84mGSoDuaQTCo33893VaoKGEVC3y1k2zFIQRm+AJkFH2mTyrDgnykSL+v
> at
> com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3480)
> at
> com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:1739)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:388)
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:829)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:874)
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:878)
> Note that this is mentioned in the AWS documentation:
> http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html
> "The Multi-Object Delete request contains a list of up to 1000 keys that you
> want to delete. In the XML, you provide the object key names, and optionally,
> version IDs if you want to delete a specific version of the object from a
> versioning-enabled bucket. For each key, Amazon S3….”
> Thanks to Matteo Bertozzi and Rahul Bhartia from AWS for identifying the
> problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
