[
https://issues.apache.org/jira/browse/HADOOP-10714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141506#comment-14141506
]
Charles Lamb commented on HADOOP-10714:
---------------------------------------
General:
* Several lines bust the 80 char limit.
* There are several places where there are extra newlines. There should only
ever be one blank line.
* Remove unused imports in your new files.
S3AFileSystem
* // deleteUnnecessaryFakeDirectories
This looks like the name of a method rather than a comment.
TestS3AContractRename
* s/s3a don't/s3a doesn't/
core-site.xml
{quote}
<!-- Values used when running unit tests. This is mostly empty, to -->
<!-- use of the default values, overriding the potentially -->
<!-- user-editted core-site.xml in the conf/ directory. -->
{quote}
can be changed to:
{quote}
<!-- Values used when running unit tests. Specify any values in here that
should override the default values. -->
{quote}
* You can remove the extra newline after <configuration>.
S3AFileSystemBaseTest
* s/If you keys/If your keys/, s/as passed/as passed./
S3AScaleTestBase
* remove the extra newline after the class comment.
S3ATestUtils
* You can move "implements S3ATestConstants" to the line above.
* testReceivedData decl has incorrect indentation. Extra newline at the end of
this method.
* generateTestData has the same indent problem in the decl.
* I'd prefer that you do import statics of various org.junit.Assert methods in
TestS3AFileSYstemBasicOps rather than extending S3AFileSystemBaseTest with it.
TestS3AFileSystemBasicOps
* remove the extra newline after the class comment.
I'm unclear about what the test renaming is about. Could you please comment on
that in the Jira?
TestS3AFileSystemContract
{quote}
if (!renameSupported()) return;
{quote}
should be changed to:
{quote}
if (!renameSupported()) {
return;
}
{quote}
* remove the blank line before the closing } of
testRenameDirectoryAsExistingDirectory(), ditto blank line before last } in the
file.
> AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call
> --------------------------------------------------------------------------
>
> Key: HADOOP-10714
> URL: https://issues.apache.org/jira/browse/HADOOP-10714
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.5.0
> Reporter: David S. Wang
> Assignee: Juan Yu
> Priority: Critical
> Labels: s3
> Attachments: HADOOP-10714-1.patch, HADOOP-10714.001.patch
>
>
> In the patch for HADOOP-10400, calls to AmazonS3Client.deleteObjects() need
> to have the number of entries at 1000 or below. Otherwise we get a Malformed
> XML error similar to:
> com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS
> Service: Amazon S3, AWS Request ID: 6626AD56A3C76F5B, AWS Error Code:
> MalformedXML, AWS Error Message: The XML you provided was not well-formed or
> did not validate against our published schema, S3 Extended Request ID:
> DOt6C+Y84mGSoDuaQTCo33893VaoKGEVC3y1k2zFIQRm+AJkFH2mTyrDgnykSL+v
> at
> com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3480)
> at
> com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:1739)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:388)
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:829)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:874)
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:878)
> Note that this is mentioned in the AWS documentation:
> http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html
> "The Multi-Object Delete request contains a list of up to 1000 keys that you
> want to delete. In the XML, you provide the object key names, and optionally,
> version IDs if you want to delete a specific version of the object from a
> versioning-enabled bucket. For each key, Amazon S3….”
> Thanks to Matteo Bertozzi and Rahul Bhartia from AWS for identifying the
> problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
