huaxiangsun opened a new pull request, #4608:
URL: https://github.com/apache/hadoop/pull/4608
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
processDeleteOnExit() is overiden in S3AFilesystem, it skips exist() check
and delete objects without checking if FileSystem is closed.
### How was this patch tested?
A new unitest case is added. And all unittest cases under
hadoop-tools/hadoop-aws passed.
mvn -Dparallel-tests clean test
Did S3A Integration tests against us-west-2 region and there are a few
failures/errors. Run the trunk code without the patch, there are same
errors/failures. The errors/failures are not caused by the patch, probably due
to misconfiguration ( I could not figure out)
mvn -Dparallel-tests clean verify
The result is
`
Tests | Errors | Failures | Skipped | Success Rate | Time
-- | -- | -- | -- | -- | --
1252 | 6 | 1 | 270 | 77.875% | 3,627.473
`
The errors are
`
org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob#testCommonCrawlLookup[1]
+ [ Detail ] | 0.324
| s3a://hbase-test-data/fork-0001/test: getFileStatus on
s3a://hbase-test-data/fork-0001/test:
com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you
provided does not exist in our records. (Service: Amazon S3; Status Code: 403;
Error Code: InvalidAccessKeyId; Request ID: XJ3DCCR6Q7SXTJDW; S3 Extended
Request ID:
fRmP3m1lThWxhj3s9VkSNEtuBz1JeBWYw65aRajrSg/H7IN+muB7d8PavSeqJ2urvLZtguTbnlc=;
Proxy: null), S3 Extended Request ID:
fRmP3m1lThWxhj3s9VkSNEtuBz1JeBWYw65aRajrSg/H7IN+muB7d8PavSeqJ2urvLZtguTbnlc=:InvalidAccessKeyId
|
| |
| testJobSubmissionCollectsTokens[1] + [ Detail ] | 0.329
| s3a://hbase-test-data/fork-0001/test: getFileStatus on
s3a://hbase-test-data/fork-0001/test:
com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you
provided does not exist in our records. (Service: Amazon S3; Status Code: 403;
Error Code: InvalidAccessKeyId; Request ID: XJ35WHK7X6EMP9B6; S3 Extended
Request ID:
rtWEsDYcGqNiaoKy2D5EQQqN+O7MbYe1bYbiSmkF+FOz9/wb6+t+dQooqj7ppCSCZMBgC3PeEw4=;
Proxy: null), S3 Extended Request ID:
rtWEsDYcGqNiaoKy2D5EQQqN+O7MbYe1bYbiSmkF+FOz9/wb6+t+dQooqj7ppCSCZMBgC3PeEw4=:InvalidAccessKeyId
`
`
org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion#testBlankRegionTriggersSDKResolution
+ [ Detail ] | 2.817
| [Client region name] expected:<"[mars-north]-2"> but was:<"[us-west]-2">
`
and
`
org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials#estSTS
| : request session credentials:
com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
Cannot call GetSessionToken with session credentials (Service:
AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request
ID: d935696d-7d98-4a1c-825f-22bf3d28ed9d; Proxy: null):AccessDenied
`
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
