[
https://issues.apache.org/jira/browse/HADOOP-15209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386708#comment-16386708
]
Steve Loughran commented on HADOOP-15209:
-----------------------------------------
And manual distcp to s3 london of hadoop-auth followed by a mvn clean then a
distcp -update -delete
{code}
2018-03-05 19:29:31,084 [pool-6-thread-1] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(486)) - Added 13 entries; ignored 0;
hasNext=true; hasMoreObjects=false
2018-03-05 19:29:31,084 [pool-6-thread-1] DEBUG s3a.S3AFileSystem
(Listing.java:sourceHasNext(378)) - Start iterating the provided status.
2018-03-05 19:29:31,088 [Thread-217] INFO tools.SimpleCopyListing
(SimpleCopyListing.java:printStats(608)) - Paths (files+dirs) cnt = 243; dirCnt
= 58
2018-03-05 19:29:31,088 [Thread-217] INFO tools.SimpleCopyListing
(SimpleCopyListing.java:doBuildListing(402)) - Build file listing completed.
2018-03-05 19:29:31,109 [Thread-217] INFO tools.DistCp
(CopyListing.java:buildListing(94)) - Number of paths in the copy list: 243
2018-03-05 19:29:31,125 [Thread-217] INFO tools.DistCp
(CopyListing.java:buildListing(94)) - Number of paths in the copy list: 243
2018-03-05 19:29:31,142 [Thread-217] INFO mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(414)) - Listing completed in 0:00:05.938
2018-03-05 19:29:31,147 [Thread-217] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - op_delete += 1 -> 1
2018-03-05 19:29:31,147 [Thread-217] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - op_get_file_status += 1 ->
481
2018-03-05 19:29:31,147 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerGetFileStatus(2106)) - Getting path status for
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes
(distcp/hadoop-common-project/hadoop-auth/target/classes)
2018-03-05 19:29:31,147 [Thread-217] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 691
2018-03-05 19:29:31,165 [Thread-217] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 692
2018-03-05 19:29:31,198 [Thread-217] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_list_requests += 1
-> 260
2018-03-05 19:29:31,222 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:s3GetFileStatus(2229)) - Found path as directory (with /):
1/0
2018-03-05 19:29:31,222 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:s3GetFileStatus(2236)) - Prefix:
distcp/hadoop-common-project/hadoop-auth/target/classes/META-INF/
2018-03-05 19:29:31,222 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1732)) - Delete path
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes
- recursive true
2018-03-05 19:29:31,222 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1737)) - delete: Path is a directory:
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes
2018-03-05 19:29:31,222 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1759)) - Getting objects for directory prefix
distcp/hadoop-common-project/hadoop-auth/target/classes/ to delete
2018-03-05 19:29:31,222 [Thread-217] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_list_requests += 1
-> 261
2018-03-05 19:29:31,260 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/META-INF/LICENSE.txt
2018-03-05 19:29:31,260 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/META-INF/NOTICE.txt
2018-03-05 19:29:31,260 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticatedURL$1.class
2018-03-05 19:29:31,260 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticatedURL$AuthCookieHandler.class
2018-03-05 19:29:31,260 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticatedURL$Token.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticatedURL.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticationException.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/Authenticator.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/ConnectionConfigurator.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/KerberosAuthenticator$1.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/KerberosAuthenticator$KerberosConfiguration.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/PseudoAuthenticator.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AltKerberosAuthenticationHandler.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationFilter$1.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationFilter.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationHandler.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationHandlerUtil.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationToken.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/CompositeAuthenticationHandler.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/HttpConstants.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler$1.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler$2.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/LdapAuthenticationHandler$1.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/LdapAuthenticationHandler.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/MultiSchemeAuthenticationHandler.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/PseudoAuthenticationHandler.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/package-info.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/AuthToken.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/CertificateUtil.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosName$BadFormatString.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosName$NoMatchingRule.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosName$Rule.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosName.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosUtil$DER.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosUtil.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/RandomSignerSecretProvider.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/RolloverSignerSecretProvider$1.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/RolloverSignerSecretProvider.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/Signer.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/SignerException.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/SignerSecretProvider.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider$1.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider$JaasConfiguration.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider$SASLOwnerACLProvider.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerDelete(1769)) - Got object to delete
distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/util/PlatformName.class
2018-03-05 19:29:31,264 [Thread-217] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_delete_requests += 1
-> 1
2018-03-05 19:29:31,695 [Thread-217] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 693
2018-03-05 19:29:31,759 [Thread-217] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 694
2018-03-05 19:29:31,775 [Thread-217] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_list_requests += 1
-> 262
2018-03-05 19:29:31,799 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:s3GetFileStatus(2229)) - Found path as directory (with /):
0/1
2018-03-05 19:29:31,799 [Thread-217] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:s3GetFileStatus(2233)) - Summary:
distcp/hadoop-common-project/hadoop-auth/target/.plxarc 30
2018-03-05 19:29:31,799 [Thread-217] INFO mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(450)) - Deleted
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes
- missing at source
2018-03-05 19:29:31,800 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/META-INF
2018-03-05 19:29:31,800 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/META-INF/LICENSE.txt
2018-03-05 19:29:31,800 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/META-INF/NOTICE.txt
2018-03-05 19:29:31,800 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org
2018-03-05 19:29:31,800 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache
2018-03-05 19:29:31,800 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop
2018-03-05 19:29:31,800 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security
2018-03-05 19:29:31,800 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication
2018-03-05 19:29:31,800 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticatedURL$1.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticatedURL$AuthCookieHandler.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticatedURL$Token.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticatedURL.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticationException.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/Authenticator.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/ConnectionConfigurator.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/KerberosAuthenticator$1.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/KerberosAuthenticator$KerberosConfiguration.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/PseudoAuthenticator.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AltKerberosAuthenticationHandler.class
2018-03-05 19:29:31,802 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationFilter$1.class
2018-03-05 19:29:31,802 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationFilter.class
2018-03-05 19:29:31,802 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationHandler.class
2018-03-05 19:29:31,802 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationHandlerUtil.class
2018-03-05 19:29:31,802 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationToken.class
2018-03-05 19:29:31,802 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/CompositeAuthenticationHandler.class
2018-03-05 19:29:31,802 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/HttpConstants.class
2018-03-05 19:29:31,802 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.class
2018-03-05 19:29:31,802 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler$1.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler$2.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/LdapAuthenticationHandler$1.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/LdapAuthenticationHandler.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/MultiSchemeAuthenticationHandler.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/PseudoAuthenticationHandler.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/package-info.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/AuthToken.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/CertificateUtil.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosName$BadFormatString.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosName$NoMatchingRule.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosName$Rule.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosName.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosUtil$DER.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosUtil.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/RandomSignerSecretProvider.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/RolloverSignerSecretProvider$1.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/RolloverSignerSecretProvider.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/Signer.class
2018-03-05 19:29:31,805 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/SignerException.class
2018-03-05 19:29:31,805 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/SignerSecretProvider.class
2018-03-05 19:29:31,805 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider$1.class
2018-03-05 19:29:31,805 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider$JaasConfiguration.class
2018-03-05 19:29:31,805 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider$SASLOwnerACLProvider.class
2018-03-05 19:29:31,805 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.class
2018-03-05 19:29:31,805 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/util
2018-03-05 19:29:31,805 [Thread-217] DEBUG mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(485)) - Skipping deletion of
s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/util/PlatformName.class
2018-03-05 19:29:31,808 [Thread-217] INFO mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(497)) - Completed deletion of files from
S3AFileSystem{uri=s3a://hwdev-steve-london,
workingDir=s3a://hwdev-steve-london/user/stevel, inputPolicy=normal,
partSize=8388608, enableMultiObjectsDelete=true, maxKeys=5000,
readAhead=524288, blockSize=33554432, multiPartThreshold=2147483647,
serverSideEncryptionAlgorithm='NONE',
blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@20796163,
metastore=NullMetadataStore, authoritative=false, useListV1=false,
magicCommitter=true,
boundedExecutor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=25,
available=25, waiting=0}, activeCount=0},
unboundedExecutor=java.util.concurrent.ThreadPoolExecutor@2e373567[Running,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0],
statistics {0 bytes read, 0 bytes written, 544 read ops, 0 large read ops, 1
write ops}, metrics {{Context=s3aFileSystem}
{s3aFileSystemId=b0a84ec2-8ac0-4905-ade5-d608f1ea1e48}
{bucket=hwdev-steve-london} {stream_opened=0} {stream_close_operations=0}
{stream_closed=0} {stream_aborted=0} {stream_seek_operations=0}
{stream_read_exceptions=0} {stream_forward_seek_operations=0}
{stream_backward_seek_operations=0} {stream_bytes_skipped_on_seek=0}
{stream_bytes_backwards_on_seek=0} {stream_bytes_read=0}
{stream_read_operations=0} {stream_read_fully_operations=0}
{stream_read_operations_incomplete=0} {stream_bytes_read_in_close=0}
{stream_bytes_discarded_in_abort=0} {files_created=0} {files_copied=0}
{files_copied_bytes=0} {files_deleted=50} {fake_directories_deleted=0}
{directories_created=0} {directories_deleted=0} {ignored_errors=0}
{op_copy_from_local_file=0} {op_create=0} {op_create_non_recursive=0}
{op_delete=1} {op_exists=2} {op_get_file_status=481} {op_glob_status=3}
{op_is_directory=0} {op_is_file=0} {op_list_files=0} {op_list_located_status=0}
{op_list_status=61} {op_mkdirs=47} {op_open=0} {op_rename=0}
{object_copy_requests=0} {object_delete_requests=1} {object_list_requests=262}
{object_continue_list_requests=0} {object_metadata_requests=694}
{object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=0}
{object_put_requests_completed=0} {stream_write_failures=0}
{stream_write_block_uploads=0} {stream_write_block_uploads_committed=0}
{stream_write_block_uploads_aborted=0} {stream_write_total_time=0}
{stream_write_total_data=0} {committer_commits_created=0}
{committer_commits_completed=0} {committer_jobs_completed=0}
{committer_jobs_failed=0} {committer_tasks_completed=0}
{committer_tasks_failed=0} {committer_bytes_committed=0}
{committer_bytes_uploaded=0} {committer_commits_failed=0}
{committer_commits_aborted=0} {committer_commits_reverted=0}
{committer_magic_files_created=0} {s3guard_metadatastore_put_path_request=481}
{s3guard_metadatastore_initialization=0} {s3guard_metadatastore_retry=0}
{s3guard_metadatastore_throttled=0} {store_io_throttled=0}
{object_put_requests_active=0} {object_put_bytes_pending=0}
{stream_write_block_uploads_active=0} {stream_write_block_uploads_pending=0}
{stream_write_block_uploads_data_pending=0}
{S3guard_metadatastore_put_path_latencyNumOps=12}
{S3guard_metadatastore_put_path_latency50thPercentileLatency=1621}
{S3guard_metadatastore_put_path_latency75thPercentileLatency=1964}
{S3guard_metadatastore_put_path_latency90thPercentileLatency=2095}
{S3guard_metadatastore_put_path_latency95thPercentileLatency=2287}
{S3guard_metadatastore_put_path_latency99thPercentileLatency=2287}
{S3guard_metadatastore_throttle_rateNumEvents=0}
{S3guard_metadatastore_throttle_rate50thPercentileFrequency (Hz)=0}
{S3guard_metadatastore_throttle_rate75thPercentileFrequency (Hz)=0}
{S3guard_metadatastore_throttle_rate90thPercentileFrequency (Hz)=0}
{S3guard_metadatastore_throttle_rate95thPercentileFrequency (Hz)=0}
{S3guard_metadatastore_throttle_rate99thPercentileFrequency (Hz)=0} }}
2018-03-05 19:29:31,808 [Thread-217] INFO mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(504)) - Deleted from target: files: 0
directories: 1; skipped 60
2018-03-05 19:29:31,808 [Thread-217] INFO mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(507)) - Duration of deletions: 0:00:00.666
2018-03-05 19:29:31,808 [Thread-217] INFO mapred.CopyCommitter
(CopyCommitter.java:deleteMissing(509)) - Total duration of operation:
0:00:06.604
2018-03-05 19:29:31,809 [Thread-217] INFO mapred.CopyCommitter
(CopyCommitter.java:cleanup(185)) - Cleaning up temporary work folder:
file:/tmp/hadoop/mapred/staging/stevel1049724566/.staging/_distcp2007172673
2018-03-05 19:29:32,289 [main] INFO mapreduce.Job
(Job.java:monitorAndPrintJob(1658)) - Job job_local1086689566_0001 completed
successfully
2018-03-05 19:29:32,297 [main] INFO mapreduce.Job
(Job.java:monitorAndPrintJob(1665)) - Counters: 24
File System Counters
FILE: Number of bytes read=1067879
FILE: Number of bytes written=1847318
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
S3A: Number of bytes read=0
S3A: Number of bytes written=0
S3A: Number of read operations=731
S3A: Number of large read operations=0
S3A: Number of write operations=0
Map-Reduce Framework
Map input records=182
Map output records=135
Input split bytes=306
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=10
Total committed heap usage (bytes)=839909376
File Input Format Counters
Bytes Read=64532
File Output Format Counters
Bytes Written=22806
DistCp Counters
Bandwidth in Btyes=0
Bytes Skipped=1493640
DIR_COPY=47
Files Skipped=135
{code}
note the S3 stats declres 731 read ops.; a {{) find . -print | wc -l }} gives
the files of hadoop auth src & target as 244, so there is still ~3x reads for
every source files, even with delete of child entries being skipped (which
happens at the end of the job), and src checks bypassed.
> DistCp to eliminate needless deletion of files under already-deleted
> directories
> --------------------------------------------------------------------------------
>
> Key: HADOOP-15209
> URL: https://issues.apache.org/jira/browse/HADOOP-15209
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Attachments: HADOOP-15209-001.patch, HADOOP-15209-002.patch,
> HADOOP-15209-003.patch, HADOOP-15209-004.patch, HADOOP-15209-005.patch
>
>
> DistCP issues a delete(file) request even if is underneath an already deleted
> directory. This generates needless load on filesystems/object stores, and, if
> the store throttles delete, can dramatically slow down the delete operation.
> If the distcp delete operation can build a history of deleted directories,
> then it will know when it does not need to issue those deletes.
> Care is needed here to make sure that whatever structure is created does not
> overload the heap of the process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
