mehakmeet opened a new pull request, #5308: URL: https://github.com/apache/hadoop/pull/5308
### Description of PR Using modification time as a way to add more checks to determine if distcp -update should skip a file or not. In specific cases like the same file name, and size but different content we used to incorrectly skip files in update since there is no checksum comparison between object stores with different algorithm for it, to mitigate this we introduce comparing modification time between the target file and the source. ### How was this patch tested? Manually tested on an environment after reproducing the scenario where we might incorrectly skip a file. Added a test in `AbstractContractDistCpTest.java` to test by changing the target file's modification time to emulate the scenario. Tested on S3A(ap-south-1), ABFS(us-west-2), and LocalFS, and the test was successful. ### For code changes: - [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [X] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
