steveloughran commented on a change in pull request #1442: HADOOP-16570. S3A
committers encounter scale issues
URL: https://github.com/apache/hadoop/pull/1442#discussion_r330183089
##########
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java
##########
@@ -349,16 +368,9 @@ public void recoverTask(TaskAttemptContext taskContext)
throws IOException {
* @throws IOException IO failure
*/
protected void maybeCreateSuccessMarkerFromCommits(JobContext context,
- List<SinglePendingCommit> pending) throws IOException {
+ ActiveCommit pending) throws IOException {
List<String> filenames = new ArrayList<>(pending.size());
- for (SinglePendingCommit commit : pending) {
- String key = commit.getDestinationKey();
- if (!key.startsWith("/")) {
- // fix up so that FS.makeQualified() sets up the path OK
- key = "/" + key;
- }
- filenames.add(key);
- }
+ filenames.addAll(pending.committedObjects);
Review comment:
we stop adding files to that pending.committed objects list once we reach an
arbitrary threshold (SUCCESS_MARKER_FILE_LIMIT == 100), so we add that subset
of entries. I'll clarify that in the comments
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
