cnauroth commented on code in PR #7693:
URL: https://github.com/apache/hadoop/pull/7693#discussion_r2098374750
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md:
##########
@@ -558,6 +558,7 @@ The table below provides a summary of each option.
| `fs.s3a.committer.threads` | Number of threads in committers for parallel
operations on files.| -4 |
| `fs.s3a.committer.generate.uuid` | Generate a Job UUID if none is passed
down from Spark | `false` |
| `fs.s3a.committer.require.uuid` |Require the Job UUID to be passed down from
Spark | `false` |
+| `fs.s3a.committer.magic.cleanup.enabled` | Cleanup the magic path after the
job is committed. | `true` |
Review Comment:
I suggest also adding a new paragraph at the bottom of the Troubleshooting
section. We can discuss slow cleanups and why you might want to use this option
(and the importance of replacing it with lifecycle rules to do the delete
asynchronously).
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTrackerUtils.java:
##########
@@ -61,4 +61,16 @@ public static boolean
isTrackMagicCommitsInMemoryEnabled(Configuration conf) {
CommitConstants.FS_S3A_COMMITTER_MAGIC_TRACK_COMMITS_IN_MEMORY_ENABLED,
CommitConstants.FS_S3A_COMMITTER_MAGIC_TRACK_COMMITS_IN_MEMORY_ENABLED_DEFAULT);
}
+
+ /**
+ * Is cleanup of magic committer staging dirs enabled.
Review Comment:
Nitpick: please remove extra space between dirs and enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
