venkata91 commented on code in PR #17820:
URL: https://github.com/apache/iceberg/pull/17820#discussion_r4569043347
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java:
##########
@@ -285,20 +287,24 @@ private void abort(WriterCommitMessage[] messages) {
private DataFileSet files(WriterCommitMessage[] messages) {
DataFileSet files = DataFileSet.create();
+ // every commit path funnels through here, so this is where task write
times are collected
+ WriteDurations durations = WriteDurations.EMPTY;
for (WriterCommitMessage message : messages) {
if (message != null) {
TaskCommit taskCommit = (TaskCommit) message;
files.addAll(Arrays.asList(taskCommit.files()));
+ durations = durations.combine(taskCommit.writeDurations());
Review Comment:
the main use case for us is the stage-level breakdown, not per-task
attribution. for a write stage we want to know how much of the time goes to
shuffle/sort vs actually writing to object storage (ABFS in our case), so we
know what to tune: shuffle partitions and sort on one side, file sizes, write
parallelism and storage settings on the other. today the stage only shows task
duration, which mixes all of that together. with the totals, in your example we
can see 35 s of the 50 s task time is the Iceberg write + close, so it's the
storage write that needs tuning, not the sort.
agree per-task would be nicer, but since close() can't be a task metric in
4.1, I think the totals are still useful for this. I'll run WritersBenchmark
with and without the timer and share the numbers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]