venkata91 commented on code in PR #17820:
URL: https://github.com/apache/iceberg/pull/17820#discussion_r4569043347


##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java:
##########
@@ -285,20 +287,24 @@ private void abort(WriterCommitMessage[] messages) {
 
   private DataFileSet files(WriterCommitMessage[] messages) {
     DataFileSet files = DataFileSet.create();
+    // every commit path funnels through here, so this is where task write 
times are collected
+    WriteDurations durations = WriteDurations.EMPTY;
 
     for (WriterCommitMessage message : messages) {
       if (message != null) {
         TaskCommit taskCommit = (TaskCommit) message;
         files.addAll(Arrays.asList(taskCommit.files()));
+        durations = durations.combine(taskCommit.writeDurations());

Review Comment:
   the main use case for us is the stage-level breakdown, not per-task 
attribution. for a write stage we want to know how much of the time goes to 
shuffle/sort vs actually writing to object storage (ABFS in our case), so we 
know what to tune: shuffle partitions and sort on one side, file sizes, write 
parallelism and storage settings on the other. today the stage only shows task 
duration, which mixes all of that together. with the totals, in your example we 
can see 35 s of the 50 s task time is the Iceberg write + close, so it's the 
storage write that needs tuning, not the sort.
   
   agree per-task would be nicer, but since close() can't be a task metric in 
4.1, I think the totals are still useful for this. I'll run WritersBenchmark 
with and without the timer and share the numbers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to