huaxingao commented on code in PR #17820:
URL: https://github.com/apache/iceberg/pull/17820#discussion_r4568197852


##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java:
##########
@@ -285,20 +287,24 @@ private void abort(WriterCommitMessage[] messages) {
 
   private DataFileSet files(WriterCommitMessage[] messages) {
     DataFileSet files = DataFileSet.create();
+    // every commit path funnels through here, so this is where task write 
times are collected
+    WriteDurations durations = WriteDurations.EMPTY;
 
     for (WriterCommitMessage message : messages) {
       if (message != null) {
         TaskCommit taskCommit = (TaskCommit) message;
         files.addAll(Arrays.asList(taskCommit.files()));
+        durations = durations.combine(taskCommit.writeDurations());

Review Comment:
   Thanks for looking into the options. Before choosing one, I'd like to step 
back on the motivation.
   
   Spark already shows per-task duration (min/med/max). For example, with 3 
tasks:
   
   | Task | Sort (not Iceberg) | Iceberg write() | Iceberg close() | Task 
duration |
   
|------|--------------------|-----------------|-----------------|---------------|
   | 1    | 5 s                | 2 s             | 3 s             | 10 s       
   |
   | 2    | 5 s                | 2 s             | 3 s             | 10 s       
   |
   | 3    | 5 s                | 16 s            | 9 s             | 30 s       
   |
   
   Today Spark shows 10 s / 10 s / 30 s, so we can see task 3 is slow, but not 
whether Iceberg is the cause.
   
   With this PR, we get writeDuration = 20 s and writeCloseDuration = 15 s for 
the whole job. That doesn't tell us which task the time came from, so we still 
can't tell that task 3 is slow because of the Iceberg write.
   
   Option 2 would show 2 s / 2 s / 16 s for write(), but close() would still be 
a total, and as you mentioned, for small files most of the time is in close().
   
   What would a user do with these these metrics that they can't do today?
   
   Also, write() is called once per row, so this adds two System.nanoTime() 
calls and possible a lambda per row on the write path. Could you run one of the 
existing write benchmarks (e.g. WritersBenchmark) with and without the timer? I 
wanted to see whether the overhead is visible.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to