venkata91 commented on code in PR #17820:
URL: https://github.com/apache/iceberg/pull/17820#discussion_r4580698335


##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java:
##########
@@ -285,20 +287,24 @@ private void abort(WriterCommitMessage[] messages) {
 
   private DataFileSet files(WriterCommitMessage[] messages) {
     DataFileSet files = DataFileSet.create();
+    // every commit path funnels through here, so this is where task write 
times are collected
+    WriteDurations durations = WriteDurations.EMPTY;
 
     for (WriterCommitMessage message : messages) {
       if (message != null) {
         TaskCommit taskCommit = (TaskCommit) message;
         files.addAll(Arrays.asList(taskCommit.files()));
+        durations = durations.combine(taskCommit.writeDurations());

Review Comment:
   I took out the per-row timing from `write()` everywhere and kept the close 
timing.
   
   Instead, `TimedFileIO` goes into `OutputFileFactory` through `ioSupplier` 
and times the storage calls (create, write, flush, close) on each output 
stream, so the clock is read per stream call, not per row. It's scoped to 
`HadoopFileIO`: for a `HadoopOutputFile` it wraps the file's `FileSystem`, 
because Parquet and ORC write through the `FileSystem` and not 
`OutputFile.create()`. That covers Parquet, ORC, Avro and DV files there. With 
other FileIOs the files are written untimed and the storage metrics read 0. 
Parquet also looked the `FileSystem` up again from the path, which I fixed 
separately in #18947. This PR carries that commit until it merges.
   
   Spark reads task metrics before `commit()`, which is where files get closed 
and most uploads finish, so the old `total (min, med, max)` string really 
showed `total (0, 0, total)`. Now the writer is wrapped to send its times in 
the commit message, and the driver reports total storage write time plus the 
min, median and max across tasks as separate metrics. Each task also logs its 
storage time and bytes at INFO, so you can find the slow one.
   
   On a local JMH (SparkWrite unpartitioned, 2.5M rows, 12 interleaved rounds 
against no timer) it's +2.2% on average with an SE of 2.3%, and half the rounds 
came out faster with the timer. Allocation per row is the same. That write 
makes about 29K timed calls, so I'd expect around 1.4 ms of clock reads in a 
3.7 s write, and I think the rest is noise on that box. The per-row version was 
4% to 12% slower.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to