huaxingao commented on code in PR #17820:
URL: https://github.com/apache/iceberg/pull/17820#discussion_r4568197852
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java:
##########
@@ -285,20 +287,24 @@ private void abort(WriterCommitMessage[] messages) {
private DataFileSet files(WriterCommitMessage[] messages) {
DataFileSet files = DataFileSet.create();
+ // every commit path funnels through here, so this is where task write
times are collected
+ WriteDurations durations = WriteDurations.EMPTY;
for (WriterCommitMessage message : messages) {
if (message != null) {
TaskCommit taskCommit = (TaskCommit) message;
files.addAll(Arrays.asList(taskCommit.files()));
+ durations = durations.combine(taskCommit.writeDurations());
Review Comment:
Thanks for looking into the options. Before choosing one, I'd like to step
back on the motivation.
Spark already shows per-task duration (min/med/max). For example, with 3
tasks:
| Task | Sort (not Iceberg) | Iceberg write() | Iceberg close() | Task
duration |
|------|--------------------|-----------------|-----------------|---------------|
| 1 | 5 s | 2 s | 3 s | 10 s
|
| 2 | 5 s | 2 s | 3 s | 10 s
|
| 3 | 5 s | 16 s | 9 s | 30 s
|
Today Spark shows 10 s / 10 s / 30 s, so we can see task 3 is slow, but not
whether Iceberg is the cause.
With this PR, we get writeDuration = 20 s and writeCloseDuration = 15 s for
the whole job. That doesn't tell us which task the time came from, so we still
can't tell that task 3 is slow because of the Iceberg write.
Option 2 would show 2 s / 2 s / 16 s for write(), but close() would still be
a total, and as you mentioned, for small files most of the time is in close().
What would a user do with these these metrics that they can't do today?
Also, write() is called once per row, so this adds two System.nanoTime()
calls and possible a lambda per row on the write path. Could you run one of the
existing write benchmarks (e.g. WritersBenchmark) with and without the timer? I
wanted to see whether the overhead is visible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]