Tpt commented on code in PR #24924:
URL: https://github.com/apache/datafusion/pull/24924#discussion_r4029157814
##########
datafusion/datasource/src/write/demux.rs:
##########
@@ -224,6 +237,7 @@ async fn row_count_demuxer(
part_idx += 1;
}
row_counts[next_send_steam] += rb.num_rows();
+ bytes_counts[next_send_steam] += rb.get_array_memory_size();
Review Comment:
Sorry for the slow reply, got trapped by other things.
This is a great point. Done. I used a `Arc<AtomicUsize>` to record and
report back the output size without blocking the code. An other approach might
be to have an other channel giving back the written size but:
- either the demuxer block on it and we lose must of the concurrency and
pipelining advantage (I guess very much a no go)
- we don't block and we get the same behavior of `Arc<AtomicUsize>` with
more overhead and code
The downside of `Arc<AtomicUsize>` is indeed that the reporting might lag
quite a lot. I have updated the documentation to highlight this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
