CTTY commented on issue #1540: URL: https://github.com/apache/iceberg-rust/issues/1540#issuecomment-3110852311
Hi @ZENOTME , > how write node and commit node Interact in this path Write node will need to serialize `Vec<DataFile>` and send it to commit node in a stream, and commit node will deserialize it. My draft [here](https://github.com/apache/iceberg-rust/pull/1511) probably will make more sense to me explaining in text:) I've discussed with @liurenjie1024 offline over the DataSink trait before, and we are not sure about some design details in `DataSink`: - The repartitioning/demuxing and the following writing process in DataSinkExec will be done on a single node using multiple threads ([link](https://github.com/apache/datafusion/blob/4084894ebe1889b6ce80f3a207453154de274b03/datafusion/datasource/src/file_sink_config.rs#L99)) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
