CTTY commented on issue #1382: URL: https://github.com/apache/iceberg-rust/issues/1382#issuecomment-3130404115
Hi @stevie9868 , Thanks for being willing to work on this! However, I don't think we need a new trait like `PartitionFIleWriter` to achieve partition-aware writing. Some of my thoughts: - For the datafusion integration, we should have a repartition node (#1543) to split the incoming data by their Iceberg partition values, so each datafusion partition will only need to handle data from a certain iceberg partition and partition-aware writer won't be necessary in this case - For the more general case, my rough idea now is to borrow/reimplement some logic from the repartition node --- to split the incoming data by their iceberg partitions. And then you can use the inner writers to handle each partition's data individually. The inner writer here also won't need to be partition-aware. Basically, `PartitionFileWriter` can just be a decorator just like `RollingFileWriter`, and don't need to be a trait. The real problem would be #1543 or the data partitioning abstraction I'll let @liurenjie1024 to comment his thoughts as well -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
