toutane commented on issue #2220: URL: https://github.com/apache/iceberg-rust/issues/2220#issuecomment-4750054372
Following up on this issue and the earlier POC in #2298, I opened a smaller PR focused on the first scoped step: #2671 This PR adds an opt-in eager scan planning path for the DataFusion integration. When `iceberg.enable_eager_scan_planning` is enabled, `IcebergTableProvider::scan()` plans `FileScanTask`s during physical planning, groups them across DataFusion `target_partitions`, and exposes the resulting output partition count as `UnknownPartitioning(N)`. Each `execute(partition)` call then reads only its assigned task group through `ArrowReaderBuilder`. Compared with the broader work discussed in #2298, this PR intentionally keeps the scope limited to file-level scan parallelism and output partition count reporting. It does not include hash partitioning, size-aware bin-packing, or row-group/sub-file planning. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
