alexanderbianchi commented on issue #6737:
URL:
https://github.com/apache/datafusion-comet/issues/6737#issuecomment-6114821853
I've done some preliminary investigation here. Sail has its own Iceberg
integration that follows a similar native-DataFusion approach:
https://github.com/lakehq/sail/tree/main/crates/sail-iceberg
The main attraction for me is directly reusing DataFusion's Parquet
configuration and optimization machinery, rather than maintaining parallel
implementations.
For example, DataFusion already has CachedParquetFileReaderFactory for
caching parsed footer and page metadata:
https://docs.rs/datafusion/latest/datafusion/datasource/physical_plan/parquet/struct.CachedParquetFileReaderFactory.html
For supported data-file filters, including dynamic filters, a native
Parquet path could also evaluate DataFusion physical expressions without
translating them into Iceberg predicates. Manifest and partition pruning would
remain a separate concern.
We've already migrated IcebergTableScan to IcebergDataSource under
DataSourceExec: https://github.com/apache/datafusion-iceberg/pull/50. That
gives filters a standard place to land, execution still goes through
iceberg-rust's ArrowReader.
I see two possible directions:
#### 1. Expose more Iceberg read preparation and use DataFusion's Parquet
scan machinery
Iceberg metadata/file planning could still use iceberg-rust, but data-file
execution would no longer go through ArrowReader.
The challenge is implementing Iceberg read semantics around the native
Parquet scans: field-ID/schema evolution, equality and position deletes, delete
vectors, and row lineage. That could involve exposing reusable helpers from
iceberg-rust or implementing those pieces in the integration.
We would need to preserve per-file metadata and original row positions
through pruning and filtering, rather than treating everything as an arbitrary
stream of already-filtered batches.
#### 2. Keep ArrowReader and introduce narrower extension points
An injectable Parquet AsyncFileReader factory could let integrations
reuse metadata caching and custom I/O while retaining Iceberg's existing schema
and delete handling.
This would be analogous in spirit to the Storage abstraction:
https://github.com/apache/iceberg-rust/issues/2051
That factory alone would not reuse DataFusion's filter evaluation or
pruning. A separate, higher-level hook would be needed to supply engine-built
row filters and pruning selections, with enough Iceberg schema context to
compose them safely with deletes.
Neither direction is trivial, and the narrower hooks might be useful
independently of a full native-Parquet integration.
FYI Thanks for the work on execution-time adaptive pruning. DataFusion
Distributed also wants dynamic filtering for Iceberg, and we're interested in
building on https://github.com/apache/iceberg-rust/issues/3343
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
