gmhelmold opened a new issue, #2819: URL: https://github.com/apache/iceberg-rust/issues/2819
### Apache Iceberg Rust version main (`d1b8eac`) ### Describe the bug The datafusion integration's metadata table provider ignores the `projection` argument in `TableProvider::scan()` (`crates/integrations/datafusion/src/table/metadata_table.rs`): it always returns an `IcebergMetadataScan` that reports the full metadata-table schema and emits unprojected batches. (`limit` is also unused, but that's only a missed optimization; filters never reach `scan()` since `supports_filters_pushdown` isn't overridden — projection is the correctness issue.) This goes unnoticed because existing tests only query *empty* metadata tables with `SELECT *` (`test_metadata_table` in `integration_datafusion_test.rs`; the `$snapshots` checks in `insert_into.slt` run before any snapshot exists). Same class as #828, which fixed this for `IcebergTableScan`. ### To Reproduce Any populated metadata table — e.g. against the seeded table in the `crates/sqllogictest` harness: ```sql INSERT INTO default.default.test_partitioned_table VALUES (1, 'a', 'b'); SELECT count(*) FROM default.default.test_partitioned_table$snapshots; -- DataFusion error: Internal error: Physical input schema should be the same as the -- one converted from logical input schema. Differences: -- - Different number of fields: (physical) 6 vs (logical) 0. SELECT operation FROM default.default.test_partitioned_table$snapshots; -- returns all 6 snapshot columns instead of just `operation` ``` ### Expected behavior `scan()` honors `projection`: make `IcebergMetadataScan` projection-aware (report the projected schema in its plan properties and project the emitted batches), mirroring how `IcebergTableScan` handles projection. ### Willingness to contribute Yes — happy to submit the fix with sqllogictest regression coverage on populated metadata tables. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
