gmhelmold opened a new pull request, #2820: URL: https://github.com/apache/iceberg-rust/pull/2820
## What `IcebergMetadataTableProvider::scan()` ignored the `projection` argument: `IcebergMetadataScan` always reported the full metadata-table schema and emitted unprojected batches. Closes #2819. ## Why On any populated metadata table, aggregate queries crash and projected queries silently return every column: ```sql INSERT INTO default.default.test_partitioned_table VALUES (1, 'a', 'b'); SELECT count(*) FROM default.default.test_partitioned_table$snapshots; -- Internal error: Physical input schema should be the same as the one converted from -- logical input schema. Differences: - Different number of fields: (physical) 6 vs (logical) 0. SELECT operation FROM default.default.test_partitioned_table$snapshots; -- returns all 6 snapshot columns ``` Same class as #828, which fixed this for `IcebergTableScan`; the metadata provider path was left behind because existing tests only exercise empty metadata tables. ## Fix `IcebergMetadataScan` is now projection-aware, mirroring `IcebergTableScan`: it carries the projection indices, reports the projected schema in its `PlanProperties`, and projects each emitted `RecordBatch`. `scan()` passes the projection through. Out of scope (behavior unchanged, correctness preserved because DataFusion applies both on top of the scan): `filters` never reach `scan()` (`supports_filters_pushdown` is not overridden), and `limit` remains a missed optimization only. `EXPLAIN` output for the metadata scan does not yet render the projection (pre-existing) — happy to add that in a follow-up if wanted. ## Tests - New `test_metadata_table_projection` integration test: aggregate on a populated `$snapshots` (`count(*)` — the empty-projection edge) plus a single-column projection. - New sqllogictest schedule `df_inspect_snapshots_manifests`: first populated-metadata-table SLT coverage (deterministic columns only), doubling as regression coverage for both failure modes. - Red→green: reverting the two source files to `main` fails the new SLT with the unprojected 12-column `$manifests` row; with the fix the full sqllogictest suite passes (run twice to confirm determinism). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
