anuragmantri opened a new pull request, #3424: URL: https://github.com/apache/iceberg-rust/pull/3424
## Which issue does this PR close? - Closes #3422. - Part of #3343. - Related to #3126. ## What changes are included in this PR? Planning drops a data file's column statistics when it builds a `FileScanTask`. Engines that want to order or merge files by key range, such as the sorted scan work in apache/datafusion-iceberg#55 and apache/datafusion-comet#5343, need per-file bounds on the sort columns. This PR lets a scan keep them, mirroring Java's `Scan.includeColumnStats`. `TableScanBuilder::with_column_stats(columns)` takes column names and resolves them to field ids in `build()` with the scan's case sensitivity. An unknown name fails the same way `select` does, as does a name that matches more than one field when the scan is case-insensitive. Each planned task then carries a new `FileScanTaskColumnStats` with the file's value, null and NaN counts and lower and upper bounds for only those field ids. The maps have the same names and types as the crate-internal `FileMetrics` in #3347, so the two convert directly. The feature is opt-in, following the concerns raised in #607. By default `FileScanTask::column_stats()` is `None`, planning copies nothing from the manifest entry, and serialization leaves the field out, so tasks without stats serialize exactly as they do today. Nothing from `key_metadata` or other `DataFile` fields is exposed. The docs spell out what the statistics mean. Bounds are inclusive and may be truncated. Any of them can be missing. They describe the whole file, with no adjustment for deletes, the filter or splits. They are typed with the schema the manifest was written with, so after type promotion a bound can be an `int` on a `long` column. Public API additions are `TableScanBuilder::with_column_stats`, `FileScanTask::column_stats` and the `FileScanTaskColumnStats` struct with its getters and builder. ## Are these changes tested? Yes. A new fixture writes manifest entries with statistics on several columns, where one file has no statistics for `x`. The scan test checks that the default scan and an empty request return no stats, that requesting `x` and `dbl` keeps only those entries in every map while `x` stays missing for the second file, that case-insensitive resolution works, and that unknown names fail in both case modes. The serde round trip with all optional fields now includes string and decimal bounds, and a new test checks that a task without stats serializes no `column_stats` key and reads back as `None`. I also mutated the filtering, the default, case handling, unknown-name handling and serialization, and each change made a test fail. Reviewers may want to look at two choices. Nested field names are accepted, unlike `select`, because manifests record stats for nested leaves. An empty column list is treated as not requested. ## AI Disclosure I used Claude Code (Claude Opus 5.5) to write this change and its tests, and I reviewed it manually. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
