anuragmantri opened a new issue, #3422: URL: https://github.com/apache/iceberg-rust/issues/3422
### Is your feature request related to a problem or challenge? `ManifestEntryContext::into_file_scan_task` drops the data file's column statistics when it builds a `FileScanTask`. The task has no lower or upper bounds and no value, null or NaN counts, so an engine reading the tasks cannot reason about a file's key range without going back to the manifests. Sorted scans need this. apache/datafusion-iceberg#55 keeps a sorted scan's order by merging one reader per file, which opens every file at once. With per-file bounds on the sort columns it could read files whose key ranges don't overlap one after another, and only merge where they do. apache/datafusion-comet#5343 needs the same information. This is the "Add opt-in file statistics retention to scan tasks" item in #3343. In #607 there was pushback on putting statistics on every task, because it makes serialized tasks larger when they are shipped to workers and because it could expose `key_metadata`. So this has to be opt-in, and it should add nothing when it isn't used. ### Describe the solution you'd like Add `TableScanBuilder::with_column_stats(columns)`, mirroring Java's `Scan.includeColumnStats(Collection<String>)`. The names resolve to field ids when the scan is built, using the scan's case sensitivity, and an unknown name fails the way `select` does. Each planned `FileScanTask` would then expose `column_stats()`, returning a new `FileScanTaskColumnStats` that holds the five maps (value counts, null value counts, NaN value counts, lower bounds and upper bounds) filtered to the requested field ids. The maps would match the shape of `FileMetrics` in #3347 so the two convert directly. The docs would state what the statistics mean: bounds are inclusive and may be truncated, any of them can be missing, they describe the whole file without adjusting for deletes, filters or splits, and they are typed with the schema the manifest was written with. When nothing is requested, `column_stats()` is `None`, planning copies nothing, and serialization leaves the field out, so serialized tasks stay byte-identical to today. No other `DataFile` fields, including `key_metadata`, are exposed. ### Willingness to contribute I can contribute to this feature independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
