rdblue opened a new pull request, #18109: URL: https://github.com/apache/iceberg/pull/18109
This PR replaces #17433 and incorporates the review feedback. This was originally a PR to update Eduard's branch, but there were substantial rebase changes after #18021 was merged so it was easier to open a new PR. This PR updates `V4ManifestReader` to support reading content stats: * When no projection is requested, the table's current manifest schema is used to read * When `forScanPlanning` is enabled, it uses `statsReadSchema` to read stats for filter fields and requested stats fields * When `select` is called, the requested schema is produced by selecting columns by name from the table's current manifest schema * When `project` is called, the requested schema is used as-is The reader supports requesting stats fields by ID: * When stats fields are requested, only the requested stats are returned but additional stats may be read and used for filtering * `project` and `select` always return the requested projection and stats fields are always copied. `projectStats` is not allowed to avoid a behavior conflict with `project` and `select` * When `forScanPlanning` is enabled, stats are only returned if requested to match the existing behavior of scan planning * When no projection is requested and `projectStats` is not called, all stats are returned This also does not yet implement content stats filtering, which will be added in the next manifest reader PR. This PR also refactors tests to validate all of the written fields that are read, including the partition tuple and content stats. Tests now use a comparator that checks the written `TrackedFile` against the one that is read. The `Tracking` tuple where inherited values are kept is currently excluded. The inheritance tests will validate the `Tracking` tuple in a PR soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
