dshogan-amzn opened a new issue, #19002: URL: https://github.com/apache/iceberg/issues/19002
### Apache Iceberg version 1.12.0 (latest release) ### Query engine None ### Please describe the bug 🐞 For an `IN` predicate on a column that is partitioned by `bucket`, partition projection correctly limits a scan to the matching bucket values. File metrics evaluation then compares **every** literal in the `IN` list against each remaining file's column bounds. A literal that hashes to a different bucket can never appear in the file, but if it falls inside the file's lower and upper bounds, the file is kept. Small example, table partitioned by `bucket(16, id)` (bucket assignments are illustrative; the tests in #18933 compute real ones): - File `F` is in bucket 7. Its `id` bounds are `[100, 400]`, and its rows only contain values that hash to bucket 7. - Query: `id IN (5000, 250)`, where `5000` hashes to bucket 7 and `250` hashes to bucket 3. - Partition projection keeps `F`, because `5000` maps to bucket 7. - Metrics evaluation keeps `F`, because `250` is inside `[100, 400]`. - `F` cannot contain either value: `5000` is outside its bounds, and `250` belongs to bucket 3. The effect grows with the size of the `IN` list. With many keys spread over many buckets, almost every file in a selected bucket has some foreign-bucket literal inside its bounds, so metrics pruning stops working for those files. Lists with more than 200 values also skip bounds evaluation entirely (`IN_PREDICATE_LIMIT`), even when only a few of those values belong to the file's bucket. The result is correct, because the extra files are filtered out when they are read. The cost is extra scan tasks and data read for queries with large `IN` lists on bucketed keys, which is a common pattern for keyed lookups and joins rewritten into `IN` filters. ### Possible fix When the evaluator knows the file's partition spec, it can index the `IN` literals by their transformed partition value once, then compare only the literals that map to the file's partition value. This never removes a file that could contain a match. A draft PR: #18933. ### Willingness to contribute - [x] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
