dshogan-amzn opened a new issue, #19002:
URL: https://github.com/apache/iceberg/issues/19002

   ### Apache Iceberg version
   
   1.12.0 (latest release)
   
   ### Query engine
   
   None
   
   ### Please describe the bug 🐞
   
   For an `IN` predicate on a column that is partitioned by `bucket`, partition 
projection correctly limits a scan to the matching bucket values. File metrics 
evaluation then compares **every** literal in the `IN` list against each 
remaining file's column bounds. A literal that hashes to a different bucket can 
never appear in the file, but if it falls inside the file's lower and upper 
bounds, the file is kept.
   
   Small example, table partitioned by `bucket(16, id)` (bucket assignments are 
illustrative; the tests in #18933 compute real ones):
   
   - File `F` is in bucket 7. Its `id` bounds are `[100, 400]`, and its rows 
only contain values that hash to bucket 7.
   - Query: `id IN (5000, 250)`, where `5000` hashes to bucket 7 and `250` 
hashes to bucket 3.
   - Partition projection keeps `F`, because `5000` maps to bucket 7.
   - Metrics evaluation keeps `F`, because `250` is inside `[100, 400]`.
   - `F` cannot contain either value: `5000` is outside its bounds, and `250` 
belongs to bucket 3.
   
   The effect grows with the size of the `IN` list. With many keys spread over 
many buckets, almost every file in a selected bucket has some foreign-bucket 
literal inside its bounds, so metrics pruning stops working for those files. 
Lists with more than 200 values also skip bounds evaluation entirely 
(`IN_PREDICATE_LIMIT`), even when only a few of those values belong to the 
file's bucket.
   
   The result is correct, because the extra files are filtered out when they 
are read. The cost is extra scan tasks and data read for queries with large 
`IN` lists on bucketed keys, which is a common pattern for keyed lookups and 
joins rewritten into `IN` filters.
   
   ### Possible fix
   
   When the evaluator knows the file's partition spec, it can index the `IN` 
literals by their transformed partition value once, then compare only the 
literals that map to the file's partition value. This never removes a file that 
could contain a match. A draft PR: #18933.
   
   
   ### Willingness to contribute
   
   - [x] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from 
the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to