Shahzeb-DataEngineer opened a new issue, #18988:
URL: https://github.com/apache/iceberg/issues/18988

   ### Apache Iceberg version
   
   None
   
   ### Query engine
   
   None
   
   ### Please describe the bug 🐞
   
   ## Description
   
   With a `decimal(9,2)` column partitioned by `bucket(value, 16)`, the query 
`WHERE value = 100.0` silently misses rows holding `100.00`, because the 
literal keeps its written scale when bound but the bucket hash is 
scale-sensitive.
   
   `Literals.DecimalLiteral.to(DECIMAL)` and `StringLiteral.to(DECIMAL)` both 
say "do not change decimal scale" and return the literal unchanged, while 
`IntegerLiteral`/`LongLiteral`/`FloatLiteral`/`DoubleLiteral.to(DECIMAL)` all 
`setScale(column scale, HALF_UP)`. `Binder` converts literals via 
`Literal.to(type)`, so a decimal literal keeps its written scale. But 
`BucketUtil.hash(BigDecimal)` hashes the raw unscaled-value bytes, so 
numerically equal decimals at different scales hash differently.
   
   The spec (format/spec.md) says hash results are "not dependent on decimal 
scale, which is part of the type, not the data value". That holds for stored 
data, where the column scale is fixed, but not for un-normalized query literals.
   
   ## To Reproduce
   
   Schema `decimal(9,2)`, partition spec `bucket(value, 16)`:
   
   - `value = 100.00` projects to `value_bucket == 8`
   - `value = 100.0` projects to `value_bucket == 0`
   
   A manifest whose stored partition value is 8 is then pruned by 
`ManifestEvaluator.eq` (lower == upper == 8 vs projected 0 returns 
ROWS_CANNOT_MATCH), so the rows are silently missing from `WHERE value = 
100.0`. The same flaw reaches IN lists via `Bucket.project` -> 
`ProjectionUtil.transformSet`.
   
   Same family at row level: binding `IN(1.1, 2.2)` on `decimal(9,2)`, then 
`BoundSetPredicate.test(1.100)` returns false, because `Evaluator.in` / 
`BoundSetPredicate.test` use `Set.contains` on raw BigDecimals and 
`BigDecimal.equals`/`hashCode` are scale-sensitive (equals false, compareTo == 
0, contains false). Residual evaluation drops matching rows.
   
   Verified on current main (bc94d8e, 2026-11-16): calibration 
`BucketUtil.hash(14.20)` = -500754589 matches the spec example.
   
   ## Expected behavior
   
   Numerically equal decimal literals project to the same bucket regardless of 
written scale, and IN evaluation matches numerically equal decimals.
   
   ## Possible fix
   
   Normalize the decimal literal to the column scale at bind time 
(`DecimalLiteral.to` / `StringLiteral.to`), taking care that values with more 
fractional digits than the column scale must not be silently rounded for EQ; or 
scale-normalize inside the bucket projection. Changing `BucketUtil.hash` itself 
would break spec compatibility with existing data.
   
   ## Notes
   
   - Closest prior report is #7882 (closed, Hive-side trimming, identity 
partition); it did not fix this code path, `Set.contains` is still in current 
main.
   - I can contribute a fix for this bug independently.
   
   ### Willingness to contribute
   
   - [ ] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from 
the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to