andygrove opened a new issue, #3141:
URL: https://github.com/apache/iceberg-rust/issues/3141
### Apache Iceberg Rust version
main @ `3d84c81353b1b23b6e4ae8eea8f8a021cc6927a7`
### Describe the bug
`Truncate::transform` in `crates/iceberg/src/transform/truncate.rs` does not
reproduce Iceberg Java's `TruncateUtil` arithmetic. There are two distinct
problems.
**1. `truncate_i32` diverges from Java for widths above 2^30**
```rust
fn truncate_i32(v: i32, width: i32) -> i32 {
v - v.rem_euclid(width)
}
```
Java computes the positive remainder with an expression that is allowed to
overflow:
```java
// org.apache.iceberg.util.TruncateUtil
public static int truncateInt(int width, int value) {
return value - (((value % width) + width) % width);
}
```
`(value % width) + width` overflows `int` whenever `value % width >
Integer.MAX_VALUE - width`, which is reachable for any `width > 2^30`.
`rem_euclid` never overflows, so the two produce different partition values.
Only positive values with a large width are affected; negative values and
widths at or below 2^30 agree.
`Truncate.get(int width)` in Iceberg Java only validates `width > 0`, so
these widths are accepted by the spec. Whether Java's overflow is desirable is
beside the point — it is the value iceberg-java writes into the partition
tuple, so a Rust writer that computes something else produces a table the Java
reader partitions differently.
**2. Overflow panics in debug builds at the integer minimum**
```rust
fn truncate_i64(v: i64, width: i64) -> i64 {
v - (((v % width) + width) % width)
}
```
For `v = i64::MIN, width = 1000` the remainder is 192 and `v - 192`
underflows. With `debug_assertions` on this panics (`attempt to subtract with
overflow`); in a release build it wraps and matches Java exactly.
`truncate_i32` has the same shape at `i32::MIN`, and `truncate_decimal_i128`
uses the same unchecked subtraction (unreachable for a 38-digit Iceberg
decimal, but the same pattern).
A panic here aborts the write task, so a debug or test build of any
downstream writer fails on data that Java handles.
### To Reproduce
Values below were produced by running `TruncateUtil` from apache-iceberg
1.11.0 on JDK 17 against the Rust expressions:
| width | value | Java `TruncateUtil.truncateInt` | iceberg-rust
`truncate_i32` |
| --- | --- | --- | --- |
| 2000000000 | 1500000000 | -2000000000 | 0 |
| 2147483647 | 2147483646 | -2147483647 | 0 |
| 1073741824 | 1500000000 | 1073741824 | 1073741824 |
| 1000 | 2147483647 | 2147483000 | 2147483000 |
For problem 2, in a debug build:
```rust
let t = create_transform_function(&Transform::Truncate(1000)).unwrap();
t.transform(Arc::new(Int64Array::from(vec![i64::MIN]))).unwrap(); // panics
```
### Expected behavior
Both kernels bit-identical to Java in debug and release. Wrapping arithmetic
gives that:
```rust
fn truncate_i32(v: i32, w: i32) -> i32 {
v.wrapping_sub((v % w).wrapping_add(w) % w)
}
fn truncate_i64(v: i64, w: i64) -> i64 {
v.wrapping_sub((v % w).wrapping_add(w) % w)
}
```
That fixes both problems at once: it reproduces Java's overflow for large
widths and removes the debug panic. The alternative for problem 1 is to reject
`width > 2^30` at construction, which overlaps with #2473, but that would
reject specs iceberg-java accepts.
### Willingness to contribute
I would be willing to contribute a fix for this bug with guidance from the
Iceberg community.
---
**How this was found.** Apache DataFusion Comet is adding native kernels for
Iceberg's Spark system functions (apache/datafusion-comet#5638). On a
partitioned write the sort key comes from those kernels (Java semantics) while
the partition values the clustered writer groups by come from iceberg-rust's
transforms, and the clustered writer fails at runtime with `The input is not
sorted! Cannot write to partition that was previously closed` when the two
disagree. Cross-checking the two implementations over boundary inputs is what
surfaced this.
_This report was drafted with LLM assistance (Claude Code); the divergences
were verified against Iceberg Java on a JVM._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
