zhangfengcdt opened a new issue, #3325:
URL: https://github.com/apache/iceberg-rust/issues/3325
### Apache Iceberg Rust version
`main`. Verified at `bb1e4a4` (2026-09-21); the code referenced below is
unchanged on `main` today.
### Describe the bug
`FanoutWriter` and `ClusteredWriter` put rows whose `float`/`double`
identity partition value is `-0.0` and rows whose value is `0.0` into the same
data file, under whichever value arrived first. iceberg-java writes two
partitions, because it compares partition keys with `Float.compare` /
`Double.compare`, which keep the two apart.
Both writers group rows by the partition `Struct`:
- `FanoutWriter` keeps `partition_writers: HashMap<Struct, _>`
- `ClusteredWriter` keeps `current_partition: Option<Struct>` and
`closed_partitions: HashSet<Struct>`
`PrimitiveLiteral` derives `PartialEq`, `Eq` and `Hash` over `OrderedFloat`,
and `OrderedFloat` treats `-0.0` and `0.0` as equal with the same hash. So the
second value finds the first value's writer.
The data file is then recorded with one partition value while holding rows
for both, so a scan that prunes on the other value drops rows. Seen through
Spark on a table `(id INT, d DOUBLE)` partitioned by `d`, after writing `(1,
-0.0)` and `(2, 0.0)` in one task:
| Writer | Partitions | `SELECT id FROM t WHERE d = 0.0` |
| --- | --- | --- |
| iceberg-rust writers | `d=-0.0` with 2 rows, in 1 file | no rows |
| iceberg-java | `d=-0.0` and `d=0.0`, 1 row each | `2` |
Related work that does not cover this:
- #1951 / #1959 made `Datum` ordering follow IEEE 754 totalOrder. That
changed ordering only; equality and hashing of partition structs are unchanged.
- #3077 adds a bit-exact `same_value` on `PrimitiveLiteral` for comparing
schema defaults. It does not change the derived `Eq` / `Hash` the writers use.
### To Reproduce
```rust
use std::collections::HashMap;
use iceberg::spec::{Literal, Struct};
let neg = Struct::from_iter([Some(Literal::double(-0.0))]);
let pos = Struct::from_iter([Some(Literal::double(0.0))]);
// Passes: the two partition values are equal.
assert_eq!(neg, pos);
// Passes: looking up 0.0 returns the writer opened for -0.0, as in
`FanoutWriter`.
let mut partition_writers: HashMap<Struct, &str> = HashMap::new();
partition_writers.insert(neg, "writer for -0.0");
assert_eq!(partition_writers.get(&pos), Some(&"writer for -0.0"));
```
The same holds for `Literal::float`.
### Expected behavior
`-0.0` and `0.0` are distinct partition values, as in iceberg-java, so the
writers produce two data files with two partition values.
Possible directions: key the writers on a bit-exact form of the partition
struct, or make equality and hashing of float literals bit-exact (one canonical
NaN, signed zeros distinct), which is the Java `Float.equals` / `Double.equals`
behavior that `same_value` in #3077 implements.
### Willingness to contribute
None
### Additional context
Found through apache/datafusion-comet#6138. Comet's native Iceberg writer
uses these writers, and is adding a fall-back for `float`/`double` identity
partitions until this is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
