Rishavbhattarai opened a new issue, #18723: URL: https://github.com/apache/iceberg/issues/18723
### Apache Iceberg version main (development) ### Query engine None ### Please describe the bug 🐞 The expressions spec says `float` and `double` "use IEEE 754 order for all non-NaN values", and under that order `-0.0 = 0.0`. The wording was chosen on purpose in #16652 ([discussion](https://github.com/apache/iceberg/pull/16652#discussion_r3471304348)) to differ from IEEE total order, which puts `-0.0 < 0.0`. The Java evaluators use `Comparator.naturalOrder()` for float and double (`Comparators.java`), which is total order. So a row holding `-0.0` doesn't match `x = 0.0`, and a data file or manifest whose only value is `-0.0` gets pruned for `x = 0.0`, `x IN (0.0, ...)` and `x >= 0.0`. Reproducer on main (be25281): ```java Schema schema = new Schema(Types.NestedField.required(1, "x", Types.DoubleType.get())); Expression expr = Expressions.equal("x", 0.0); new Evaluator(schema.asStruct(), expr).eval(TestHelpers.Row.of(-0.0)); // false, expected true ByteBuffer negZero = Conversions.toByteBuffer(Types.DoubleType.get(), -0.0); DataFile file = new TestHelpers.TestDataFile( "file.parquet", TestHelpers.Row.of(), 10, Map.of(1, 10L), Map.of(1, 0L), Map.of(1, 0L), Map.of(1, negZero), Map.of(1, negZero)); new InclusiveMetricsEvaluator(schema, expr).eval(file); // false (file skipped), expected true ``` `ManifestEvaluator` behaves the same way on an identity partition summary of `[-0.0, -0.0]`. Engines follow the spec here. Spark 3.5.3 and PostgreSQL 16 both evaluate `CAST('-0.0' AS DOUBLE) = 0.0` as true and keep a `-0.0` row for `WHERE x = 0.0`. If an engine pushes that filter down to Iceberg, files holding only `-0.0` would be skipped and those rows lost. I haven't run that end to end with an Iceberg table. Bounds are a separate rule: `spec.md` requires `-0.0` to sort before `+0.0` in lower/upper bounds, so `Comparators` can't simply switch to IEEE equality for everything. Where should the fix go: in the predicate comparisons only (`Evaluator` and the metrics/manifest evaluators), or somewhere else? I'm happy to work on it once the approach is agreed. I used an AI assistant to help write the reproducer and checked the results and spec references myself. ### Willingness to contribute - [ ] I can contribute a fix for this bug independently - [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
