mhatipog commented on issue #9130:
URL: https://github.com/apache/iceberg/issues/9130#issuecomment-6916032118

   This still happens on Iceberg 1.12.0 (Spark 4.0.1, 
`iceberg-spark-runtime-4.0_2.13:1.12.0`, Hadoop catalog, default properties). 
Could it be reopened? The issue was closed as stale, not fixed.
   
   ```sql
   CREATE TABLE local.db.nan (id BIGINT, d DOUBLE) USING iceberg;
   INSERT INTO local.db.nan VALUES (1, 1.0), (2, CAST('NaN' AS DOUBLE));
   INSERT INTO local.db.nan VALUES (3, CAST('NaN' AS DOUBLE));
   ```
   
   | query | Iceberg | same rows as a Parquet table (Spark semantics) |
   | --- | --- | --- |
   | `WHERE d > 1.0` | `[]` | `[2, 3]` |
   | `WHERE d >= 1.0` | `[1, 2]` | `[1, 2, 3]` |
   | `WHERE NOT (d <= 1.0)` | `[]` | `[2, 3]` |
   | `SELECT count(*) ... WHERE d > 1.0` | `0` | `2` |
   
   Spark orders NaN above every other value, so `NaN > 1.0` is true. Iceberg 
pruning treats it as false and drops:
   - files whose bounds exclude NaN (the first file, bounds `[1.0, 1.0]`)
   - files that contain only NaN (the second file), even though its 
`nan_value_counts` says so
   
   In a wider sweep, the same happens for:
   - identity partitions holding NaN
   - files added with `add_files` (NaN counts unknown)
   - pyiceberg-written files
   - `float` columns
   - the predicates `>=`, `NOT (d < x)`, `> Infinity` and `NOT BETWEEN`
   
   The suggestion above, translating Spark's `col > x` into Iceberg's `col > x 
OR is_nan(col)` for float and double columns (and similarly for `>=` and 
negated `<` / `<=`), would cover all of these.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to