turbolytics opened a new issue, #4403:
URL: https://github.com/apache/iceberg-python/issues/4403

   ### Apache Iceberg version
   
   0.12.0 (latest release)
   
   ### Please describe the bug 🐞
   
   A `truncate` partition loses rows whose value is near the minimum int or 
long. `EqualTo`, `In` and range filters such as `LessThan(x, 0)` all miss them. 
It reproduces on 0.12.0 and on `main` (33b4ce5).
   
   ```python
   import tempfile
   
   import pyarrow as pa
   
   from pyiceberg.catalog.sql import SqlCatalog
   from pyiceberg.expressions import EqualTo, LessThan
   from pyiceberg.partitioning import PartitionField, PartitionSpec
   from pyiceberg.schema import Schema
   from pyiceberg.transforms import TruncateTransform
   from pyiceberg.types import IntegerType, LongType, NestedField
   
   tmp = tempfile.mkdtemp()
   catalog = SqlCatalog("demo", uri=f"sqlite:///{tmp}/catalog.db", 
warehouse=f"file://{tmp}/warehouse")
   catalog.create_namespace("ns")
   
   schema = Schema(NestedField(1, "id", LongType(), required=True), 
NestedField(2, "x", IntegerType()))
   spec = PartitionSpec(PartitionField(source_id=2, field_id=1000, 
transform=TruncateTransform(1000), name="x_trunc"))
   t = catalog.create_table("ns.trunc", schema=schema, partition_spec=spec)
   rows = [{"id": 0, "x": -(2**31)}, {"id": 1, "x": -(2**31) + 100}, {"id": 2, 
"x": 5}]
   t.append(pa.Table.from_pylist(rows, schema=schema.as_arrow()))
   
   
   def ids(row_filter):
       return sorted(t.scan(row_filter=row_filter).to_arrow()["id"].to_pylist())
   
   
   print(sorted(task.file.partition[0] for task in t.scan().plan_files()))  # 
[0, 2147483296]
   print(ids(EqualTo("x", -(2**31))))  # [] -- expected [0]
   print(ids(EqualTo("x", -(2**31) + 100)))  # [] -- expected [1]
   print(ids(LessThan("x", 0)))  # [] -- expected [0, 1]
   ```
   
   For `truncate[1000]` on an int, every value from -2147483648 to -2147483001 
is affected. A long column behaves the same way near `-(2**63)`.
   
   **Cause.** The writer computes the partition value with pyiceberg-core. Its 
math wraps around, as Java's 
[`TruncateUtil.truncateInt`](https://github.com/apache/iceberg/blob/2517ec2a7f70721166bd2b2ca0061984e4eb2bdc/api/src/main/java/org/apache/iceberg/util/TruncateUtil.java#L55-L61)
 does. So these rows land in partition `2147483296`, a large positive value. 
Scan planning projects the filter with the Python transform, [`v - v % 
self._width`](https://github.com/apache/iceberg-python/blob/33b4ce511ec0c9dc0f74e38fff5d478ef2ff47dd/pyiceberg/transforms.py#L867),
 which does not wrap. `EqualTo(x, -2147483648)` projects to `-2147484000`, so 
it never matches the stored value. `LessThan(x, 0)` projects to `x_trunc <= 
-1000`, and the stored value is positive. Either way, the scan prunes the file.
   
   **What a fix could look like.**
   
   - The writer should keep its current output. Java writes the wrapped value 
too, so existing tables already hold it.
   - Wrapping the Python transform like Java fixes `EqualTo` and `In`.
   - Range filters need more than that, because the wrapped value does not sort 
with the others. Java's projection 
([ProjectionUtil.java#L42-L58](https://github.com/apache/iceberg/blob/2517ec2a7f70721166bd2b2ca0061984e4eb2bdc/api/src/main/java/org/apache/iceberg/transforms/ProjectionUtil.java#L42-L58))
 has the same gap. A [comment on 
apache/iceberg#12915](https://github.com/apache/iceberg/issues/12915#issuecomment-2880454693)
 notes the `Integer.MIN_VALUE` overflow, and that issue is closed. 
apache/iceberg#18111 covers a different overflow in the same function.
   
   I'm not sure of the right fix for range filters. It may need a note in the 
spec. I can raise it in apache/iceberg if that helps.
   
   ### How I found it
   
   I used [Temper](https://github.com/turbolytics/temper) to audit each table 
layout against the same scan guarantees. The reference is the same filter 
written as plain Python over the appended rows. Temper helped identify this 
bug. In 
[`MATRIX.md`](https://github.com/turbolytics/iceberg-python/blob/6de93440a50ea37c96e204a321814db385385911/MATRIX.md),
 only the `truncate` layout fails "int and long extremes are found". The tests 
are 
[here](https://github.com/turbolytics/iceberg-python/blob/6de93440a50ea37c96e204a321814db385385911/tests/table/test_temper_scan_layout.py).
   
   Once the approach is agreed, I can send a PR for `EqualTo` and `In`, with 
regression tests in the existing test files.
   
   I used an AI assistant to run the tests and draft this issue. I verified the 
example on 0.12.0 and on `main`, with pyarrow 26.0.1 and Python 3.12.9.
   
   ### Willingness to contribute
   
   - [x] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from 
the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to