turbolytics opened a new issue, #4404:
URL: https://github.com/apache/iceberg-python/issues/4404
### Apache Iceberg version
0.12.0 (latest release)
### Please describe the bug 🐞
Filtered scans raise or return wrong rows at two kinds of edge values. Both
bugs reproduce on 0.12.0 and on `main` (33b4ce5). A third bug, in `truncate`
partitions, is in #4403.
```python
import datetime as dt
import math
import tempfile
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.expressions import EqualTo, In, LessThanOrEqual, NotEqualTo
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import YearTransform
from pyiceberg.types import DateType, FloatType, LongType, NestedField
tmp = tempfile.mkdtemp()
catalog = SqlCatalog("demo", uri=f"sqlite:///{tmp}/catalog.db",
warehouse=f"file://{tmp}/warehouse")
catalog.create_namespace("ns")
def table(name, column, spec, values):
schema = Schema(NestedField(1, "id", LongType(), required=True),
NestedField(2, "x", column))
t = catalog.create_table(f"ns.{name}", schema=schema,
partition_spec=spec)
t.append(pa.Table.from_pylist([{"id": i, "x": v} for i, v in
enumerate(values)], schema=schema.as_arrow()))
return t
def ids(t, row_filter):
try:
return
sorted(t.scan(row_filter=row_filter).to_arrow()["id"].to_pylist())
except OverflowError as e:
return repr(e)
# 1. year partitions and the last representable date
spec = PartitionSpec(PartitionField(source_id=2, field_id=1000,
transform=YearTransform(), name="x_year"))
t = table("year", DateType(), spec, [dt.date(2024, 5, 1), dt.date(9999, 12,
31)])
print(ids(t, LessThanOrEqual("x", dt.date(9999, 12, 31)))) #
OverflowError('date value out of range') -- expected [0, 1]
# 2. inf in a float column
t = table("floats", FloatType(), PartitionSpec(), [1.5, math.inf])
print(ids(t, EqualTo("x", math.inf))) # [] -- expected [1]
print(ids(t, NotEqualTo("x", math.inf))) # [0, 1] -- expected [0]
print(ids(t, In("x", {math.inf, 1.5}))) # [0] -- expected [0, 1]
```
#### 1. `year` and `month` partitions raise `OverflowError` at 9999-12-31
and 0001-01-01
`LessThanOrEqual` and `GreaterThan` raise at 9999-12-31.
`GreaterThanOrEqual` and `LessThan` raise at 0001-01-01. Timestamp columns
raise the same way at `9999-12-31 23:59:59.999999` and `0001-01-01 00:00:00`.
`EqualTo` and `In` work. `day`, `hour`, `identity` and `bucket` partitions
return the right rows. 9999-12-31 is a common "no end date" value.
**Cause.** The projections step one day past the bound. The strict
projection turns `<= 9999-12-31` into `< 10000-01-01`
([transforms.py#L1066-L1070](https://github.com/apache/iceberg-python/blob/33b4ce511ec0c9dc0f74e38fff5d478ef2ff47dd/pyiceberg/transforms.py#L1066-L1070)).
The inclusive projection turns `> 9999-12-31` into `>= 10000-01-01`
([transforms.py#L1041-L1046](https://github.com/apache/iceberg-python/blob/33b4ce511ec0c9dc0f74e38fff5d478ef2ff47dd/pyiceberg/transforms.py#L1041-L1046)).
Both then convert the day count to a Python `date`
([datetime.py#L242-L243](https://github.com/apache/iceberg-python/blob/33b4ce511ec0c9dc0f74e38fff5d478ef2ff47dd/pyiceberg/utils/datetime.py#L242-L243)),
and Python dates stop at year 9999. Java computes years and months with
`LocalDate`
([DateTimeUtil.java#L207-L225](https://github.com/apache/iceberg/blob/2517ec2a7f70721166bd2b2ca0061984e4eb2bdc/api/src/main/java/org/apache/iceberg/util/DateTimeUtil.java#L207-L225)),
which has a much wider r
ange.
#### 2. Filters on `inf` in a `float` column return wrong rows
`EqualTo` and `In` miss the `inf` row, and `NotEqualTo` keeps it.
`LessThanOrEqual(x, -math.inf)` misses the `-inf` row. Every layout behaves the
same way. `double` columns are not affected.
**Cause.** A Python float becomes a `DoubleLiteral`.
[`DoubleLiteral.to(FloatType)`](https://github.com/apache/iceberg-python/blob/33b4ce511ec0c9dc0f74e38fff5d478ef2ff47dd/pyiceberg/expressions/literals.py#L434-L440)
returns `FloatAboveMax` for any value above the float32 maximum, and `inf` is
above it. Binding then turns `EqualTo` into `AlwaysFalse` and `NotEqualTo` into
`AlwaysTrue`. Java has the same conversion
([Literals.java#L372-L380](https://github.com/apache/iceberg/blob/2517ec2a7f70721166bd2b2ca0061984e4eb2bdc/api/src/main/java/org/apache/iceberg/expressions/Literals.java#L372-L380)).
Java engines such as Spark pass a `Float` for a float column, so they do not
take this path. In PyIceberg, every Python float does. `inf` and `-inf` are
valid float values, so they could convert to `FloatLiteral`.
#### Already reported
While probing, I also hit three known issues. I do not repeat them here.
- #3498 item 3, the `TypeError` for a null identity partition under an `Or`
filter, still reproduces on `main`. The stale bot closed its fix, #3817.
- apache/arrow#51491: Parquet row-group statistics drop NaN rows for `!=`
and `is_nan`.
- apache/arrow#46183: `isin([0.0])` drops a row group whose values are all
`0.0`.
### How I found them
I used [Temper](https://github.com/turbolytics/temper) to audit each table
layout against the same scan guarantees. Every layout holds the same rows. The
reference is the same filter written as plain Python over the appended rows.
Temper helped identify these bugs. The table shows the result.
| Guarantee | unpartitioned | many files | identity | bucket | truncate |
year | month | day | hour |
|---|---|---|---|---|---|---|---|---|---|
| random filters match Python | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| int and long extremes are found | ✅ | ✅ | ✅ | ✅ | ❌ #4403 | ✅ | ✅ | ✅ | ✅ |
| date and time sentinels scan | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ 1 | ❌ 1 | ✅ | ✅ |
| float `inf` is found | ❌ 2 | ❌ 2 | ❌ 2 | ❌ 2 | ❌ 2 | ❌ 2 | ❌ 2 | ❌ 2 | ❌ 2
|
✅ passes. ❌ fails, marked with the bug number. The random row runs 300
seeded filters on each layout, and all of them matched.
The map is
[`MATRIX.md`](https://github.com/turbolytics/iceberg-python/blob/6de93440a50ea37c96e204a321814db385385911/MATRIX.md),
built from [these
tests](https://github.com/turbolytics/iceberg-python/blob/6de93440a50ea37c96e204a321814db385385911/tests/table/test_temper_scan_layout.py)
and [these
declarations](https://github.com/turbolytics/iceberg-python/tree/6de93440a50ea37c96e204a321814db385385911/temper).
I can send a PR for each bug, with regression tests in the existing test
files. I'm happy to split them into separate issues.
I used an AI assistant to run the tests and draft this issue. I verified
each example on 0.12.0 and on `main`, with pyarrow 26.0.1 and Python 3.12.9.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]