RyanLin5967 opened a new issue, #4185:
URL: https://github.com/apache/iceberg-python/issues/4185
### Apache Iceberg version
0.12.0 (latest release)
### Please describe the bug 🐞
Currently, applying `RemoveSnapshotsUpdate` drops only the expired
snapshots' own `snapshot-log` entries, so a timestamp at which one of them was
current resolves to an older snapshot. The lookup is right when every expired
entry is older than every kept one.
```python
import tempfile, time
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
cat = SqlCatalog("c", uri="sqlite:///:memory:",
warehouse=f"file://{tempfile.mkdtemp()}")
cat.create_namespace("n")
t = cat.create_table("n.t", schema=pa.schema([("x", pa.int64())]))
for x in (1, 2, 3): # s1 holds [1], s2 [1, 2] and s3 [1, 2, 3]
t.append(pa.table({"x": [x]}))
time.sleep(0.01)
s1, s2, s3 = t.history()
ts = s3.timestamp_ms - 1 # while s2 was current
def rows_at(ts):
s = t.snapshot_as_of_timestamp(ts)
return None if s is None else
sorted(t.scan(snapshot_id=s.snapshot_id).to_arrow()["x"].to_pylist())
print(rows_at(ts))
# [1, 2]
t.maintenance.expire_snapshots().by_id(s2.snapshot_id).commit()
print(rows_at(ts))
# [1], should be None
# the spec on snapshot-log: "all entries before a snapshot that has expired
should be removed"
```
`older_than` does the same when a tag keeps an older snapshot, and this log
is what reaches the metadata file, so Iceberg Java's
`SnapshotUtil.snapshotIdAsOfTime` also returns s1.
tested on pyiceberg 0.12.0 and main, and Iceberg Java 1.12.0
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
