qzyu999 opened a new issue, #3857:
URL: https://github.com/apache/iceberg-python/issues/3857
## Description
`_DeleteFiles.delete_data_file()` silently drops explicit file references
because `_compute_deletes` resets `self._deleted_data_files = set()` before
scanning manifests by predicate.
The `delete_data_file()` method is inherited from `_SnapshotProducer` and
adds to `self._deleted_data_files`. However, when
`_DeleteFiles._compute_deletes` runs (triggered by `_deleted_entries()`), it
resets this field to an empty set and repopulates it only from files matched by
the predicate evaluator. Any explicitly added files are silently lost.
While the high-level `table.delete()` API only uses predicates (so this
isn't hit in normal usage), the low-level `update_snapshot().delete()` API
exposes `delete_data_file()` as a public method. Calling it produces no error
and no effect.
## Reproduction
```python
import pyarrow as pa
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import LongType, NestedField
catalog = load_catalog("default")
catalog.create_namespace("default")
table = catalog.create_table(
"default.delete_explicit",
Schema(NestedField(1, "x", LongType(), required=False)),
)
table.append(pa.table({"x": [1, 2, 3]}))
data_file = next(iter(table.scan().plan_files())).file
# This should delete the file but silently does nothing
with table.transaction() as tx:
delete_snapshot = tx.update_snapshot().delete()
delete_snapshot.delete_data_file(data_file)
# Bug: table still has [1, 2, 3]
print(table.scan().to_arrow()["x"].to_pylist())
```
Expected: table is empty after commit.
Actual: table still has `[1, 2, 3]`.
## Root cause
In `pyiceberg/table/update/snapshot.py`, `_DeleteFiles._compute_deletes`
(line ~598):
```python
self._deleted_data_files = set() # <-- overwrites any explicit files added
via delete_data_file()
```
Then it repopulates from predicate matches only. Since no predicate was set
(`AlwaysFalse` by default), nothing matches, nothing is deleted.
## Suggested fix
Before resetting, preserve explicit files and include them in the deletion
scan:
```python
# Preserve files explicitly requested for deletion
explicit_deletes = set(self._deleted_data_files)
self._deleted_data_files = set()
# ... existing predicate-based scan ...
# After the scan, also mark explicitly requested files as deleted:
for entry in manifest_entries:
if entry.data_file in explicit_deletes:
# mark as deleted
```
Alternatively, prevent calling `delete_data_file()` on `_DeleteFiles`
entirely by raising `NotImplementedError`.
## Related
Follow-up observation from #3818 review. The `_OverwriteFiles` path handles
`delete_data_file()` correctly because it does not reset the field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
