rahulsmahadev opened a new pull request, #4411:
URL: https://github.com/apache/iceberg-python/pull/4411
# Rationale for this change
A position delete file can list rows of several data files. `ArrowScan`
loads the position deletes of every task and
merges them into one map keyed by data file path. A data file then gets the
positions of any delete file that names
its path, even when the planner did not attach that delete file to its task.
So a delete file that also names another
data file can remove rows from a file it was not planned for, undoing the
sequence number and partition checks that
`DeleteFileIndex` made for it.
The trigger, on a v2 table with a position delete file written by another
engine. P has ids 0..9, Q has ids 100..109,
and the delete file, committed before the first line, lists row 0 of P and
row 0 of Q:
```python
tbl.delete("id < 10") # removes P as a whole file
tbl.add_files([p_path]) # registers P again, so its data
sequence number is newer than the delete's
tbl.scan().to_arrow() # 18 rows, id 0 missing;
expected 0..9 and 101..109
tbl.scan().count() # 19
tbl.scan(row_filter="id < 10").to_arrow() # 0..9, but the unfiltered scan
restricted to id < 10 gives 1..9
```
The delete predates P, so P must keep its row 0. Scan Planning in
`format/spec.md` says that a position delete applies
only when "The data file's data sequence number is _less than or equal to_
the delete file's data sequence number", and
that the data file and position delete file must have equal partitions (both
spec and partition values). Java gives each task only its
own deletes: `GenericDeleteFilter` is built from `task.deletes()`, and
`BaseDeleteLoader` reads the delete files it is
given and keeps the positions for the data file path it is asked for.
This change keeps, for each delete file, only the positions of the data
files whose task has that delete file attached
(`task.delete_files`). Each delete file is still read once, and the key is
unchanged: the path plus the content offset
and size, so deletion vector ranges in one Puffin file stay separate. The
root cause is in the loader, which drops the
association between a delete file and its task, so the lookup in
`_record_batches_from_scan_tasks_and_deletes` is left
as it is. Planning is unchanged; the read path adds one set entry per delete
file and task.
`_DeleteFileKey` comes from #3478. PR #3285 also edits
`_read_all_delete_files` (it skips non-position delete files in
the inner loop over `task.delete_files`). The two changes merge cleanly, and
the merged function keeps both.
## Are these changes tested?
Yes. Both new tests fail on main and pass with this change.
- `test_position_delete_respects_sequence_number_of_readded_file`
(tests/table/test_init.py, `catalog` fixture) runs the
trigger above, and a second shape where the delete is committed after P is
registered again, so it removes row 0 of
both files. It asserts the exact rows, that `count()` agrees with
`to_arrow()`, and that the row-filtered scan agrees
with the unfiltered scan. On main, the first shape fails because id 0 is
missing.
-
`test_read_all_delete_files_only_loads_positions_for_tasks_it_is_attached_to`
(tests/io/test_pyarrow.py) checks the
loader output for three attachment shapes. The third gives the first data
file a newer delete file as well, so a fix
that merges the positions of all tasks also fails it.
`commit_partition_position_delete` (tests/table/test_commit_retry.py) now
takes a list of `(file_path, pos)` rows, so one
delete file can name two data files. Its existing callers pass one row.
## Are there any user-facing changes?
Yes, a read bug fix: scans no longer drop rows that a newer data file keeps,
and an upsert over such a file no longer
inserts a duplicate of a row it should update. There is no API change.
Please add the `changelog` label.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]