mattfaltyn opened a new issue, #1849:
URL: https://github.com/apache/iceberg-go/issues/1849
## Version
`main` at `8f6e791182d4268c02bd0261b135f8134047f124` (also affects the
equality-delete read path released in v0.6.0).
## Problem
A scan fails when an applicable equality delete references a field that is
not part of the selected output columns.
The Arrow scanner builds the physical data-file projection from the
requested output fields and row-filter fields. Equality-delete fields are
needed to evaluate row liveness, but are not added to that physical projection.
The equality-delete processor then cannot resolve the omitted field and returns
an error.
Output projection should not change which rows are considered deleted.
## Reproduction
1. Create a v2 table with schema `id: long, data: string`.
2. Append rows `(1, "one")`, `(2, "two")`, and `(3, "three")`.
3. Add an equality delete for `id = 2` with equality field ID `1`.
4. Read using
`tbl.Scan(table.WithSelectedFields("data")).ToArrowRecords(ctx)`.
5. Consume the returned record iterator.
This fails deterministically with:
```text
equality field ID 1 (id) not found in <table>/data/data.parquet
```
The existing equality-delete round-trip succeeds when selecting `id`, which
is why current tests do not expose the projection dependency.
## Expected behavior
The scan returns the projected `data` values `"one"` and `"three"`. The
scanner may read `id` internally to apply the delete, but it should not expose
`id` in the result schema.
## Actual behavior
The record iterator returns an error and no result because `id` was pruned
before equality deletes were applied.
## Proposed fix
Include equality-delete field IDs in the physical data-file projection for
each scan task, apply equality deletes, and retain the existing final
projection to remove execution-only columns from the output. Add a regression
test whose selected output excludes the equality key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
