Neuw84 commented on issue #18620:
URL: https://github.com/apache/iceberg/issues/18620#issuecomment-6358574710
Started digging into this. Posting an interim finding with evidence to
narrow where the bug is, because it rules out the layer I'd have guessed first.
### What I ruled out
I reproduced the `DeleteFilter` equality path in isolation against the
`StructProjection` + `StructLikeSet` machinery, with a `StructLike` that mimics
the Spark wrapper's key property — a **single shared nested-struct wrapper that
is re-pointed to the current row on every read** (the pattern
`InternalRowWrapper`'s struct getter uses: one `nestedWrapper` instance,
`.wrap(row.getStruct(...))` per call). Setup mirrors
`DeleteFilter.applyEqDeletes`:
- `deleteSchema = TypeUtil.selectInIdOrder(fullSchema, {4})` → `{2: st:
struct<4: k>}`
- delete set loaded with independent child records for `k in {5,10}`
- probe = one shared `StructProjection` wrapping one shared row wrapper
whose `st` read re-points a shared child, iterated over `k = 0,1,5,6,10,11`
Result: **only `k=5` and `k=10` matched (2/6), which is correct.** So
`StructProjection` reading a nested key through a shared, re-pointed child
wrapper, and `StructLikeSet.contains` (eager `JavaHashes` +
`StructLikeComparator` recursion into the nested struct), are **not** where the
"deletes every row" behavior comes from. The shared-nested-wrapper reuse is not
sufficient to cause it — Flink's `RowDataWrapper` and core's
`InternalRecordWrapper` use the same shared-nested-wrapper pattern and are
unaffected, which is consistent.
### Where that leaves it
The defect is therefore below `StructProjection` — in `InternalRowWrapper`'s
own nested read specifically, i.e. `get(pos, StructLike.class)` for a `STRUCT`
field returning `nestedWrapper.wrap(row.getStruct(pos, size))`, and/or how the
nested `InternalRowWrapper` then reads the leaf `k`. The likely candidates to
confirm against a real Spark row (`UnsafeRow` vs the columnar
`ColumnarBatchRow` the vectorized reader produces):
1. whether `row.getStruct(pos, n)` returns a row the nested wrapper reads
the leaf from correctly, or a view whose positions/null-flags don't line up for
the projected key;
2. whether the nested `InternalRowWrapper` is built with a struct type whose
leaf position matches what the nested `StructProjection` expects.
I don't have a Spark test environment on this machine (the build needs JDK
17/21 and the full Spark harness), so I can't yet run the end-to-end Spark
repro to pin which of those it is. If someone can run the issue's repro with a
breakpoint in `InternalRowWrapper.get` for the `st` column — checking what
`row.getStruct(...)` returns and what the nested wrapper reads for `k` on two
different rows — that should isolate it immediately. Happy to take the fix from
there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
