rishi-rana opened a new pull request, #18159:
URL: https://github.com/apache/iceberg/pull/18159
Closes #18131
## Problem
A copy-on-write `DELETE` on a v3 table fails Spark's per-rule plan
validation:
```
[PLAN_VALIDATION_FAILED_RULE_IN_BATCH] Rule
GroupBasedRowLevelOperationScanPlanning
in batch Early Filter and Projection Push-Down generated an invalid plan:
The plan becomes unresolved: 'ReplaceData RelationV2[id, data] ...
+- Filter NOT ((id = 101) <=> true)
+- RelationV2[id, data, _file, _pos, _row_id,
_last_updated_sequence_number] ...
```
`SparkCopyOnWriteScan.rowLineageAsDataCols` removes the `__metadata_col`
marker from
`_row_id` and `_last_updated_sequence_number` so the 3.5 row-level rules do
not drop them.
`ReplaceData.dataInput` filters out metadata attributes, so both lineage
columns stay in it,
giving four columns against a target relation that supplies two, and
`outputResolved` fails.
Only the intermediate plan is affected, which is why this surfaces just
under plan validation.
Spark 4.x does not override `readSchema()` at all and relies on the native
metadata column
semantics from SPARK-50820, matching the version split in the issue.
## Fix
`UPDATE` and `MERGE` already append the lineage attributes to the target
relation output in
`RewriteUpdateTableForRowLineage` and `RewriteMergeIntoTableForRowLineage`.
`DELETE` had no
equivalent, so this adds `RewriteDeleteFromTableForRowLineage` alongside
them. It needs no
assignments because copy-on-write `DELETE` carries surviving rows over
unchanged.
`shouldUpdatePlan` now also returns false for non-Iceberg tables. Its inner
match had no
default branch, and `DELETE` newly reaches it, so a `DELETE` against another
v2 catalog in a
session with these extensions loaded would hit a `MatchError`.
I also considered dropping the `rowLineageAsDataCols` workaround instead.
That is not viable:
with it removed the lineage columns are no longer written, and
`TestCopyOnWriteWithLineage`
fails with `Structs do not match` and `ArrayIndexOutOfBoundsException`
rather than a plan
validation error. The workaround is still required.
## Testing
No new test was needed. The existing `testDelete` reproduces this once
validation is on, so
this enables `spark.sql.planChangeValidation` in `ExtensionsTestBase`. Spark
gates per-rule
validation on the `spark.testing` **system property**, but the harness sets
`spark.testing`
as a Spark conf, so validation was never actually running.
With validation enabled and without the fix, 30 tests fail across
`TestCopyOnWriteDelete`
and `TestCopyOnWriteWithLineage`. With the fix the full `spark-extensions`
suite passes:
2501 passed, 180 skipped, 0 failures
(`TestRewritePositionDeleteFilesProcedure` hit an
unrelated flaky REST catalog 404 during initialization in one run and passes
on its own).
## Notes for reviewers
- The rule is not restricted to copy-on-write.
`RewriteUpdateTableForRowLineage` and
`RewriteMergeIntoTableForRowLineage` do not distinguish either, and I
verified merge-on-read
is unaffected (`TestMergeOnReadWithLineage` 40/40). Say the word if you
would rather it were
narrowed to the group-based path.
- Unlike the sibling rules, this one falls through to the unchanged plan
instead of risking a
`MatchError`, and guards on `resolved` the way Spark's own
`RewriteDeleteFromTable` and
`RewriteMergeIntoTableForRowLineage` do.
- Enabling plan validation for the whole extensions suite is deliberate, to
stop this class of
bug recurring, but it is the part of this change most likely to affect
unrelated tests. Happy
to scope it to the row lineage tests instead.
---
**AI Disclosure**
- Model: Claude Opus 5
- Platform/Tool: Claude Code
- Human Oversight: partially reviewed
- Prompt Summary: Investigate #18131, identify the root cause of the
copy-on-write DELETE plan
validation failure, and implement a fix with regression coverage. The
agent reproduced the
failure, traced it to the missing DELETE counterpart of the existing row
lineage rules,
empirically ruled out the alternative fix, and ran the full
spark-extensions suite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
