rishi-rana commented on PR #18159: URL: https://github.com/apache/iceberg/pull/18159#issuecomment-5725398118
@manuzhang Yes, this one is Spark 3.5 specific — there is no 4.2 counterpart to port it to. The row lineage rules for row-level operations only exist under `spark/v3.5`: ``` v3.5 RewriteOperationForRowLineage.scala RewriteUpdateTableForRowLineage.scala RewriteMergeIntoTableForRowLineage.scala v4.0 (none) v4.1 (none) v4.2 (none) ``` Same for the trigger. `SparkCopyOnWriteScan` overrides `readSchema()` to strip the metadata column marker off the lineage columns only in 3.5 (added in #12736); 4.0/4.1/4.2 have no override and rely on the native metadata column semantics from SPARK-50820. Without that marker being stripped there is no mismatch between `ReplaceData.dataInput` and the target relation output, so the plan stays resolved. I verified this rather than assuming it: I enabled `spark.sql.planChangeValidation` in the 4.2 `ExtensionsTestBase` and ran the same three classes that fail on 3.5: | | Spark 4.2 | Spark 3.5 before this PR | |---|---|---| | TestCopyOnWriteDelete | 112 passed, 20 skipped | 26 failed | | TestCopyOnWriteWithLineage | 40 passed | 4 failed | | TestMergeOnReadWithLineage | 40 passed | 40 passed | 192 passed / 0 failures on 4.2. I reverted that local 4.2 change since it is not part of this PR. Happy to open a separate PR enabling `spark.sql.planChangeValidation` in the 4.x test harnesses if you think it is worth having as a guard there too — it passes today, so it would be purely preventative and seemed like a different concern from this fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
