parthchandra opened a new issue, #7427: URL: https://github.com/apache/datafusion-comet/issues/7427
### What is the problem the feature request solves? Comet runs `MergeRowsExec` natively on Spark 4.0 and later. On Spark 3.4 and 3.5, `MERGE INTO` on Iceberg tables plans Iceberg's own `MergeRowsExec` from `iceberg-spark-extensions`, which Comet does not convert. The operators above it, including the copy-on-write `ReplaceData` write, then stay in Spark. ### Describe the potential solution Add a serde and shim for Iceberg's Spark 3.4 and 3.5 `MergeRowsExec` that reuses the native merge-rows operator from the Spark 4.0 path. Cover matched and not-matched clauses, conditions on each, and the cardinality check. ### Additional context Related: #5122 (row-level MERGE / UPDATE / DELETE plans). This only pays off fully together with native `ReplaceData` (#5644) and `WriteDelta` (#6240). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
