unikdahal commented on code in PR #5318:
URL: https://github.com/apache/datafusion-comet/pull/5318#discussion_r4126241925
##########
spark/src/test/scala/org/apache/comet/CometIcebergWriteActionSuite.scala:
##########
@@ -904,16 +904,9 @@ class CometIcebergWriteActionSuite
}
}
- test("native acceleration: ReplaceData (CoW MERGE) falls back (MergeRowsExec
not Comet)") {
- // TODO(comet-merge-rows): native MERGE engagement requires a Comet
equivalent of Iceberg's
- // `MergeRowsExec` (the per-row dispatch operator that assigns
__row_operation codes from
- // MATCHED/NOT MATCHED clauses). Without it, `MergeRowsExec` stays JVM,
the upstream chain
- // breaks Comet-native partway, and `requiresNativeChildren=true` declines
the
- // `IcebergWriteExec -> CometIcebergWriteExec` conversion. Until that
lands, MERGE
- // falls back to the JVM two-op path -- this test pins that contract.
Native `MergeRowsExec`
- // is being added in https://github.com/apache/datafusion-comet/pull/5318;
when that lands
- // this test will start failing and needs to flip to
`assertNativeWriteEngages`.
+ test("native acceleration: ReplaceData (CoW MERGE) honors the versioned
native contract") {
assumeNativeAcceleration()
+ assume(isSpark35Plus, "MergeRowsExec requires Spark 3.5+")
Review Comment:
Fixed. Kept the fallback path for 3.4 as well instead of skipping the test.
##########
docs/source/user-guide/latest/compatibility/operators.md:
##########
@@ -113,6 +113,45 @@ runs natively; it is controlled by
`spark.comet.exec.windowGroupLimit.enabled` (
Scalar `FLOAT` and `DOUBLE` keys are normalized and match Spark; see
[floating-point ordering](./floating-point.md).
+## MERGE INTO (MergeRowsExec)
+
+Spark `MergeRowsExec` appears as `CometMergeRows` when native execution is
enabled.
Review Comment:
Added the MergeRowsExec entry under Writes with the supported versions and
opt-in note.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
