andygrove opened a new pull request, #7423: URL: https://github.com/apache/datafusion-comet/pull/7423
## Which issue does this PR close? Related to #7422. Does not close it: the Iceberg failure it describes is still open. ## Rationale for this change The Spark 3.4 jobs fail on two tests that were never run against 3.4 when they merged. They only surfaced on #7402, which carried `run-all-spark-profiles`. ## What changes are included in this PR? - `CometCollectMetricsSuite`: the two "public foreachPartition preserves observation row demand" tests now require Spark 3.5+. Spark 3.4's `Dataset.foreachPartition` calls `withNewRDDExecutionId` without a name, so no `QueryExecutionListener` callback reports the plan the test asserts on. - `CometIcebergNativeSuite`: "AQE DPP - V2 broadcast query stage creation order (SPARK-34637)" now requires Spark 3.5+. On 3.4 it fails deterministically with `Column index 1 is out of bound. Schema: .` The root cause is a real bug tracked in #7422, so the test should be re-enabled when that is fixed. The now-dead `isSpark35Plus` check inside the test is removed. ## How are these changes tested? Ran each test locally with `-Pspark-3.4`: all three report as canceled, and before the change the Iceberg test failed with the error above. The Spark 3.5+ behavior is unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
