andygrove opened a new pull request, #4038:
URL: https://github.com/apache/datafusion-comet/pull/4038
## Which issue does this PR close?
Closes #3442.
## Rationale for this change
Non-AQE DPP support for `CometNativeScanExec` was added in #4011. The
`DynamicPartitionPruningV1Suite."static scan metrics"` test had been tagged
`IgnoreCometNativeDataFusion("...issues/3442")` and was left ignored by that PR
because the test's `getFactScan` helper only matched `FileSourceScanExec` /
`BatchScanExec` / `CometScanExec`, not `CometNativeScanExec`. With the
native_datafusion DPP path in place, adding a `CometNativeScanExec` case lets
the test run.
## What changes are included in this PR?
For the Spark 3.4.3, 3.5.8, and 4.0.1 test diffs:
- Remove `IgnoreCometNativeDataFusion("...issues/3442")` from the `"static
scan metrics"` test.
- Add a `CometNativeScanExec` case to the `find(plan)` block in
`getFactScan`.
## How are these changes tested?
Ran the test locally against Spark 3.5.8 with Comet enabled and `SCAN_AUTO`
(resolves to native_datafusion):
```
ENABLE_COMET=true ENABLE_COMET_ONHEAP=true build/sbt \
'sql/testOnly org.apache.spark.sql.DynamicPartitionPruningV1SuiteAEOff
org.apache.spark.sql.DynamicPartitionPruningV1SuiteAEOn -- -z "static scan
metrics"'
DynamicPartitionPruningV1SuiteAEOff:
- static scan metrics (681 milliseconds)
DynamicPartitionPruningV1SuiteAEOn:
- static scan metrics (disabled when AQE is on) !!! IGNORED !!!
Tests: succeeded 1, failed 0, canceled 0, ignored 1
```
The AEOn variant is ignored by Spark's own `DisableAdaptiveExecution` tag,
not Comet. Spark SQL CI will cover 3.4.3 and 4.0.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
