andygrove opened a new pull request, #4041:
URL: https://github.com/apache/datafusion-comet/pull/4041
## Which issue does this PR close?
Related to #3321.
## Rationale for this change
#3321 is an umbrella tracking issue for Spark SQL tests that fail with
`native_datafusion` in auto scan mode. All its original sub-issues (#3311,
#3313, #3314, #3315, #3316) are closed, but several tests tagged against the
umbrella still fail on Spark 4.0.1. Rather than continue tagging everything
against the umbrella, unignore the ones that now pass and re-point the rest at
the narrower issues so progress is easier to track.
## What changes are included in this PR?
Updates `dev/diffs/4.0.1.diff`:
- **Unignored** (tests now pass):
- `DynamicPartitionPruningSuite`: "join key with multiple references on
the filtering plan" (verified across all six V1/V2/V2Filter * AEOff/AEOn
variants)
- `ParquetTypeWideningSuite`: "parquet widening conversion DateType ->
TimestampNTZType"
- **Retagged** to more specific issues:
- 5 `ParquetTypeWideningSuite` sites -> #3720 (native_datafusion silently
succeeds on schema mismatches where Spark errors)
- 3 missing/corrupt-file sites -> #3314 (`FileBasedDataSourceSuite`
ignoreMissingFiles, `SimpleSQLViewSuite` storeAnalyzedPlanForView,
`ParquetV1QuerySuite` ignoreCorruptFiles)
- `SubquerySuite` "SPARK-43402 ... push down data filter with scalar
subquery" -> #3315 (plan collection does not match `CometNativeScanExec`)
- `DynamicPartitionPruningSuite` "Subquery reuse across the whole plan" ->
#3313 (whole-plan subquery reuse count is wrong in V1SuiteAEOff)
#3313, #3314, and #3315 were reopened with comments listing the specific
remaining tests.
## How are these changes tested?
Each test was run locally against Spark 4.0.1 with `native_datafusion` in
auto scan mode via `ENABLE_COMET=true ENABLE_COMET_ONHEAP=true build/sbt
"sql/testOnly <suite> -- -z \"<test>\""`. The unignored tests pass; the
retagged tests still fail as expected, matching the referenced issue
descriptions. CI will exercise the full `ParquetTypeWideningSuite` and all six
DPP variants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
