comphead opened a new issue, #5986:
URL: https://github.com/apache/datafusion-comet/issues/5986
### What is the problem the feature request solves?
The Spark SQL CI matrix runs `sql/testOnly *` (see
`dev/ci/spark-sql-modules.py`), so every
RocksDB state-store suite under
`sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/`
runs with Comet
enabled. None of them exercise a Comet code path:
- `CometExecRule` and `CometScanRule` both bail out at the top of `_apply`
for any streaming
plan (`ShimCometStreaming.isStreamingPlan`), so the suites that drive
streaming queries
(`RocksDBStateStoreIntegrationSuite`,
`RocksDBStateStoreCheckpointFormatV2Suite`,
`RocksDBCheckpointFailureInjectionSuite`) execute plain Spark from end to
end.
- The rest (`RocksDBSuite`, `RocksDBStateStoreSuite`, `RocksDBLineageSuite`)
call
`StateStore`/`RocksDB` APIs directly and never build a query plan at all.
Current state, per `dev/diffs/`:
| Suite | 3.4.3 | 3.5.9 | 4.0.4 | 4.1.3 | shard |
| --- | --- | --- | --- | --- | --- |
| `RocksDBStateStoreIntegrationSuite` | runs | runs | runs | ignored |
`sql_core-1` |
| `RocksDBStateStoreSuite` | runs | runs | runs | runs | `sql_core-2` |
| `RocksDBSuite` | runs | runs | runs | runs | `sql_core-3` |
| `RocksDBStateStoreCheckpointFormatV2Suite` | n/a | n/a | runs | runs |
`sql_core-1` |
| `RocksDBCheckpointFailureInjectionSuite` | n/a | n/a | n/a | runs |
`sql_core-3` |
| `RocksDBLineageSuite` | n/a | n/a | n/a | runs | `sql_core-1` |
So one suite out of six is skipped, on one Spark version out of four, and
the asymmetry looks
accidental rather than deliberate.
There is already precedent for skipping this family. `StateStoreSuite` -- the
`StateStoreSuiteBase[HDFSBackedStateStoreProvider]` twin of
`RocksDBStateStoreSuite` -- is
skipped under Comet on 4.0 and 4.1, with the comment "The tests target
streaming state-store
internals (StateStore.get/put/commit), not SQL execution paths, and the
`maintenance` test is
flaky in CI" (#4221). The same reasoning applies verbatim to the RocksDB
provider, which is the
heavier of the two.
These suites are not cheap. `RocksDBSuite` is ~4000 lines and `@SlowSQLTest`,
`RocksDBStateStoreSuite` is `@ExtendedSQLTest`, and
`AlsoTestWithRocksDBFeatures` /
`AlsoTestWithEncodingTypes` multiply each test case across
changelog-checkpointing, column-family
and encoding permutations. They land in all three `sql_core` shards, which
are the rows already
tuned for the 7 GB runner budget.
### Describe the potential solution
Mix `IgnoreCometSuite` into the RocksDB state-store suites in all four
`dev/diffs/*.diff` files,
so the family is skipped consistently rather than on one version only.
Leave `RocksDBStateStoreLockHardeningSuite` and `RocksDBStateEncoderSuite`
running: they are
plain `SparkFunSuite`s that never start a `SparkSession`, so Comet is never
loaded and skipping
them would only cost upstream regression detection for nothing.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
