andygrove commented on PR #4591: URL: https://github.com/apache/datafusion-comet/pull/4591#issuecomment-5080744268
The first full CI run after the `main` merge is failing. Every `[exec]` job fails on all Spark profiles (3.4, 3.5, 4.0, 4.1, 4.2, and macOS 4.0), all in `CometInMemoryCacheSuite`. 9 of the 11 tests fail identically across versions, so this is deterministic rather than flaky. The two tests that pass are the telling part: - `Comet cache serializer can read DefaultCachedBatch fallback data` (feature disabled, expects `DefaultCachedBatch`) - `Comet plugin respects user-provided cache serializer` (pure unit test, no cluster) Everything that fails is a test that enables the feature and then expects either `CometCachedBatch` in the cache or `CometInMemoryTableScan` in the plan: - `CometInMemoryTableScan over CometCachedBatch` (`:79`): `cachedBatchTypes` was not `CometCachedBatch` - `Comet in-memory cache disabled keeps SparkToColumnar fallback path` (`:112`): fails in its first block, which enables the feature and expects `CometCachedBatch` - `handles multi-partition cache` (`:203`), `projection-only read` (`:266`), `shuffle after cache read` (`:299`), `stats-based batch pruning` (`:333`): all `cachedBatchTypes` mismatches - `handles empty cache` (`:240`), `empty projection scan` (`:440`), `NaN floating-point pruning` (`:494`): plan `did not contain CometInMemoryTableScan` So when the feature is enabled, the cache is being materialized in Spark's default format instead of `CometCachedBatch`, and the native scan is never selected. The fallback read path and the plugin wiring both work, which localizes the break to the enabled write path or the native-scan selection, not the read path. The place I would look first is the write-path gating. `supportsColumnarInput` returns `enabled(SQLConf.get)` and the row path in `convertInternalRowToCachedBatch` gates on `enabled(conf)`, so if the enabled flag is not visible as `true` at cache-materialization time under this configuration, both write paths fall back to `DefaultCachedBatch` and the observed symptoms follow. Worth confirming whether these pass locally against current `main`, since this is the first run that includes the merge. Failing job (Spark 3.5): https://github.com/apache/datafusion-comet/actions/runs/30173884134/job/89722848819 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
