andygrove opened a new issue, #6890:
URL: https://github.com/apache/datafusion-comet/issues/6890
### Describe the bug
`org.apache.spark.sql.execution.ui.SQLAppStatusListenerMemoryLeakSuite` ("no
memory leak"), which runs in the `sql_core-1` shard of the Spark SQL tests,
fails intermittently:
```
[info] - no memory leak *** FAILED *** (1 second, 338 milliseconds)
[info] statusStore.listener.get.noLiveData() was false
(SQLAppStatusListenerSuite.scala:1123)
```
It is the only failing test in the shard when it happens (nightly
2026-10-10, Spark 4.0: `Tests: succeeded 7847, failed 1`).
The cause is a race in the Spark test itself. Upstream Spark fixed it on
`master`, but none of the Spark versions Comet's tests run against has the fix.
**The race.** In `DAGScheduler.failJobAndIndependentStages`, the job waiter
is notified before `SparkListenerJobEnd` is posted:
```scala
cleanupStateForJobAndIndependentStages(job)
job.listener.jobFailed(error) // unblocks the failing action on the test
thread
listenerBus.post(SparkListenerJobEnd(job.jobId, clock.getTimeMillis(),
JobFailed(error)))
```
The test runs 100 failing jobs (`df.foreach(_ => throw new
RuntimeException("Oops"))`), then calls `sc.listenerBus.waitUntilEmpty()` and
asserts `noLiveData()`. If the last `JobEnd` is posted after the bus has been
drained, the last failed execution never reaches its end-event count and stays
in `liveExecutions`, so the assertion sees live data. In the 2026-10-10 log the
assertion fails 22 ms after the last `Oops` task failure, and the shard log has
no dropped-event messages.
### Steps to reproduce
Not deterministic. It occurred in 2 of 252 CI executions of the shard (see
below). Upstream reports that it reproduces under SBT when the test is run
repeatedly.
### Expected behavior
The suite passes consistently.
### Additional context
**Sightings.** Every nightly, merge-queue and label run from 2026-10-02 to
2026-10-12 was checked. The `sql_core-1` job ran 252 times and the test failed
in 2 of them, about 0.8%. Commit runs of PRs that carry a `run-*` label also
execute the shards and were not counted.
| Date | Run | Job | Result |
| --- | --- | --- | --- |
| 2026-10-06 | [label run on
#6709](https://github.com/apache/datafusion-comet/actions/runs/37416673887),
head 4cf4f406b | Spark 4.1 `sql_core-1` | `succeeded 8334, failed 1`. The
`Comet CI` run at the same head passed. |
| 2026-10-10 | [nightly on
`main`](https://github.com/apache/datafusion-comet/actions/runs/38030901791),
head bfacc43a5 | Spark 4.0 `sql_core-1` | `succeeded 7847, failed 1`. The Spark
3.5 and 4.2 `sql_core-1` jobs of the same nightly passed. The nightly's
auto-filed issue is #6848. |
By run type: 1 of 32 nightly executions, 1 of 39 label-run executions and 0
of 181 merge-queue executions. The other failed `sql_core-1` jobs in the window
failed other tests on PR branches, or had no failed step and no log (a lost
runner).
**Impact.** CI only. It turns the nightly or a label run red. In the merge
queue it would evict the PR, which has not happened yet.
**Every Spark version Comet tests has the race.** The assertion is unguarded
and the `DAGScheduler` order above is the same in 3.4.3, 3.5.9, 4.0.4, 4.1.3
and 4.2.0, and in the 4.0.5-rc2, 4.1.4-rc2 and 4.2.1-rc1 release candidates. A
patch-version bump will not fix it.
**Upstream fix.**
[apache/spark#56790](https://github.com/apache/spark/pull/56790) (SPARK-57711)
wraps the final assertion in `eventually(timeout(5.seconds),
interval(10.milliseconds))`. It was merged to `master` and `branch-4.x` on
2026-06-27 and is in 4.3.0-rc1. It is not on `branch-4.2`, `branch-4.1`,
`branch-4.0` or `branch-3.5`. Upstream fixed the test, and the scheduler order
is unchanged in 4.3.0-rc1.
**Is Comet involved?** The suite builds its own `SparkContext`, and #6415
(2026-10-02) made suites like it run with Comet. The race is in Spark's
scheduler and listener bus, and upstream hit the same failure on plain Spark
SBT builds ([example
run](https://github.com/apache/spark/actions/runs/28000645341)). Whether
running with Comet makes the window wider has not been measured.
**Options.**
1. Carry the upstream test change (one import and the `eventually` wrapper)
in `dev/diffs/{3.4.3,3.5.9,4.0.4,4.1.3,4.2.0}.diff`. The diffs have to be
regenerated as described in the [Spark SQL tests
guide](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/spark-sql-tests.md),
not edited by hand. This is the smallest change and matches upstream.
2. Do nothing and re-run the failed job when it happens.
3. Mark the test `IgnoreComet`. This also drops the check that a Comet plan
leaves no live SQL status data behind, so it is the least attractive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
