huan233usc opened a new pull request, #16976: URL: https://github.com/apache/iceberg/pull/16976
## Problem `TestStructuredStreamingRead3` is the largest single class in the Spark CI wall-time report (~1000s). Profiling shows the cost is **not** in the streaming reads (each `processAllAvailable` is <1s) — it's in **teardown**. The class sets `STREAMING_SNAPSHOT_POLLING_INTERVAL_MS=1` for the `async=true` parameter, so `AsyncSparkMicroBatchPlanner`'s background thread refreshes the table from the catalog ~1000×/second. The class also has two `@AfterEach` methods — `stopStreams()` and `removeTables()` — whose relative order is not guaranteed. When `DROP TABLE` runs while the planner's background thread is still alive, that flood of catalog refreshes contends with the drop and stalls teardown by **~20s per async test execution**. Measured (instrumenting teardown): ``` DROP TABLE, async=true : 20142 ms DROP TABLE, async=false : 9 ms ``` ## Change Stop active streams before dropping the table in `removeTables()`, so the background refresh thread is gone before the drop. One-line behavioral change; `stopStreams()` is unchanged and still runs as its own `@AfterEach`. ## Result Full-class `TestStructuredStreamingRead3` on spark v3.5: **~305s → ~188s** (66 tests, 0 failures). Applied identically to v3.5, v4.0 and v4.1 (v4.0/v4.1 smoke-tested green). > Note: the underlying 1ms polling interval refreshing the catalog regardless of need is a planner-side smell worth a separate look; this PR just makes the test teardown robust to it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
