FrankChen021 opened a new issue, #20414: URL: https://github.com/apache/druid/issues/20414
_This issue was generated automatically by Claude Code (Anthropic's AI coding agent) running a scheduled CI-triage routine on behalf of @FrankChen021. Analysis and suggested fixes are AI-produced; please verify before acting on them._ This triage covers the 8 commits merged to master on 2026-09-23 (local time, UTC+8). Three commits had at least one failed push-triggered job (four failed jobs in total). None of the failures is caused by the commit itself: all are flaky tests or a CI infrastructure hiccup. Each failing shard passed on the PR's own pre-merge run and on the neighbouring master commits, and none of the three commits touches the failing code (two add Google/Azure deep-storage features, one bumps `nimbus-jose-jwt`). No re-runs had been triggered on any failed job. ### Summary | Commit | Failed job | Failure log | Root cause | Verdict | |---|---|---|---|---| | a379ea849f (#20397, build(deps): bump com.nimbusds:nimbus-jose-jwt from 10.9.1 to 10.10) | `openrewrite` | [job 107010233252](https://github.com/apache/druid/actions/runs/35807088352/job/107010233252) | Maven could not resolve the parent POM `org.apache:apache:pom:39` ("Could not transfer artifact ... from/to ${repoOrgId} (${repoOrgUrl})"), so the build aborted before OpenRewrite ran | Infra | | a379ea849f (#20397, build(deps): bump com.nimbusds:nimbus-jose-jwt from 10.9.1 to 10.10) | `unit tests (25, K*,U*,Z*,Y*,X*)` | [job 107010233701](https://github.com/apache/druid/actions/runs/35807088521/job/107010233701) | `KinesisFaultToleranceTest.test_supervisorRecovers_afterSuspendResume_withEmptyShards`: `SELECT COUNT(*)` returned 2001 vs expected 3000 right after 3000 rows were reported published, failed 1/1 attempt | Flaky (race) | | 15051a5c09 (#20382, push/pull/kill segments in GCS without zip) | `unit tests (25, H*,D*,T*,O*)` | [job 106879002642](https://github.com/apache/druid/actions/runs/35767013909/job/106879002642) | `TaskQueueScaleTest.doMassLaunchAndExit`: `running+pending+waiting` count 995/997/998/998 vs expected 1000, failed all 4 surefire attempts | Flaky (race) (fix in #20291) | | 4eede66fcf (#20380, Azure SegmentRangeReader for partial loads) | `unit tests (25, K*,U*,Z*,Y*,X*)` | [job 106878085304](https://github.com/apache/druid/actions/runs/35766742278/job/106878085304) | `KinesisFaultToleranceTest.test_supervisorRecovers_afterChangeInTopicPartitions_withEmptyShards`: timed out after 240s waiting for `ingest/rows/published` to reach 2000 in `verifyAndTearDown`, failed 1/1 attempt | Flaky | ### Analysis and suggested fixes **1. `TaskQueueScaleTest.doMassLaunchAndExit` (indexing-service, H\*,D\*,T\*,O\* shard)** This is the same count-snapshot race already triaged in #20312 and #20366. After adding 1000 `NoopTask`s, the test sums `getRunningTaskCount()`, `getPendingTaskCount()` and `getWaitingTaskCount()`, which are three separate snapshots taken while the 1 ms management loop is still moving tasks from waiting to pending to running. A task that moves between two snapshots is counted in neither; the test's own comment acknowledges this (`TaskQueueScaleTest.java:182`). The four attempts missed 5/3/2/2 tasks, a different number each time, which points to timing rather than a logic bug. #20382 only touches `extensions-core/google-extensions`, and the H\* shard passed on the PR's pre-merge run (35393834449) and on the neighbouring master commits 4eede66fcf and a379ea849f. Suggested fix: merge #20291, which removes the cross-snapshot sum assertion and waits for cleanup instead of sleeping for a fixed time. If the assertion is kept, take all three counts under the queue's lock in a single call, or assert only `taskQueue.getTasks().size() == numTasks`. **2. `KinesisFaultToleranceTest.test_supervisorRecovers_afterSuspendResume_withEmptyShards` (embedded-tests, K\*,U\*,Z\*,Y\*,X\* shard)** `verifyAndTearDown` waits for `ingest/rows/published` to sum to 3000 (this passed), suspends the supervisor, deletes the stream, and then calls `verifyRowCount`, which first calls `waitForAllSegmentsToBeAvailable` and then runs `SELECT COUNT(*)`. `waitForAllSegmentsToBeAvailable` reads the used-segment count from metadata and waits until the cumulative Broker `segment/schemaCache/refresh/count` reaches that number. The metric also counts earlier refreshes of realtime segments served by the indexer, so the threshold can be met before the segments handed off from the last batch are loaded and announced by the Historical. Meanwhile the suspend has just stopped the realtime tasks, which drops their segments from the Broker timeline. The query therefore sees only the first two batches (2001 rows). #20397 is a pure `nimbus-jose-jwt` version bump; the K\* shard passed on its pre-merge run (35684318646) and on the next master commits (a9eae5baab onwards). Suggested fix: in `StreamIndexTestBase.verifyRowCount`, switch to the new `EmbeddedClusterApis.waitForAllSegmentsToBeQueryable` added in #20319 (b7b560dfd2), which polls `sys.segments` instead of counting refresh metrics. Alternatively, poll `SELECT COUNT(*)` until it equals `expectedRowCount` within a timeout rather than asserting once. **3. `KinesisFaultToleranceTest.test_supervisorRecovers_afterChangeInTopicPartitions_withEmptyShards` (embedded-tests, K\*,U\*,Z\*,Y\*,X\* shard)** The test publishes 1000 records to a single Kinesis shard, reshards the stream from 2 to 4 shards, publishes another 1000 records to one shard, and then waits up to 240s for 2000 published rows. Resharding closes the parent shards. The supervisor then has to notice the closed and empty shards, let the reading tasks finish the closed shard, publish, and start new tasks on the child shards, all on the emulated Kinesis backend on a busy runner. The wait timed out, so either handoff after the reshard never finished or it needed more than 240s on this run. The sibling `test_supervisorRecovers_afterChangeInTopicPartitions` (non-empty shards) and the other 6 tests in the class passed in the same run. #20380 only touches `extensions-core/azure-extensions`; the K\* shard passed on its pre-merge run (35392600777) and on the next master commit 15051a5c09. Suggested fix: before waiting on the final aggregate, wait for the supervisor to register the new shard set (e.g. poll the supervisor status until the partition count is 4, or wait for the task-completion metric from the tasks reading the closed shards). On timeout, log the supervisor status and the current `ingest/rows/published` sum so that the next occurrence shows whether the tasks were stuck or just slow. **4. `openrewrite` job: parent POM `org.apache:apache:39` could not be resolved (Static Checks CI)** Maven failed while building the project model: `Non-resolvable parent POM ... org.apache:apache:pom:39 (absent): Could not transfer artifact ... from/to ${repoOrgId} (${repoOrgUrl})`. The unexpanded `${repoOrgId}`/`${repoOrgUrl}` placeholders mean Maven fell back to a repository declared with unresolved properties after the normal download of the Apache parent POM failed or the cached copy was marked absent. This is a runner or cache problem, not a problem with the change: #20397 does not touch the root `pom.xml` parent, and the `openrewrite` job passed on the next master commit (a9eae5baab), one minute later, with the same parent version. Master has since moved to a newer `org.apache:apache` parent (f541ae200c). Suggested fix: none needed if it does not recur. If it does, add `-U` or a retry around the dependency resolution step in `.github/scripts/openrewrite.sh`, and check whether a cached `~/.m2` entry for `org.apache:apache` can be saved in a "not found" state (clear `*.lastUpdated` files before the build). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
