JRobTS opened a new issue, #20696:
URL: https://github.com/apache/druid/issues/20696
During Coordinator shutdown (e.g. during a rolling update on Kubernetes) the
Coordinator fails to exchange leadership. This results in peons being shutdown
and task logs falling into the ether.
### Affected Version
37.0.0
### Description
I've tested this in an MM-less cluster with 2 coordinators (separate from
overlords).
It looks to me like the leadership election gets interrupted by the shutdown
sequence itself. We have given ample `terminationGracePeriodSeconds` to the
Coordinator so there isn't a SIGKILL happening. It appears as if there could be
a race condition where the SIGTERM terminates the leadership election process
before it has a chance to finish. Either that or the new Coordinator isn't
actually ready to accept leadership even though it is passing it's probe
(`/status/health`).
Please include as much detailed information about the problem as possible.
- Cluster size
- Configurations in use
- Steps to reproduce the problem
- The error message or stack traces encountered. Providing more context,
such as nearby log messages or even entire logs, can be helpful.
- Any debugging that you have already done
Logs:
```
2026-10-22T22:57:59,615 INFO [LeaderElectorAsyncWrapper]
io.kubernetes.client.extended.leaderelection.LeaderElector - Successfully
acquired lease, became leader
2026-10-22T22:57:59,616 INFO [leader-elector-hook-worker-1]
org.apache.druid.server.coordinator.DruidCoordinator - I am the leader of the
coordinators, all must bow! Starting coordinatio
n in [PT30S].
2026-10-22T22:57:59,616 INFO [leader-elector-hook-worker-1]
org.apache.druid.metadata.segment.SqlSegmentsMetadataManagerV2 - Using segments
in metadata cache to build timeline.
2026-10-22T22:57:59,616 INFO [leader-elector-hook-worker-1]
org.apache.druid.metadata.segment.cache.HeapMemorySegmentMetadataCache - We are
now leader. Cache is now in state[LEADER_FIRST
_SYNC_PENDING].
2026-10-22T22:57:59,617 INFO [pool-30-thread-1]
org.apache.druid.metadata.SqlSegmentsMetadataManager - Populating column
'used_status_last_updated' with non-NULL values for unused segmen
ts in table[druid_segments].
2026-10-22T22:57:59,617 INFO [SegmentMetadataCache-0]
org.apache.druid.metadata.segment.cache.HeapMemorySegmentMetadataCache - Sync
with metadata store was cancelled (java.util.concurren
t.CancellationException: Task was cancelled.)
2026-10-22T22:57:59,619 INFO [pool-30-thread-1]
org.apache.druid.metadata.SqlSegmentsMetadataManager - Populated column
'used_status_last_updated' in table[druid_segments] in [0] rows.
2026-10-22T22:57:59,623 INFO [leader-elector-hook-worker-1]
org.apache.druid.server.http.BaseDynamicConfigSyncer - Starting [coordinator]
config sync to brokers on leader node.
2026-10-22T22:57:59,623 INFO [leader-elector-hook-worker-1]
org.apache.druid.server.http.BaseDynamicConfigSyncer - Starting [broker] config
sync to brokers on leader node.
2026-10-22T22:57:59,629 INFO [leader-elector-hook-worker-1]
org.apache.druid.server.coordinator.duty.CoordinatorDutyGroup - Created
dutyGroup[HistoricalManagementDuties] with period[PT30
S] and
duties[[org.apache.druid.server.coordinator.duty.PrepareBalancerAndLoadQueues,
org.apache.druid.server.coordinator.duty.RunRules,
org.apache.druid.server.coordinator.DruidCoordina
tor$UpdateReplicationStatus,
org.apache.druid.server.coordinator.DruidCoordinator$CollectSegmentStats,
org.apache.druid.server.coordinator.duty.UnloadUnusedSegments,
org.apache.druid.ser
ver.coordinator.duty.MarkOvershadowedSegmentsAsUnused,
org.apache.druid.server.coordinator.duty.MarkEternityTombstonesAsUnused,
org.apache.druid.server.coordinator.duty.BalanceSegments,
org.apache.druid.server.coordinator.duty.CloneHistoricals,
org.apache.druid.server.coordinator.DruidCoordinator$CollectLoadQueueStats]].
2026-10-22T22:57:59,631 INFO [leader-elector-hook-worker-1]
org.apache.druid.server.coordinator.duty.KillUnusedSegments - Kill task
scheduling enabled with period[PT600S], durationToReta
in[PT86400S], bufferPeriod[PT21600S], maxSegmentsToKill[10000],
maxIntervalToKill[P30D]
2026-10-22T22:57:59,632 INFO [leader-elector-hook-worker-1]
org.apache.druid.server.coordinator.duty.CoordinatorDutyGroup - Created
dutyGroup[IndexingServiceDuties] with period[PT600S] a
nd duties[[org.apache.druid.server.coordinator.duty.KillUnusedSegments,
org.apache.druid.server.coordinator.duty.KillStalePendingSegments,
org.apache.druid.server.coordinator.duty.Compac
tSegments]].
2026-10-22T22:57:59,633 INFO [leader-elector-hook-worker-1]
org.apache.druid.server.coordinator.duty.CoordinatorDutyGroup - Created
dutyGroup[MetadataStoreManagementDuties] with period[P
T3600S] and
duties[[org.apache.druid.server.coordinator.duty.KillSupervisors,
org.apache.druid.server.coordinator.duty.KillAuditLog,
org.apache.druid.server.coordinator.duty.KillRules, o
rg.apache.druid.server.coordinator.duty.KillDatasourceMetadata,
org.apache.druid.server.coordinator.duty.KillCompactionConfig]].
2026-10-22T22:57:59,633 WARN [leader-elector-hook-worker-1]
org.apache.druid.server.coordinator.DruidCoordinator - Created [3] duty groups.
DUTY RUNS WILL NOT BE LOGGED. Use API '/druid/
coordinator/v1/duties' to get current status.
2026-10-22T22:58:00,617 INFO [SegmentMetadataCache-0]
org.apache.druid.metadata.segment.cache.HeapMemorySegmentMetadataCache -
Started sync of latest updates from metadata store. Cache i
s now in state[LEADER_FIRST_SYNC_STARTED].
2026-10-22T22:58:00,681 INFO [SegmentMetadataCache-0]
org.apache.druid.metadata.segment.cache.HeapMemorySegmentMetadataCache -
Finished sync with metadata store in [63] millis. Cache is
now in state[LEADER_READY].
2026-10-22T22:58:01,623 INFO [LookupCoordinatorManager--0]
org.apache.druid.server.lookup.cache.LookupCoordinatorManager - Not updating
lookups because no data exists
2026-10-22T22:58:11,576 INFO [Thread-48]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Lifecycle [module]
running shutdown hook
2026-10-22T22:58:11,576 INFO [Thread-48]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [ANNOUNCEMENTS] for handlers [[org.apache.druid.cli.Ser
verRunnable$DiscoverySideEffectsProvider$1@7c0b792e]]
2026-10-22T22:58:11,576 INFO [Thread-48]
org.apache.druid.k8s.discovery.K8sDruidNodeAnnouncer - Unannouncing
DiscoveryDruidNode[DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/
coordinator', host='10.251.36.138', bindOnHost=false, port=-1,
plaintextPort=8081, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false,
labels=null}, nodeRole='COORDINATOR', servic
es={}', startTime=2026-10-22T06:43:36.553Z, availableProcessors=1,
totalMemory=4294967296}]
2026-10-22T22:58:11,668 INFO [Thread-48]
org.apache.druid.k8s.discovery.K8sDruidNodeAnnouncer - Unannounced
DiscoveryDruidNode[DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/c
oordinator', host='10.251.36.138', bindOnHost=false, port=-1,
plaintextPort=8081, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false,
labels=null}, nodeRole='COORDINATOR', service
s={}', startTime=2026-10-22T06:43:36.553Z, availableProcessors=1,
totalMemory=4294967296}]
2026-10-22T22:58:11,668 INFO [Thread-48]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [SERVER] for handlers [[org.apache.druid.server.http.Se
lfDiscoveryResource$1@39b95a80,
org.apache.druid.server.initialization.jetty.JettyServerModule$2@4b9ed99d]]
2026-10-22T22:58:11,668 INFO [Thread-48] org.eclipse.jetty.server.Server -
Stopped oejs.Server@7388486e{STOPPING}[12.1.8,sto=0]
2026-10-22T22:58:11,670 INFO [Thread-48]
org.eclipse.jetty.server.AbstractConnector - Stopped
oejs.ServerConnector@5fe024a7{HTTP/1.1, (http/1.1)}{0.0.0.0:8081}
2026-10-22T22:58:11,678 INFO [Thread-48]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [NORMAL] for handlers [[org.apache.druid.java.util.comm
on.concurrent.ExecutorServices$1@2809cf3c,
org.apache.druid.java.util.common.concurrent.ExecutorServices$1@753aede,
org.apache.druid.java.util.common.concurrent.ExecutorServices$1@47d584
24,
org.apache.druid.guice.http.JettyHttpClientModule$HttpClientProvider$1@7f0f896c,
org.apache.druid.java.util.common.concurrent.ExecutorServices$1@1999c99e,
org.apache.druid.java.util.
common.concurrent.ExecutorServices$1@3a6b12d, DruidCoordinator,
BrokerDynamicConfigSyncer, CoordinatorDynamicConfigSyncer,
org.apache.druid.java.util.common.concurrent.ExecutorServices$1
@6b33892a,
org.apache.druid.java.util.common.concurrent.ExecutorServices$1@335896bd,
DiscoveryServiceLocator, NettyHttpClient,
org.apache.druid.java.util.http.client.HttpClientInit$2@58e
5fbe5, org.apache.druid.java.util.http.client.HttpClientInit$1@1293f8d7,
IndexerSQLMetadataStorageCoordinator, SqlIndexingStateStorage,
SQLMetadataRuleManager, org.apache.druid.metadata.
SQLMetadataRuleManagerProvider$1@54e18a46, SQLMetadataSupervisorManager,
SqlSegmentsMetadataManagerV2,
org.apache.druid.metadata.SqlSegmentsMetadataManagerProvider$1@67d180e4,
org.apache
.druid.java.util.common.concurrent.ExecutorServices$1@7816454d,
ConfigManager, org.apache.druid.guice.JacksonConfigManagerModule$1@7096d451,
SQLAuditManager, CoordinatorServerView, HttpS
erverInventoryView, K8sDruidNodeDiscoveryProvider, NettyHttpClient,
org.apache.druid.java.util.http.client.HttpClientInit$2@78e97d4d,
org.apache.druid.java.util.http.client.HttpClientIni
t$1@758d4aa9, BasicMonitorScheduler, OshiSysMonitor, JvmMonitor,
JvmCpuMonitor, JvmThreadsMonitor, ServiceEmitter, SwitchingEmitter,
LoggingEmitter, KafkaEmitter, InfluxdbEmitter]]
2026-10-22T22:58:11,679 INFO [Thread-48]
org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler -
Stopping lifecyle [DruidCoordinator#stop].
2026-10-22T22:58:11,679 ERROR [LeaderElectorAsyncWrapper]
io.kubernetes.client.extended.leaderelection.LeaderElector - LeaderElection
renew loop gets interrupted
java.lang.InterruptedException: sleep interrupted
at java.base/java.lang.Thread.sleep(Native Method)
at
io.kubernetes.client.extended.leaderelection.LeaderElector.renewLoop(LeaderElector.java:264)
at
io.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:164)
at
io.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:129)
at
org.apache.druid.k8s.discovery.DefaultK8sLeaderElectorFactory$1.run(DefaultK8sLeaderElectorFactory.java:77)
at
org.apache.druid.k8s.discovery.LeaderElectorAsyncWrapper.lambda$run$0(LeaderElectorAsyncWrapper.java:81)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2026-10-22T22:58:11,679 INFO [LeaderElectorAsyncWrapper]
io.kubernetes.client.extended.leaderelection.LeaderElector - Failed to renew
lease, lose leadership
2026-10-22T22:58:21,679 WARN [Thread-48]
org.apache.druid.k8s.discovery.LeaderElectorAsyncWrapper - Failed to terminate
[LeaderElectorAsyncWrapper] executor.
2026-10-22T22:58:21,685 INFO [Thread-48]
org.apache.druid.metadata.segment.cache.HeapMemorySegmentMetadataCache -
Stopped sync with metadata store. Cache is now in state[STOPPED].
2026-10-22T22:58:21,685 INFO [Thread-48]
org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler -
Stopping lifecyle [BrokerDynamicConfigSyncer#stop].
2026-10-22T22:58:21,685 INFO [LeaderElectorAsyncWrapper]
org.apache.druid.server.coordinator.DruidCoordinator - I am no longer the
leader...
2026-10-22T22:58:21,685 INFO [Thread-48]
org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler -
Stopping lifecyle [CoordinatorDynamicConfigSyncer#stop].
2026-10-22T22:58:21,685 INFO [Thread-48]
org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler -
Stopping lifecyle [DiscoveryServiceLocator#close].
2026-10-22T22:58:21,685 INFO [Thread-48]
org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler -
Stopping lifecyle [NettyHttpClient#stop].
2026-10-22T22:58:21,685 INFO [LeaderElectorAsyncWrapper]
org.apache.druid.server.coordinator.loading.LoadQueueTaskMaster - Stopping load
queue peon for [0] servers.
2026-10-22T22:58:21,686 INFO [LeaderElectorAsyncWrapper]
org.apache.druid.server.http.BaseDynamicConfigSyncer - Not leader, stopping
[coordinator] config sync to brokers.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
