flysen opened a new pull request, #18030: URL: https://github.com/apache/iceberg/pull/18030
The Worker's control-topic reader used a single-member, never-reused, randomly-named consumer group via `consumer.subscribe()`. That group gets no benefit from consumer-group management (it never has more than one member) but is still exposed to the full JoinGroup/SyncGroup protocol, including broker-side member-fencing failures that can permanently starve a task's commits with no error or warning logged anywhere. Switch to `consumer.assign()` for this reader, which never registers a consumer group with the broker at all, making this failure mode structurally impossible rather than just less likely. `Coordinator` is unaffected: its control-topic consumer uses a stable, reused group id and relies on real group membership for rebalance-based leader detection, so it keeps using `subscribe()`. Also hardens the new `assign()`-based path against the control topic not existing yet at Worker startup (`subscribe()` tolerates this; a one-shot `assign()` does not), using `Tasks` for the bounded retry to match this package's existing retry idiom (see `Coordinator.java`). Related: #15293, #16016 (open issues describing related control-topic/coordinator instability under rebalance; this fixes a distinct root cause found while investigating the same symptom class). Adds a Testcontainers-based regression test asserting via `AdminClient.listConsumerGroups()` that the Worker never registers an ephemeral control-topic consumer group. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
