debu99 opened a new issue, #12608:
URL: https://github.com/apache/iceberg/issues/12608
### Feature Request / Improvement
# Improve coordinator election logging in Iceberg Kafka Connect Sink
## Problem Description
The Iceberg Kafka Connect Sink connector fails silently when there's a
mismatch between:
- The Kafka Connect consumer group ID (`consumer.group.id`)
- The Iceberg connector control topic group ID (`iceberg.connect.group-id`)
When this mismatch occurs, no coordinator is elected and no commits happen,
but there are no clear error messages to help users identify the issue.
## Impact
When group IDs don't match:
1. No coordinator is elected
2. No START_COMMIT events are sent
3. Data is consumed but not committed to Iceberg tables
4. Users have limited visibility into why coordination isn't working
## Steps to Reproduce
The issue can be observed in these scenarios:
### Scenario 1: Explicit Different Group IDs
Configure Kafka Connect sink connector with:
```properties
consumer.group.id=connect-sink-group
iceberg.connect.group-id=different-group-name
```
### Scenario 2: Using Default Group ID (More Common)
Configure Kafka Connect sink connector with only:
```properties
consumer.group.id=connect-sink-group
# iceberg.connect.group-id not set, will use default value
'connect-iceberg-sink'
```
Key code references:
```java
// In CommitterImpl.java - Coordinator election process
private boolean hasLeaderPartition(Collection<TopicPartition>
currentAssignedPartitions) {
ConsumerGroupDescription groupDesc;
try (Admin admin = clientFactory.createAdmin()) {
groupDesc =
KafkaUtils.consumerGroupDescription(config.connectGroupId(), admin);
}
...
// In IcebergSinkConfig.java - Default group ID configuration
public static final String CONNECT_GROUP_ID = "iceberg.connect.group-id";
public static final String CONNECT_GROUP_ID_DEFAULT = "connect-iceberg-sink";
```
In both scenarios:
1. Start the connector
2. Observe that:
- Data is being consumed
- No commits are happening
- No clear error messages about group ID mismatch
## Root Cause
The `hasLeaderPartition` method in `CommitterImpl` checks for the control
topic's consumer group existence but doesn't validate if it matches the actual
Kafka Connect consumer group ID. This leads to a situation where:
1. The connector looks for members in the wrong consumer group (either
explicitly configured or default `connect-iceberg-sink`)
2. No coordinator is elected because the group appears empty
3. The actual consumer group (where data is being consumed) is different
## Proposed Solution
1. Add appropriate logging when consumer group is not found
2. Provide clear messages to help users identify potential group ID
mismatches
## Compatibility
This change is backwards compatible as it only improves logging without
changing any behavior.
### Query engine
None
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with
guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
