michalantkowicz commented on issue #13752:
URL: https://github.com/apache/iceberg/issues/13752#issuecomment-3160241801
oh actually it seems that the same pod is rejoining to the `control-iceberg`
twice
```
GROUP TOPIC PARTITION
CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
CLIENT-ID
connect-mantkowicz-test-iceberg-coord control-iceberg 0 457321
457486 165 6124a0d2-be09-... /10.4.79.156
6124a0d2-be09-...
connect-mantkowicz-test-iceberg-coord control-iceberg 1 675247
675362 115 6124a0d2-be09-... /10.4.79.156
6124a0d2-be09-...
connect-mantkowicz-test-iceberg-coord control-iceberg 2 603218
603225 7 aa726279-97e7-... /10.4.79.156
aa726279-97e7-...
GROUP TOPIC PARTITION
CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
CLIENT-ID
connect-mantkowicz-test-iceberg-coord control-iceberg 0 457321
457490 169 6124a0d2-be09-... /10.4.79.156
6124a0d2-be09-...
connect-mantkowicz-test-iceberg-coord control-iceberg 1 675247
675362 115 6124a0d2-be09-... /10.4.79.156
6124a0d2-be09-...
connect-mantkowicz-test-iceberg-coord control-iceberg 2 603224
603225 1 aa726279-97e7-... /10.4.79.156
aa726279-97e7-...
GROUP TOPIC PARTITION
CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
CLIENT-ID
connect-mantkowicz-test-iceberg-coord control-iceberg 0 457321
457490 169 6124a0d2-be09-... /10.4.79.156
6124a0d2-be09-...
connect-mantkowicz-test-iceberg-coord control-iceberg 1 675247
675362 115 6124a0d2-be09-... /10.4.79.156
6124a0d2-be09-...
connect-mantkowicz-test-iceberg-coord control-iceberg 2 603224
603225 1 aa726279-97e7-... /10.4.79.156
aa726279-97e7-...
GROUP TOPIC PARTITION
CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
CLIENT-ID
connect-mantkowicz-test-iceberg-coord control-iceberg 0 457321
457490 169 6124a0d2-be09-... /10.4.79.156
6124a0d2-be09-...
connect-mantkowicz-test-iceberg-coord control-iceberg 1 675247
675362 115 6124a0d2-be09-... /10.4.79.156
6124a0d2-be09-...
connect-mantkowicz-test-iceberg-coord control-iceberg 2 603224
603225 1 aa726279-97e7-... /10.4.79.156
aa726279-97e7-...
GROUP TOPIC PARTITION
CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
CLIENT-ID
connect-mantkowicz-test-iceberg-coord control-iceberg 0 457321
457490 169 6124a0d2-be09-... /10.4.79.156
6124a0d2-be09-...
connect-mantkowicz-test-iceberg-coord control-iceberg 1 675247
675362 115 6124a0d2-be09-... /10.4.79.156
6124a0d2-be09-...
connect-mantkowicz-test-iceberg-coord control-iceberg 2 603224
603225 1 aa726279-97e7-... /10.4.79.156
aa726279-97e7-...
Warning: Consumer group 'connect-mantkowicz-test-iceberg-coord' is
rebalancing.
GROUP TOPIC PARTITION
CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
CLIENT-ID
connect-mantkowicz-test-iceberg-coord control-iceberg 0 457321
457490 169 - - -
connect-mantkowicz-test-iceberg-coord control-iceberg 1 675247
675362 115 - - -
connect-mantkowicz-test-iceberg-coord control-iceberg 2 603224
603225 1 - - -
GROUP TOPIC PARTITION
CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
CLIENT-ID
connect-mantkowicz-test-iceberg-coord control-iceberg 0 457321
457490 169 6124a0d2-be09-... /10.4.79.156
6124a0d2-be09-...
connect-mantkowicz-test-iceberg-coord control-iceberg 1 675247
675362 115 6124a0d2-be09-... /10.4.79.156
6124a0d2-be09-...
connect-mantkowicz-test-iceberg-coord control-iceberg 2 603224
603225 1 6124a0d2-be09-... /10.4.79.156
6124a0d2-be09-...
Warning: Consumer group 'connect-mantkowicz-test-iceberg-coord' is
rebalancing.
GROUP TOPIC PARTITION
CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
CLIENT-ID
connect-mantkowicz-test-iceberg-coord control-iceberg 0 457321
457490 169 - - -
connect-mantkowicz-test-iceberg-coord control-iceberg 1 675247
675362 115 - - -
connect-mantkowicz-test-iceberg-coord control-iceberg 2 603224
603225 1 - - -
GROUP TOPIC PARTITION
CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
CLIENT-ID
connect-mantkowicz-test-iceberg-coord control-iceberg 0 457321
457490 169 1300dc38-a65e-... /10.4.79.156
1300dc38-a65e-...
connect-mantkowicz-test-iceberg-coord control-iceberg 1 675247
675364 117 1300dc38-a65e-... /10.4.79.156
1300dc38-a65e-...
connect-mantkowicz-test-iceberg-coord control-iceberg 2 603224
603225 1 6124a0d2-be09-... /10.4.79.156
6124a0d2-be09-...
GROUP TOPIC PARTITION
CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
CLIENT-ID
connect-mantkowicz-test-iceberg-coord control-iceberg 0 457321
457490 169 1300dc38-a65e-... /10.4.79.156
1300dc38-a65e-...
connect-mantkowicz-test-iceberg-coord control-iceberg 1 675247
675364 117 1300dc38-a65e-... /10.4.79.156
1300dc38-a65e-...
connect-mantkowicz-test-iceberg-coord control-iceberg 2 603224
603225 1 6124a0d2-be09-... /10.4.79.156
6124a0d2-be09-...
```
logs from the pod:
```
2025-08-06T13:29:16.053Z INFO [iceberg-coord ]
[o.a.k.c.c.i.ConsumerCoordinator ] {}: [Consumer
clientId=6124a0d2-be09-..., groupId=connect-mantkowicz-test-iceberg-coord]
Request joining group due to: group is already rebalancing
2025-08-06T13:29:16.053Z INFO [iceberg-coord ]
[o.a.k.c.c.i.ConsumerRebalanceListenerInvoker] {}: [Consumer
clientId=6124a0d2-be09-..., groupId=connect-mantkowicz-test-iceberg-coord]
Revoke previously assigned partitions control-iceberg-0, control-iceberg-1,
control-iceberg-2
2025-08-06T13:29:16.053Z INFO [iceberg-coord ]
[o.a.k.c.c.i.ConsumerCoordinator ] {}: [Consumer
clientId=6124a0d2-be09-..., groupId=connect-mantkowicz-test-iceberg-coord]
(Re-)joining group
2025-08-06T13:29:16.055Z INFO [iceberg-coord ]
[o.a.k.c.c.i.ConsumerCoordinator ] {}: [Consumer
clientId=1300dc38-a65e-..., groupId=connect-mantkowicz-test-iceberg-coord]
Successfully joined group with generation Generation{generationId=1895,
memberId='1300dc38-a65e-...', protocol='range'}
2025-08-06T13:29:16.055Z INFO [iceberg-coord ]
[o.a.k.c.c.i.ConsumerCoordinator ] {}: [Consumer
clientId=6124a0d2-be09-..., groupId=connect-mantkowicz-test-iceberg-coord]
Successfully joined group with generation Generation{generationId=1895,
memberId='6124a0d2-be09-...', protocol='range'}
2025-08-06T13:29:16.055Z INFO [iceberg-coord ]
[o.a.k.c.c.i.ConsumerCoordinator ] {}: [Consumer
clientId=6124a0d2-be09-..., groupId=connect-mantkowicz-test-iceberg-coord]
Finished assignment for group at generation 1895:
{1300dc38-a65e-...=Assignment(partitions=[control-iceberg-0,
control-iceberg-1]),
6124a0d2-be09-...=Assignment(partitions=[control-iceberg-2])}
2025-08-06T13:29:16.072Z INFO [iceberg-coord ]
[o.a.k.c.c.i.ConsumerCoordinator ] {}: [Consumer
clientId=1300dc38-a65e-..., groupId=connect-mantkowicz-test-iceberg-coord]
Successfully synced group in generation Generation{generationId=1895,
memberId='1300dc38-a65e-...', protocol='range'}
2025-08-06T13:29:16.072Z INFO [iceberg-coord ]
[o.a.k.c.c.i.ConsumerCoordinator ] {}: [Consumer
clientId=6124a0d2-be09-..., groupId=connect-mantkowicz-test-iceberg-coord]
Successfully synced group in generation Generation{generationId=1895,
memberId='6124a0d2-be09-...', protocol='range'}
2025-08-06T13:29:16.072Z INFO [iceberg-coord ]
[o.a.k.c.c.i.ConsumerCoordinator ] {}: [Consumer
clientId=6124a0d2-be09-..., groupId=connect-mantkowicz-test-iceberg-coord]
Notifying assignor about the new Assignment(partitions=[control-iceberg-2])
2025-08-06T13:29:16.072Z INFO [iceberg-coord ]
[o.a.k.c.c.i.ConsumerCoordinator ] {}: [Consumer
clientId=1300dc38-a65e-..., groupId=connect-mantkowicz-test-iceberg-coord]
Notifying assignor about the new Assignment(partitions=[control-iceberg-0,
control-iceberg-1])
2025-08-06T13:29:16.073Z INFO [iceberg-coord ]
[o.a.k.c.c.i.ConsumerRebalanceListenerInvoker] {}: [Consumer
clientId=6124a0d2-be09-..., groupId=connect-mantkowicz-test-iceberg-coord]
Adding newly assigned partitions: control-iceberg-2
2025-08-06T13:29:16.073Z INFO [iceberg-coord ]
[o.a.k.c.c.i.ConsumerRebalanceListenerInvoker] {}: [Consumer
clientId=1300dc38-a65e-..., groupId=connect-mantkowicz-test-iceberg-coord]
Adding newly assigned partitions: control-iceberg-0, control-iceberg-1
```
I saw that in the 'healthy' situation one of the consumer should
pro-actively leave the group like in this example:
```
2025-08-06T13:29:07.509Z INFO [iceberg-coord ]
[o.a.k.c.c.i.ConsumerRebalanceListenerInvoker] {}: [Consumer
clientId=aa726279-97e7-..., groupId=connect-mantkowicz-test-iceberg-coord]
Revoke previously assigned partitions control-iceberg-2
2025-08-06T13:29:07.510Z INFO [iceberg-coord ]
[o.a.k.c.c.i.ConsumerCoordinator ] {}: [Consumer
clientId=aa726279-97e7-..., groupId=connect-mantkowicz-test-iceberg-coord]
Member aa726279-97e7-... sending LeaveGroup request to coordinator
10.129.161.68:9092 (id: 2147483636 rack: null isFenced: false) due to the
consumer is being closed
2025-08-06T13:29:07.510Z INFO [iceberg-coord ]
[o.a.k.c.c.i.ConsumerCoordinator ] {}: [Consumer
clientId=aa726279-97e7-..., groupId=connect-mantkowicz-test-iceberg-coord]
Resetting generation and member id due to: consumer pro-actively leaving the
group
2025-08-06T13:29:07.510Z INFO [iceberg-coord ]
[o.a.k.c.c.i.ConsumerCoordinator ] {}: [Consumer
clientId=aa726279-97e7-..., groupId=connect-mantkowicz-test-iceberg-coord]
Request joining group due to: consumer pro-actively leaving the group
```
is it possible that there's a bug in rebalancing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
