lucasbru opened a new pull request, #20118:
URL: https://github.com/apache/kafka/pull/20118
The current assignor used in KIP-1071 is verbatim the assignor used on the
client-side. The assignor performance was not a big concern on the client-side,
and it seems some additional performance overhead has crept in during the
adaptation to the broker-side interfaces, so we expect it to be too slow for
groups of non-trivial size.
We base ourselves on the share-group parameters for these benchmarks:
- Up to 1000 members
- Up to 100 topics
- Up to 100 partitions per topic
Note, however, that the parameters influencing the Streams assignment are
different and more complicated compared to regular consumer groups / share
consumer groups. The assignment logic is independent of the number of topics,
but depends on the number of subtopologies. A subtopology may read from
multiple topics. We simplify this relationship by assuming one topic per
subtopology Members may be part of the same process or separate processes. We
introduce a parameter membersPerProcess to tune two extreme configurations (1,
50).
We define 50% of the subtopologies to be stateful. Stateful subtopologies
get standby replicas assigned, if enabled. For example, if we have 100
subtopologies with 100 partitions, we get 10,000 active tasks and 5,000 standby
tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
