flashmouse commented on issue #31531: URL: https://github.com/apache/doris/issues/31531#issuecomment-1976265890
say we have 3 brokers(ba,bb,bc) in region ra,rb,rc respectively, each one hold one partition's leader and another partition's follower, and we have 3 BE (BEa, BEb, BEc) in region ra,rb,rc respectively, so whole deployment like below: |region|kafka broker|partition-replica| BE | |------|-------------|----------------|----| |ra | ba| p1-leader, p2-follower| BEa| |rb | bb| p1-follower, p3-leader| BEb| |rc | bc| p2-leader, p3-follower| BEc| consider different assign result below: 1. FE keep assign logic with no change, so assign result is: BEa-p1, BEb-p2, BEc-p3. since p1-leader & BEa are all in region ra, and according to kafka fetch-from-follower function, BEc could fetch p3 from p3-follower, so consume p1 & p3 have no cross network. in such scenario, BEb consume p2 has cross network. 2. FE do modification to aware of BEs' region and brokers' region, only give BE partition that any replication's region is the same with BE, so assign result may like: BEa-p1, BEb-p3, BEc-p2, in such scenario, no cross network generate. I'm still not know the reason to add "region" parameter to routine load, if you mean one routine load set region only consume partitions have replica in this region, then we must create routine load for each region, that's hard to maintain. what's more, kafka topic partition assignment in kafka cluster is not immutable, it may change at any time, one partition may hold in regionA, and next minute move to regionB, make it more hard to do this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
