HHoflittlefish777 commented on issue #31531: URL: https://github.com/apache/doris/issues/31531#issuecomment-1973084046
> > Hi, @flashmouse would you like to submit this PR ? :) > > generally kafka rack-aware has 2 parts: > > 1. when client send fetch request to broker A, A know this partition has replica hold by broker B(B and client are in the same rack), so A tell client redirect this fetch request to Broker B (Fetch From Follower) > 2. during kafka consumer rebalance, assignor only assign partitions to client when any replica's rack is the same with client(this one is the [confluentinc/librdkafka#4252](https://github.com/confluentinc/librdkafka/pull/4252) implement). > > for implement method 1, we need add a new parameter ("region") into BE configuration, and rdkafka client set `client.rack={region_value}`. I think this would be simple relatively. for implement method 2, since doris assign partitions itself, this may be more complex, ~the function `KafkaDataConsumerGroup::assign_topic_partitions` need do a refactor(ex. use assign logic from rdkafka_assignor)~ I guess FE should do some modification to implement this. > > I can try implement method 1 first to familiar with pr flow. If you think this is viable I will try do this. A very clear idea can be broken down into two steps, but I have some suggestions for the first point: Adding a configuration to be may not be a good idea. In my opinion, adding a kafka attribute to the routing load job, such as ``` From KAFKA （ "Region"="XXX" ）; ``` 1. Job is scheduled by FE, and if configured in BE, FE needs to perceive the need for additional implementation 2. A global configuration without job flexibility requires consideration of **dynamic configuration changes**, which can be achieved through **schema change** Perhaps you have a better idea, but before that, I recommend you to read https://doris.apache.org/docs/data-operate/import/import-way/routine-load-manual?_highlight=routine -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
