yashmayya opened a new pull request, #17215: URL: https://github.com/apache/pinot/pull/17215
- Fix for https://github.com/apache/pinot/issues/17179: > - The proposal to fix this issue is an overhaul of the MultiStageReplicaGroupSelector logic. We can track the instance partitions in the ideal state (through new list fields); these lists will contain both the old set of instances and the new set of instances during a rebalance with instance reassignment. After the rebalance is successfully completed, the lists will be updated to contain only the new set of instances. The MultiStageReplicaGroupSelector can use this new metadata from the ideal state to choose the instances for a request, instead of relying on the ZK instance partitions. This also paves the way for other replica group based routing strategies. > - The difference between the ideal state instance partitions and the instance partitions stored separately in the property store in ZK is that the ideal state version will be used for query routing (and can contain intermediate states) whereas the dedicated instance partitions ZNode will always contain the target instance partitions which is used for making assignment decisions for any new segments (except for upsert tables). - The existing algorithm in `MultiStageReplicaGroupSelector` is retained as a fallback because existing tables won't have the necessary metadata to support the new algorithm until and unless a rebalance is performed with some actual instance reassignment. - The remaining edge case is in rare scenarios where a server is moved from one replica group / instance partition to another in a rebalance. In most regular scenarios like changing server tags and adding / removing instances with minimize data movement enabled, this shouldn't occur, but there's nothing in the instance assignment algorithm that precludes this possibility. As a future enhancement for accurate replica group based routing strategies, the instance assignment algorithm should be updated to make sure that instances don't move from one replica group / partition to another (at least in a single direct step). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
