klsince commented on issue #13284: URL: https://github.com/apache/pinot/issues/13284#issuecomment-2153094150
This reminded me of an improvement I tried for strictReplicaGroup (https://github.com/apache/pinot/pull/11847) but didn't finish. As you mentioned in the issue description, it's a bit too rigid to skip the instance if any one of segments hosted on it was unavailable, as often we'd have to skip all instances, and reporting that a huge number of segments were unavailable (which was kinda misleading). Basically the improvement I was trying to add was to pick an instance, even though it has unavailable segments and reported status of the unavailable segments back. The key abstraction in that PR was `InstanceGroup`, which caches the mapping from `a set of instances` to `a set of segments on them`. With replica group assignment, the set of instances should host the same set of segments, but some instances might have unavailable segments and some instance might be fine. The mapping info is updated whenever IS/EV gets updated. While selecting instances, InstanceGroup is used to quickly identify a instance. The `InstanceGroup` in the PR simply tracks the segments in a `Set`, but we can group segments further by their partitions, then we may do server selection based on Instance-Partition as proposed here. Just some quick thoughts based on the implementation I tried earlier on. I didn't get enough time to finish that improvement, but if it makes sense and could be reused by this feature, then I can try my best to get it. btw, feel free to comment that PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
