yashmayya commented on issue #17179: URL: https://github.com/apache/pinot/issues/17179#issuecomment-3578480286
@ankitsultana that makes sense - however, in these scenarios where a table's instance assignment is updated and it is rebalanced, there will most likely be a brief period where the two tables that are supposed to be colocated will no longer be colocated (until the second table is also rebalanced with instance reassignment enabled). During this period, the existing behavior of `MultiStageReplicaGroupSelector` is to silently drop segments in the `SelectionResult` as demonstrated in https://github.com/apache/pinot/pull/17180. This then leads to incorrect query results that aren't even flagged as such. Is that the desired behavior? In the proposed fix in https://github.com/apache/pinot/pull/17215, the new behavior in these rebalance scenarios would be to continue fully serving queries, but potentially without supporting colocation (since the instance partitions of the two tables could be out of sync, and so could the segment assignments). We're planning to introduce the new ideal state instance partitions metadata anyway, because that is useful to us for other reasons, but in case you want to retain the older behavior for `MultiStageReplicaGroupSelector` I'm happy to revert those changes. This new ideal state instance partitions metadata will be gated by a new controller config defaulting to `false` though, so `MultiStageReplicaGroupSelector` will retain its default behavior unless the new config is explicitly enabled and all the tables are rebalanced so that their ideal states contain the new metadata for routing. Based on https://github.com/apache/pinot/pull/15843 though, I'm guessing that the existing behavior is not intentional or desirable (since the existing logic would also break colocation support in case of partial segment unavailability where we select instances across replica groups). cc - @Jackie-Jiang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
