klsince commented on PR #14516: URL: https://github.com/apache/pinot/pull/14516#issuecomment-2492048086
Even with this fix, there might be another subtle race condition when multiple table rebalances are conducted by different threads (or even different controllers) in parallel, e.g. - one rebalancer decides to set segment tier to A to move segment to a server in tier A; - another rebalancer decides to set segment tier to B to move segment to a server in tier B, around the same time. Due to race condition, server in tier A may load the segment, whose tier field in ZK metadata might have been set to tier B. But this race condition should be rare, as tier migration usually goes to one direction, e.g. from default tier (hot), then to next tier (warm), then to next tier (cold) etc.. and with enough time gaps between tiers. So at each time point, rebalancers just decide if a segment should stay on current tier, or move to the immediate next tier. In this case, the race condition said above wouldn't happen, as rebalancers are not trying to move segment to different tiers around the same time. And if table just has two tiers, this wouldn't happen either. To address it, I think we'd need a distributed lock via ZK to ensure that at any time only one table rebalance happens even across controllers. This can be addressed later if we see the need. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
