lnbest0707-uber commented on PR #12336: URL: https://github.com/apache/pinot/pull/12336#issuecomment-1928495920
> sounds good. so do we have > > * 0 : OFFLINE > * 1 : ?? > * expected - 1 : SEGMENTS_WITH_LESS_REPLICAS > * expected : HEALTHY > > am I correct? Yes, correct, right now the second (1) case is not treated separately. > 1. how do we plan to avoid these alerts during say rebalance or server upgrade? (it will be one less during operation which might be a noise to monitoring system) The patch only produces metrics, and alert wise relies on how the admin monitoring the system. From my point of view, it could be based on the number and duration of SEGMENTS_WITH_LESS_REPLICAS. For example, to improve the node replacement operation. The key metrics is **how many** segments are running with SEGMENTS_WITH_LESS_REPLICAS for **how long**. And then in the real operation, if this metrics runs over the threshold we set (by experiment), it could indicate something unusual happened. Same for rebalancing and restart, if it goes over some threshold, then either something goes wrong or the existing workflow needs to be improved. > 2. do we want to address the naming of the "only 1 replica available, where configured to have more than 1" ? Right now, I do not have enough data to justify if it is critical to tell "only 1 replica available". In most of cases I saw, monitoring SEGMENTS_WITH_LESS_REPLICAS could uncover most of issues. > 3. what if the RF is set to two? do we honor "only 1 replica" or "1 less replica" or both? If replica is set to 2, then 1 == expected - 1, the SEGMENTS_WITH_LESS_REPLICAS in the patch could tell both. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
