tarun11Mavani commented on code in PR #17352:
URL: https://github.com/apache/pinot/pull/17352#discussion_r2663513763
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/MinionTaskUtils.java:
##########
@@ -334,4 +335,47 @@ public static ValidDocIdsType
getValidDocIdsType(UpsertConfig upsertConfig, Map<
}
return validDocIdsType;
}
+
+ /**
+ * Checks if all replicas have consensus on validDoc counts for a segment.
+ * SAFETY LOGIC:
+ * 1. Only proceed with operations when ALL replicas agree on totalValidDocs
count
+ * 2. Skip operations if ANY server hosting the segment is not in READY state
+ * 3. Include all replicas (even those with CRC mismatches) in consensus for
safety
+ *
+ * @param segmentName the name of the segment being checked
+ * @param replicaMetadataList list of metadata from all replicas of the
segment
+ * @return true if all replicas have consensus on validDoc counts, false
otherwise
+ */
+ public static boolean hasValidDocConsensus(String segmentName,
Review Comment:
I haven't seen this deviation due to live ingestion (have seen it after
restart/replace) mostly.
I have added new metric , so we will be able to monitor if there is a large
scale deviation across multiple segments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
