tarun11Mavani commented on code in PR #17352:
URL: https://github.com/apache/pinot/pull/17352#discussion_r2665899376
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/MinionTaskUtils.java:
##########
@@ -334,4 +335,47 @@ public static ValidDocIdsType
getValidDocIdsType(UpsertConfig upsertConfig, Map<
}
return validDocIdsType;
}
+
+ /**
+ * Checks if all replicas have consensus on validDoc counts for a segment.
+ * SAFETY LOGIC:
+ * 1. Only proceed with operations when ALL replicas agree on totalValidDocs
count
+ * 2. Skip operations if ANY server hosting the segment is not in READY state
+ * 3. Include all replicas (even those with CRC mismatches) in consensus for
safety
+ *
+ * @param segmentName the name of the segment being checked
+ * @param replicaMetadataList list of metadata from all replicas of the
segment
+ * @return true if all replicas have consensus on validDoc counts, false
otherwise
+ */
+ public static boolean hasValidDocConsensus(String segmentName,
Review Comment:
The reason for skipping a segment when a deviation is detected is that using
a validDocId (which may differ from the one on another replica for some reason)
could potentially result in data loss. While I can’t think of a concrete
scenario where this would happen, the risk still exists, so it’s safer to take
a conservative approach.
we already have a metric to track skipped segment, I will monitor this for
our production tables and if we see too many segment being skipped and
impacting compaction throughput, we can revisit this check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
