rajagopr commented on code in PR #15008:
URL: https://github.com/apache/pinot/pull/15008#discussion_r1951769418
##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##########
@@ -135,6 +138,18 @@ private boolean shouldEnsureConsuming(String
tableNameWithType) {
if (isTablePaused &&
pauseStatus.getReasonCode().equals(PauseState.ReasonCode.ADMINISTRATIVE)) {
return false;
}
+ try {
+ boolean isResourceUtilizationWithinLimits =
+
_resourceUtilizationManager.isResourceUtilizationWithinLimits(tableNameWithType);
+ if (!isResourceUtilizationWithinLimits) {
+ LOGGER.warn("Resource utilization limit exceeded for table: {}",
tableNameWithType);
+ _llcRealtimeSegmentManager.pauseConsumption(tableNameWithType,
Review Comment:
`I fear edge cases of 1)` --> Unless we perform the resource utilization
check prior to every segment upload, we won't be able to cover the edge cases.
For example, the `RealtimeSegmentValidationManager` runs every hour (default
configuration) and during this hour it's possible that the new segments
ingested end up utilizing all the disk space.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
