swaminathanmanish commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1685973069
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -1696,6 +1705,32 @@ private IdealState updatePauseStatusInIdealState(String
tableNameWithType, boole
return updatedIdealState;
}
+ /**
+ * Updates the table IS property 'isQuotaExceeded' based on provided
'quotaExceeded'.
+ * Will be a no op in case the IS already has the same value.
+ * @param tableNameWithType table on which to update the IS
+ * @param quotaExceeded boolean indicating whether table has exceeded the
quota limits
+ * @return true if the IS was successfully updated for the table. Returns
false in case of no op or the update fails.
+ */
+ public boolean updateStorageQuotaExceededInIdealState(String
tableNameWithType, boolean quotaExceeded) {
+ IdealState is = getIdealState(tableNameWithType);
+ if (is.getRecord().getBooleanField(IS_QUOTA_EXCEEDED, false) !=
quotaExceeded) {
+ is = HelixHelper.updateIdealState(_helixManager, tableNameWithType,
idealState -> {
+ ZNRecord znRecord = idealState.getRecord();
+ znRecord.setSimpleField(IS_QUOTA_EXCEEDED,
Boolean.valueOf(quotaExceeded).toString());
+ return new IdealState(znRecord);
+ }, RetryPolicies.noDelayRetryPolicy(1));
+ if (is == null) {
+ LOGGER.error("Failed to set 'isQuotaExceeded' to {} in the Ideal State
for table {}.", quotaExceeded,
Review Comment:
This can mean that quota can be breached because we could not set the state ?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -1696,6 +1705,32 @@ private IdealState updatePauseStatusInIdealState(String
tableNameWithType, boole
return updatedIdealState;
}
+ /**
+ * Updates the table IS property 'isQuotaExceeded' based on provided
'quotaExceeded'.
+ * Will be a no op in case the IS already has the same value.
+ * @param tableNameWithType table on which to update the IS
+ * @param quotaExceeded boolean indicating whether table has exceeded the
quota limits
+ * @return true if the IS was successfully updated for the table. Returns
false in case of no op or the update fails.
+ */
+ public boolean updateStorageQuotaExceededInIdealState(String
tableNameWithType, boolean quotaExceeded) {
+ IdealState is = getIdealState(tableNameWithType);
Review Comment:
Discussed with @shounakmk219 to explore stateless approach, where we always
read table size, instead of persisting in Zk.
Ideal state update read-modify-write is expensive we will incur large number
of writes when there are large number of segments. This state is also updated
from RealtimeSegmentValidation thread on the controller.
cc @Jackie-Jiang
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -1696,6 +1705,20 @@ private IdealState updatePauseStatusInIdealState(String
tableNameWithType, boole
return updatedIdealState;
}
+ public boolean updateStorageQuotaExceededInIdealState(String
tableNameWithType, boolean quotaExceeded) {
+ IdealState is = getIdealState(tableNameWithType);
+ if (is.getRecord().getBooleanField(IS_QUOTA_EXCEEDED, false) !=
quotaExceeded) {
+ HelixHelper.updateIdealState(_helixManager, tableNameWithType,
idealState -> {
+ ZNRecord znRecord = idealState.getRecord();
+ znRecord.setSimpleField(IS_QUOTA_EXCEEDED,
Boolean.valueOf(quotaExceeded).toString());
+ return new IdealState(znRecord);
+ }, RetryPolicies.noDelayRetryPolicy(1));
Review Comment:
If we do retry, we will have to move this check also inside the updater
lambda right
IdealState is = getIdealState(tableNameWithType);
if (is.getRecord().getBooleanField(IS_QUOTA_EXCEEDED, false) !=
quotaExceeded)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
