swaminathanmanish commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1692536389
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -1696,6 +1705,32 @@ private IdealState updatePauseStatusInIdealState(String
tableNameWithType, boole
return updatedIdealState;
}
+ /**
+ * Updates the table IS property 'isQuotaExceeded' based on provided
'quotaExceeded'.
+ * Will be a no op in case the IS already has the same value.
+ * @param tableNameWithType table on which to update the IS
+ * @param quotaExceeded boolean indicating whether table has exceeded the
quota limits
+ * @return true if the IS was successfully updated for the table. Returns
false in case of no op or the update fails.
+ */
+ public boolean updateStorageQuotaExceededInIdealState(String
tableNameWithType, boolean quotaExceeded) {
+ IdealState is = getIdealState(tableNameWithType);
Review Comment:
> Notes from discussion with @swaminathanmanish Need to maintain the state
for below reasons:
>
> The ideal state update happens only during the point where table storage
exceeds the quota, making it a less frequent operation (not affected by segment
commit rate)
>
> * All further segment commits will just undergo read operation (IS is
already being fetched during segment commit so not additional read is made for
quota exceed check).
> * All prior segment commits will only perform size calculation and no IS
operation (reuse already available IS)
>
> Whereas in stateless approach we will keep querying all servers to get the
table size for all the further segment commits which feels unnecessary.
>
> The resume consumption flow gets tricky with stateless as we don’t have a
pointer on whether the consuming segments were not created due to quota breach
or were manually deleted.
>
> * Right now the segment validation task does not recreate consuming
segments. Unless we have a context on whether the quota was earlier breached or
not, we would end up recreating the missing consuming segment at every run
which is not the existing behaviour.
> * Also this job runs every hour so the IS update operation done by this
job won’t be a concern.
Thanks @shounakmk219. This makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
