mcvsubbu commented on PR #13584: URL: https://github.com/apache/pinot/pull/13584#issuecomment-2274055552
> On a separate note, @mcvsubbu thinks it's better to have all the logic for hanlding storage quota (pausing/resuming) in one place, in "realtime segment validation manager" periodic job. Of course the benefit is that all the logic is in one place, so it's better in terms of maintainability. The downside of doing that is the the storage won't be checked till the next round of the periodic job. Maybe that's ok because normally consuming segments take hours to complete, and even checking quota upon segment completion is also delayed. WDYT? There is enough going on during the segment completion, that I would hate to add more logic there with more conditions. Can you state why you need the quota to be checked during segment completion? I am even fine adding logic in the background task to remove latest segments (and adjust consumption offsets, of course) if the quota is exceeded (it may matter in some cases that the excess quota will result in charges). On the other hand, if this is the case, the metric on current storage use should be monitored in order to flag such situations early and in advance -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
