9aman opened a new pull request, #16856:
URL: https://github.com/apache/pinot/pull/16856
# PR Description
## Problem
The metric update relies on Task Manager runs, which can cause the metric
value to become stale and impact any configured alerts.
### Solution: Transform OfflineSegmentInternalChecker to
OfflineSegmentValidationManager
Why this transformation is needed:
- OfflineSegmentValidationManager can serve as a controller periodic job for
OFFLINE tables, similar to how RealtimeSegmentValidationManager works for
REALTIME tables
- It can handle periodic checks at both:
- Table level
- Segment level
- Table and segment level checks can run at different frequencies, following
the same approach used in RealtimeSegmentValidationManager
- No need to create another controller periodic job just to reset the metric
## Alternative Considered
### Option 1: Create a new controller periodic job for resource utilization
checks
#### Pros:
Checks might expand beyond disk usage, potentially justifying a separate
periodic job
#### Cons:
- Cannot make pause/unpause decisions for realtime tables
- RealtimeSegmentValidationManager makes operational decisions based on
table pause/unpause status
- Race condition risk (sequence of events ):
- RealtimeSegmentValidationManager finds table unpaused and continues
checks
- New periodic job pauses the table (triggering force commit)
- RealtimeSegmentValidationManager doesn't find any CONSUMING segment,
assumes table is unpaused, and creates a new CONSUMING segment
### Option 2:Using RealtimeSegmentValidationManager Instead
#### Pros:
Simple solution using existing controller periodic job
Cons:
#### RealtimeSegmentValidationManager is specifically designed for REALTIME
tables
## Additional Work Needed in this PR.
Implement different frequencies for table-level and segment-level validations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
