suvodeep-pyne opened a new pull request, #17099:
URL: https://github.com/apache/pinot/pull/17099
## Overview
This PR implements Phase 1 of segment reload failure tracking by introducing
an in-memory status cache on Pinot servers. This enables the controller and
users to query detailed failure information for reload operations, addressing a
critical gap where reload failures were logged but not queryable via API.
## Problem
Currently, the segment reload status API uses timestamp-based heuristics
(`loadTimeMs >= submissionTime`) which cannot distinguish between:
- Segments that haven't started reloading (PENDING)
- Segments currently being reloaded (IN_PROGRESS)
- Segments that failed to reload (FAILED)
When reloads fail, the system logs errors and increments metrics, but does
NOT store failure details for later API queries. Users have no way to determine
which segments failed or why.
## Solution
This PR implements a lightweight in-memory cache on each server to track
reload job status:
### Key Components
1. **ServerReloadJobStatusCache** - Guava-based in-memory cache
- LRU eviction with configurable max size (10,000 jobs default)
- 30-day TTL for automatic cleanup
- Thread-safe for concurrent access
- Tracks failure count per reload job
2. **ReloadJobStatus** - Per-job status tracking
- Phase 1: Tracks failure count only (using AtomicInteger)
- Future phases: Will track per-segment status with error details
3. **Job ID Preservation** - Modified SegmentReloadMessage
- Stores reload job ID in message payload (ZNRecord)
- Survives Helix's message cloning during distribution
- Threaded through entire server-side call chain
### API Changes
**Server API Enhancement:**
- Added optional `reloadJobId` query parameter to `GET
/controllerJob/reloadStatus/`
- Response now includes `failureCount` field (nullable Long)
- Backward compatible - existing clients work without changes
**Controller API Enhancement:**
- Controller queries servers with job ID
- Aggregates failure counts from all servers
- `PinotTableReloadStatusResponse` includes `failureCount` field
### Example Response
```json
{
"totalSegmentCount": 10,
"successCount": 7,
"failureCount": 2,
"timeElapsedInMinutes": 5.2,
"estimatedTimeRemainingInMinutes": 1.04
}
```
## Implementation Details
### Dependency Injection Flow
```
BaseServerStarter (creates cache)
→ ServerInstance
→ HelixInstanceDataManager
→ TableDataManager
→ BaseTableDataManager (tracks failures)
```
### Failure Tracking
- Cache integrated into `BaseTableDataManager.reloadSegments()` catch block
- Increments failure count when segment reload throws exception
- Uses real job ID from Helix message (not placeholder)
### Jersey DI Integration
- Cache registered with HK2 container in `AdminApiApplication`
- REST resources use `@Inject` annotation
- Same singleton instance used across data path and API path
## Testing
- Updated 9 test files with mock cache instances
- All existing tests pass with new required parameter
- Backward compatible - null checks allow operation without job ID
## Future Work
Phase 1 (this PR) only tracks aggregate failure counts. Future phases will
add:
- Per-segment status tracking (SUCCESS, FAILED, IN_PROGRESS, PENDING)
- Full error details with stack traces
- Complete failure tracking for all reload paths (currently only batch
reload tracked)
- See design spec: `spec/reload-server-status-cache.md`
## Files Changed
### New Files
-
`pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/ServerReloadJobStatusCache.java`
-
`pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/ReloadJobStatus.java`
-
`pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/ServerReloadJobStatusCacheConfig.java`
### Modified Files
- Message layer: `SegmentReloadMessage.java` - store/retrieve job ID
- Server API: `ControllerJobStatusResource.java` - query cache and return
failure counts
- Controller API: `PinotTableReloadStatusResponse.java`,
`PinotTableReloadStatusReporter.java` - aggregate failure counts
- Data managers: `BaseTableDataManager.java`, `InstanceDataManager.java` -
thread job ID and track failures
- Server initialization: `BaseServerStarter.java`, `ServerInstance.java`,
`AdminApiApplication.java` - create and wire cache
## Backward Compatibility
✅ Fully backward compatible:
- New `reloadJobId` parameter is optional
- New `failureCount` field is nullable (null = data not available)
- Existing API clients continue to work without changes
- Old reload messages (without job ID) work with null checks
## Related Issues
Addresses the failure tracking requirements outlined in:
- `spec/reload-failure-tracking.md`
- `spec/reload-server-status-cache.md`
---
**Note:** This is Phase 1 implementation focusing on basic failure count
tracking. The infrastructure is in place for future enhancements including
per-segment status and complete error details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
