shounakmk219 opened a new pull request, #15118:
URL: https://github.com/apache/pinot/pull/15118
# Description
This PR
- Adds new stats to track using `MinionProgressObserver`
- Expose minion endpoint to fetch the task progress stats
## Newly added stats
1. `_currentStage` : while the `_currentState` is limited to only hold
`MinionTaskState`, we need more granular idea about which stage the minion task
is currently in. This property has hold any task specific stage to give better
tracking.
2. `_endTimestamp`
3. `_stageTimes` : with the new `_currentStage` property, this property
stores a map of stage and its start and total time spent. This stat can help
user to identify which stage is taking more time and fine tune the task configs
accordingly.
These new stats are leveraged to breakdown the `SegmentProcessorFramework`
observability further into map, reduce and segment generate stage.
## Endpoint to fetch the task progress stats
Path : `/tasks/subtask/progressStats`
Method : `GET`
Query params : `subtaskName`
Sample Response:
```
{
"Task_RealtimeToOfflineSegmentsTask_d5f1d3cb-53f7-4020-9276-a672ccba6a53_1740399840094_0":{
"currentState":"SUCCEEDED",
"taskId":"Task_RealtimeToOfflineSegmentsTask_d5f1d3cb-53f7-4020-9276-a672ccba6a53_1740399840094_0",
"currentStage":"SUCCEEDED",
"endTimestamp":1740399840317,
"startTimestamp":1740399840190,
"stageTimes":{
"IN_PROGRESS":{
"startTimeMs":1740399840191,
"totalTimeMs":59
},
"GENERATE_SEGMENT":{
"startTimeMs":1740399840256,
"totalTimeMs":61
},
"SUCCEEDED":{
"startTimeMs":1740399840326,
"totalTimeMs":0
},
"MAP":{
"startTimeMs":1740399840250,
"totalTimeMs":6
},
"REDUCE":{
"startTimeMs":1740399840256,
"totalTimeMs":0
}
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
