phillipleblanc commented on code in PR #2175:
URL:
https://github.com/apache/datafusion-ballista/pull/2175#discussion_r3653744043
##########
ballista/scheduler/src/state/execution_graph.rs:
##########
@@ -950,12 +953,15 @@ impl ExecutionGraph for StaticExecutionGraph {
running_stage
.update_task_metrics(task_id,
operator_metrics)?;
+ let SuccessfulTask {
+ partitions,
+ runtime_stats,
+ ..
+ } = successful_task;
+
running_stage.append_runtime_stats_reports(runtime_stats);
Review Comment:
Reports are not associated with task attempts. When successful work is
invalidated and rerun (i.e. lost executor, and the scheduler requeues the
task), the old report cannot be removed.
i.e. for this scenario:
```text
Task succeeds
v
Shuffle files remain on that executor
v
Stage becomes Successful
v
Executor is lost
v
Its shuffle files become unavailable
v
Tasks producing those files are marked ResultLost
v
Stage returns to Running and those partitions are recomputed
```
This is relatively minor because this affects the quality of the calculated
partition boundaries, not row correctness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
