andygrove commented on code in PR #2038:
URL:
https://github.com/apache/datafusion-ballista/pull/2038#discussion_r3587454211
##########
ballista/core/proto/ballista.proto:
##########
@@ -536,22 +557,33 @@ message TaskStatus {
message PollWorkParams {
ExecutorRegistration metadata = 1;
- uint32 num_free_slots = 2;
+ uint32 num_free_vcores = 2;
// All tasks must be reported until they reach the failed or completed state
repeated TaskStatus task_status = 3;
}
+// Pull-based single-task dispatch. One task processes a slice of
+// partitions; `task_index` names the task within the stage,
+// `partition_slice` gives the concrete global partition ids it covers. The
+// task's plan is scheduler-side shrink-restricted so its leaves report
+// `slice.len()` partitions; the writer uses `partition_slice` to attach
+// global identity to shuffle files (except in cases where the plan itself
+// resets partitioning: the writer walks its child plan to detect
+// SortPreservingMergeExec or RepartitionExec::Hash and picks the right
+// global mapping).
message TaskDefinition {
uint32 task_id = 1;
uint32 task_attempt_num = 2;
string job_id = 3;
uint32 stage_id = 4;
uint32 stage_attempt_num = 5;
- uint32 partition_id = 6;
+ uint32 task_index = 6;
Review Comment:
What is the difference between `task_id` and `task_index`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
