andygrove commented on code in PR #2038:
URL:
https://github.com/apache/datafusion-ballista/pull/2038#discussion_r3584384424
##########
ballista/core/proto/ballista.proto:
##########
@@ -289,10 +289,20 @@ message PartitionId {
uint32 partition_id = 4;
}
+// Within-stage task identity (paired with job_id + stage_id on the parent
+// MultiTaskDefinition). One task processes a slice of partitions, so
+// `task_index` names the task within the stage; `partition_slice` gives
+// the concrete global partition ids the task's restricted plan is
+// covering. Writers use these to name shuffle files with global identity
+// (via `create_shuffle_path`) so downstream reads have a stable, canonical
+// address regardless of how the scheduler split the work.
message TaskId {
uint32 task_id = 1;
uint32 task_attempt_num = 2;
- uint32 partition_id = 3;
+ uint32 task_index = 3;
+ // Global partition ids covered by this task, in slice order. Position i in
+ // the restricted plan corresponds to `partition_slice[i]` globally.
+ repeated uint32 partition_slice = 4;
Review Comment:
So `partition_slice` / `i` is the `partition_id`?, or an index into some
list of partition ids?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
