avantgardnerio commented on code in PR #2038:
URL:
https://github.com/apache/datafusion-ballista/pull/2038#discussion_r3587449585
##########
ballista/core/proto/ballista.proto:
##########
@@ -289,10 +289,20 @@ message PartitionId {
uint32 partition_id = 4;
}
+// Within-stage task identity (paired with job_id + stage_id on the parent
+// MultiTaskDefinition). One task processes a slice of partitions, so
+// `task_index` names the task within the stage; `partition_slice` gives
+// the concrete global partition ids the task's restricted plan is
+// covering. Writers use these to name shuffle files with global identity
+// (via `create_shuffle_path`) so downstream reads have a stable, canonical
+// address regardless of how the scheduler split the work.
message TaskId {
uint32 task_id = 1;
uint32 task_attempt_num = 2;
- uint32 partition_id = 3;
+ uint32 task_index = 3;
+ // Global partition ids covered by this task, in slice order. Position i in
+ // the restricted plan corresponds to `partition_slice[i]` globally.
+ repeated uint32 partition_slice = 4;
Review Comment:
Globally, partitions might be `[0,1,2,3,4,5,6,7]`. There might be 2
executors with 4 `vcores`. Barring other concurrent usage, the scheduler will
now assign them to executors as:
```
exec A: [0,1,2,3]
exec B: [4,5,6,7]
```
However, exec B just gets 4 partitions, so without `partition_slice`, it
just sees a zero-indexed `Vec` of 4 partitions `[0,1,2,3]`. So to enable it to
write output shuffles with the correct name / location, we send it
`partition_slice: [4,5,6,7]` so it knows its local `[0,1,2,3]` maps to the
global `[4,5,6,7]`.
1. I tried sending empty partitions
`[[],[],[],[],[file4],[file5],[file6],[file7]` but that had execution
ramifications since it would have to drive empty streams among other issues.
2. I tried _not_ sending `partition_slice` so the local DF context knew
nothing about the global situation, and have the scheduler map all the local
results back to global ones but it was more complex.
I'm happy to re-evaluate anything.
edit: short answer `partition_slice[i]` is the global partition number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
