andygrove commented on code in PR #2038:
URL:
https://github.com/apache/datafusion-ballista/pull/2038#discussion_r3587431846
##########
ballista/core/proto/ballista.proto:
##########
@@ -289,10 +289,20 @@ message PartitionId {
uint32 partition_id = 4;
}
+// Within-stage task identity (paired with job_id + stage_id on the parent
+// MultiTaskDefinition). One task processes a slice of partitions, so
+// `task_index` names the task within the stage; `partition_slice` gives
+// the concrete global partition ids the task's restricted plan is
+// covering. Writers use these to name shuffle files with global identity
+// (via `create_shuffle_path`) so downstream reads have a stable, canonical
+// address regardless of how the scheduler split the work.
message TaskId {
uint32 task_id = 1;
uint32 task_attempt_num = 2;
- uint32 partition_id = 3;
Review Comment:
Should we just make `partition_id` repeated? The current case of passing a
single id would carry on working, and it would also support the multiple
partitions per task approach?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
