avantgardnerio commented on PR #24035:
URL: https://github.com/apache/datafusion/pull/24035#issuecomment-5207704600
Pseudo code, stripped directly from the
`test_prefix_scan_across_tasks_matches_single_bwag` test, for those who want to
understand the point of this PR without reading the code:
```
// Two tasks under range partition on sn:
let (task1_out, task1_total) = run_running_sum_task(&[1, 1, 2, 2, 3,
3, 4, 4]);
let (task2_out, task2_total) = run_running_sum_task(&[5, 5, 6, 6, 7,
7, 8, 8]);
// Local (uncorrected) outputs and totals — first pass.
assert_eq!(task1_out, vec![1, 2, 4, 6, 9, 12, 16, 20]);
assert_eq!(task2_out, vec![5, 10, 16, 22, 29, 36, 44, 52]);
// Prefix scan over per-task totals → carry-in for each task. Task
0's
// carry-in is 0; task N's carry-in is the sum of tasks [0, N).
let carry_ins = [0u64, task1_total];
// Second pass: shift each task's local values by its carry-in.
let task1_final: Vec<u64> = task1_out.iter().map(|v| v +
carry_ins[0]).collect();
let task2_final: Vec<u64> = task2_out.iter().map(|v| v +
carry_ins[1]).collect();
let parallel_result: Vec<u64> =
task1_final.iter().chain(task2_final.iter());
// Oracle: single BWAG over the full concatenated input.
let (single_result, single_total) = run_running_sum_task(
&[1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8]);
assert_eq!(parallel_result, single_result);
assert_eq!(single_result,
vec![1, 2, 4, 6, 9, 12, 16, 20, 25, 30, 36, 42, 49, 56, 64, 72]
);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
