andygrove commented on PR #26277:
URL: https://github.com/apache/datafusion/pull/26277#issuecomment-6222818656
> The description cites `join_fuzz` for the bulk path, but I think the fuzz
inputs rarely produce a run of more than one row.
You were right. I counted the appends with a temporary counter. Across the
21 joins of `test_inner_join_1k`, 74 of 2,253 appends were runs of more than
one row, and none was cut by the output batch. The other Inner, Left, Right and
Full cases were the same, 3–5% and none cut. The binary-key cases don't match
at all, since their random 4–16 byte keys never collide.
I added `make_staggered_batches_i32_with_key_range`, which
`make_staggered_batches_i32` now calls with 100, and eight cases that draw the
keys from `0..5` instead:
`test_{inner,left,right,full}_join_1k_low_cardinality`, each with a `_filtered`
variant. They use `JoinFuzzTestCase` and its batch sizes as they are. With 25
distinct keys, each streamed row matches about 40 buffered rows. In each case
about 30% of the appends are runs of more than one row, 14,000–19,000 runs are
cut by the output batch, and 51,000–56,000 end at a buffered batch boundary
with the group continuing in the next one. I added Right as well, since it
takes the same path with the sides swapped.
Their joins return about 40,000 rows, so each new case takes about 40 s of
CPU in the `ci` profile, against 15–25 s for the existing 1k cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
