andygrove opened a new pull request, #4906:
URL: https://github.com/apache/datafusion-comet/pull/4906
> This PR was created by an LLM as a draft PR. I will mark it as ready for
review after human review.
## Which issue does this PR close?
N/A — autonomous exploratory PR.
## Rationale for this change
Replaced the per-probe-element Arrow eq kernel call (which allocated a
BooleanArray per element) with typed scans over the flat values buffers plus a
hash probe for large lists, cutting allocations and kernel dispatches to zero
on flat element types.
## What changes are included in this PR?
Replaced the per-probe-element Arrow eq kernel call (which allocated a
BooleanArray per element) with typed scans over the flat values buffers plus a
hash probe for large lists, cutting allocations and kernel dispatches to zero
on flat element types.
## How are these changes tested?
Correctness: unit tests + seeded differential fuzz (bit-identical Arrow
output vs `main`).
Benchmark (criterion):
- spark_arrays_overlap_ int32 medium lists: 92.925% faster (base 11690342ns
-> cand 827066ns)
- spark_arrays_overlap_ utf8 short lists: 77.473% faster (base 3281623ns ->
cand 739251ns)
- spark_arrays_overlap_ utf8 long lists: 94.733% faster (base 11544605ns ->
cand 608100ns)
- spark_arrays_overlap_ int32 long lists: 94.947% faster (base 12375951ns ->
cand 625320ns)
- spark_arrays_overlap_ int32 short lists: 94.474% faster (base 3018371ns ->
cand 166793ns)
Full criterion output:
```text
spark_arrays_overlap: int32 short lists
time: [166.19 µs 166.36 µs 166.55 µs]
change: [−94.489% −94.474% −94.460%] (p = 0.00 <
0.05)
Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) low mild
1 (1.00%) high mild
1 (1.00%) high severe
spark_arrays_overlap: int32 medium lists
time: [823.79 µs 825.75 µs 827.78 µs]
change: [−92.947% −92.925% −92.903%] (p = 0.00 <
0.05)
Performance has improved.
spark_arrays_overlap: int32 long lists
time: [623.94 µs 625.77 µs 627.46 µs]
change: [−94.965% −94.947% −94.930%] (p = 0.00 <
0.05)
Performance has improved.
spark_arrays_overlap: utf8 short lists
time: [728.82 µs 731.53 µs 734.73 µs]
change: [−77.607% −77.473% −77.335%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
spark_arrays_overlap: utf8 long lists
time: [608.52 µs 609.75 µs 610.95 µs]
change: [−94.755% −94.733% −94.712%] (p = 0.00 <
0.05)
Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
3 (3.00%) low mild
1 (1.00%) high mild
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
