devanbenz commented on issue #26169:
URL: https://github.com/apache/datafusion/issues/26169#issuecomment-6115196804
> It would shock me if the problem wasn't with the join order that is
selected or that our HashJoin algorithm could be much better
@alamb After running the main branch of datafusion against polars I do see
slight performance improvements, specifically Q18 looks a lot better.
Biggest gaps:
- q17: 4.7x (6.71s vs 1.42s)
- q02: 4.3x (0.73s vs 0.17s)
- q11: 4.2x (0.70s vs 0.17s)
- q07: 3.5x (6.90s vs 1.95s)
- q21: 3.3x (11.99s vs 3.62s)
- q15: 3.2x (3.13s vs 0.98s)
- q18: 2.7x (6.98s vs 2.59s)
Closest:
- q22: 0.96x (DF faster, 0.42s vs 0.43s)
- q10: 0.98x (DF faster, 2.41s vs 2.47s)
- q13: 0.99x (2.40s vs 2.41s)
- q16: 1.1x (0.49s vs 0.44s)
- q12: 1.3x (0.87s vs 0.65s)
- q01: 1.3x (2.68s vs 2.00s)
Total is 2.15x (75.1s vs 34.9s)
In my previous comment I was running the exact benchmark the polars team
published, which was 54.0.0.
The flamegraphs I have attached above are all based on main FYI.
Here's some explain analyzes for q17,q02,q11 since those were all ~4x slower
than polars.
[datafusion_q02.txt](https://github.com/user-attachments/files/33315609/datafusion_q02.txt)
[datafusion_q11.txt](https://github.com/user-attachments/files/33315608/datafusion_q11.txt)
[datafusion_q17.txt](https://github.com/user-attachments/files/33315610/datafusion_q17.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
