andygrove opened a new pull request, #2563: URL: https://github.com/apache/datafusion-ballista/pull/2563
## Summary Refreshes the Ballista column of the TPC-H SF1000 results at `c76278cd` (#2553), the first commit where the `tpch` runner declares the tables' Hive partition columns. Ballista now skips the partitions a filter excludes, as Spark, Comet and Trino do, instead of relying on Parquet statistics alone. The Spark, Comet and Trino numbers are unchanged. - **Ballista total: 571.55 s → 502.43 s (−12%).** Same cluster, executor config and settings as the `d6d8bd91` run. Mean of 2 iterations, including the cold first iteration, as before. - **Faster:** Q6 4.68 → 1.59 s, Q14 8.88 → 5.54 s, Q3 25.29 → 17.22 s, Q8 28.93 → 20.13 s, Q20 20.40 → 14.46 s, Q5 63.94 → 46.88 s, Q7 45.85 → 37.26 s. - **Slower:** Q4 8.80 → 15.31 s, Q19 10.65 → 14.59 s, Q10 38.18 → 43.36 s. In the scheduler event logs, the stages that scan the pruned tables got faster. The time was lost in scans that pruning does not affect: the unfiltered `lineitem` scan in Q4 and Q19, and one slow `customer` scan task in Q10. Other queries that scan all of `lineitem` (Q1, Q9, Q17, Q18) did not slow down, and Q4's two iterations differed by 40%, so this may be noise. It needs more runs before anyone files an issue. - The Data paragraph under Environment no longer says the results predate partition columns. ## Test plan - [x] `npx [email protected] --check docs/source/contributors-guide/benchmarking.md` - [ ] Preview the rendered docs page -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
