kosiew commented on PR #25044: URL: https://github.com/apache/datafusion/pull/25044#issuecomment-5723704272
Several findings in the [benchmark run](https://github.com/apache/datafusion/pull/25410#issuecomment-5717397728) <html><head></head><body><p>Most benchmarks sit around <strong>1.00–1.02×</strong>, so construction and the expanded <code inline="">AND</code>/<code inline="">OR</code> paths are essentially unchanged. The interesting results appear in evaluation with larger workloads:</p> Benchmark | HEAD | bench-25044 | Change -- | -- | -- | -- 256_containers/in_list/1024 | 14.4 µs | 20.2 µs | ~40% slower 256_containers/in_list/256 | 12.6 µs | 18.0 µs | ~42% slower 256_containers/in_list_with_null/1025 | 14.0 µs | 20.2 µs | ~44% slower evaluate/in_list/256 | 193.3 µs | 231.0 µs | ~20% slower evaluate/in_list_with_null/257 | 192.5 µs | 230.7 µs | ~20% slower evaluate/not_in_list/256 | 157.5 µs | 142.8 µs | ~9% faster uniform_singleton/not_in_list/21 | 88.8 µs | 79.0 µs | ~11% faster uniform_singleton/not_in_list/1024 | 130.3 µs | 118.6 µs | ~9% faster <p>So there's an interesting split: <strong>large <code inline="">IN</code> cases regress, while several large <code inline="">NOT IN</code> cases improve.</strong></p><p>One particularly useful clue is the apparent threshold around <strong>20/21 elements</strong>. For example, <code inline="">in_list/20</code> behaves normally, while <code inline="">in_list/21</code> suddenly drops from roughly ~100 µs into a much faster path; similarly, the 256- and 1024-element cases behave very differently from the smaller lists. That strongly suggests these benchmarks are exercising <strong>different internal pruning strategies depending on list size</strong>, rather than showing a uniform slowdown of the implementation.</p><p>Also, the <code inline="">not_in_list_with_null_constant_false</code> cases are extremely fast—hundreds of <strong>nanoseconds</strong>—because they appear to hit a constant-false shortcut.</p><p><strong>Bottom line:</strong> the benchmark does <strong>not</strong> show a general regression. Most results are flat. The main thing worth investigating is the <strong>~20–44% slowdown for large <code inline="">IN</code> / <code inline="">IN ... NULL</code> evaluation</strong>, especially under <code inline="">256_containers</code>; meanwhile, <code inline="">NOT IN</code> actually gets noticeably faster in several comparable cases. That pattern looks structural enough that I wouldn't dismiss it as benchmark noise.</p></body></html> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
