DevShiba commented on issue #25011: URL: https://github.com/apache/datafusion/issues/25011#issuecomment-5570609381
Opened #25020 with a fix: skip the two tests under `force_hash_collisions` (same pattern already used for `count_distinct_spill` in `datafusion/core/tests/memory_limit/mod.rs`, added in #24918 for an analogous problem). On the question of intended coverage: the assertions in `assert_lazy_is_not_worse` compare allocator sizes between a lazily-built and a pre-allocated set at cardinalities chosen to span the warm-up threshold and the convergence point - none of that depends on hash distribution, so forcing every key into one bucket doesn't add or remove any coverage the tests were meant to provide, it just makes every insert O(n) instead of O(1), which is what drives the runtime to hours. Full reasoning and empirical timing data in the PR description. @adriangb tagging you too in case the intended coverage is broader than what I read from the test - happy to adjust the PR if so. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
