gabotechs commented on PR #23720: URL: https://github.com/apache/datafusion/pull/23720#issuecomment-5104141874
After trying several things out, and a lot of profiling, I don't think it's worth pursuing this. Some things that I've tried: - Only reusing hashes on non-primitive columns - Ensuring at planning hash propagation machinery only triggers where it makes sense - Always threading around the reused hashes as Arrow's Uint64Array instead of converting from Vec<u8> The performance gains are so small that, even if they can be replicated locally, they fall within the typical noise of the benchmarks. Looking at profiles, the % of CPU samples we can save by reusing hashes looks something like this: TPCH-SF10: 1.3% -> 1.3% of CPU time hashing. Almost no saves, because hashes are mainly over primitive types, or because hashing the first time dominates massively, as Aggregate(partial) compresses a lot. ClickBench: 3.3% -> 1.9% of CPU time hashing. It's something, but not enough show up in any benchmark. So my conclusion is that, unless we come up with specific workloads where hashing demonstrates to be a bottleneck, I don't think the complexity is worth it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
