shauryachats commented on issue #14685: URL: https://github.com/apache/pinot/issues/14685#issuecomment-2620293887
After a deep dive into the current implementation, it turned out that the hashmap was being resized many times due to the high cardinality of the group by column, and running benchmarks on latency differences in `computeIfAbsent` for reserved vs unreserved hashmaps confirmed this. ``` Benchmark (_cardinality) Mode Cnt Score Error Units BenchmarkObjectOpenHashMap.object2IntOpenHashMap 500000 avgt 20 85.720 ± 9.901 ms/op BenchmarkObjectOpenHashMap.object2IntOpenHashMap 1000000 avgt 20 274.285 ± 32.380 ms/op BenchmarkObjectOpenHashMap.object2IntOpenHashMap 5000000 avgt 20 2011.841 ± 97.977 ms/op BenchmarkObjectOpenHashMap.object2IntOpenHashMap 10000000 avgt 20 4918.238 ± 119.845 ms/op BenchmarkObjectOpenHashMap.object2IntOpenHashMap 20000000 avgt 20 11265.309 ± 524.745 ms/op BenchmarkObjectOpenHashMap.object2IntReservedOpenHashMap 500000 avgt 20 62.205 ± 6.396 ms/op BenchmarkObjectOpenHashMap.object2IntReservedOpenHashMap 1000000 avgt 20 195.235 ± 30.416 ms/op BenchmarkObjectOpenHashMap.object2IntReservedOpenHashMap 5000000 avgt 20 2007.643 ± 35.480 ms/op BenchmarkObjectOpenHashMap.object2IntReservedOpenHashMap 10000000 avgt 20 4264.061 ± 171.140 ms/op BenchmarkObjectOpenHashMap.object2IntReservedOpenHashMap 20000000 avgt 20 7771.687 ± 107.806 ms/op ``` We can observe a clear ~30% improvement between reserved and unreserved openHashmap for ~20M distinct groups per server, a common occurrence in Uber use cases. A simple fix inspired by this was to simply reserve `numGroupsLimit` in the `GroupIdGenerator` and test it out, and the results were impressive. <img width="856" alt="Image" src="https://github.com/user-attachments/assets/016d721e-1fd0-43d6-b43c-52efa3d895fa"; /> _The two indicators indicate the start and the end of the deploy of the fix._ The aforementioned use case is low QPS and hence there were no OOMs observed due to reserving higher amounts of memory, but the possibility exists in high QPS scenarios, and therefore to eliminate that possibility, we suggest introducing a new configuration `pinot.server.query.executor.group.generator.reserveMap` flag which can be toggled to enable/disable reserving the GroupIDGenerator hash map. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
