xinyiZzz opened a new pull request, #12496: URL: https://github.com/apache/doris/pull/12496
# Proposed changes Issue Number: close #xxx ## Problem summary 1. Before jemalloc was compiled with arrow, it was compiled separately 2. Modify the default parameters of jemalloc to achieve better performance and lower memory usage. This will significantly improve multi-threading and high concurrency memory performance. refer to: https://github.com/jemalloc/jemalloc/blob/dev/TUNING.md https://jemalloc.net/jemalloc.3.html https://github.com/jemalloc/jemalloc/issues/1621 https://github.com/jemalloc/jemalloc/wiki/Getting-Started ## Performance Verification Results 1、single thread sequential execution > the sum of the averages of multiple executions of each query 1) Clickbench: | | sum of time(s) | peak mem(M) | | :------| ------: | :------: | | tcmalloc | 210 | 11927 | | jemalloc default conf | 203.17 | 20777 | | jemalloc optimize conf | 191.87 | 13594 | 2) SSB: | | sum of time(s) | peak mem(M) | | :------| ------: | :------: | | tcmalloc | 15.845 | 832 | | jemalloc default conf | 15.052 | 2920 | | jemalloc optimize conf | 13.828 | 2091 | Looking at the flame graph, the time-consuming of submitting sql in a single thread is not in memory, so the performance improvement is less. 2、jmeter stress test, disable page cache and chunk allocator > Take the results of the second stress test for each sql, because jemalloc has a cold start, the first stress test is more aggressively cached, and the second stress test starts to get faster. 1) Clickbench, only q13 + q14 | | sum of time(s) | | :------| ------: | :------: | | tcmalloc | 82611 | | jemalloc default conf | 72688 | | jemalloc optimize conf | 42506 | After optimization, the performance of jemalloc is doubled. 4、SSB, jmeter stress test, disable page cache and chunk allocator > only q13 + q14，Take the results of the second stress test for each sql, because jemalloc has a cold start, the first stress test is more aggressively cached, and the second stress test starts to get faster. | | sum of time(s) | | :------| ------: | :------: | | tcmalloc | 82611 | | jemalloc default conf | 72688 | | jemalloc optimize conf | 42506 | After optimization, the performance of jemalloc is doubled. 1、Clickbench, single thread sequential execution Reproduce： ``` be.conf add `enable_tcmalloc_hook=false` `disable_storage_page_cache=true` `disable_mem_pools=true` `chunk_reserved_bytes_limit=1` vim tools/clickbench-tools/run-clickbench-queries.sh pre_set "set global parallel_fragment_exec_instance_num=1;" pre_set "set global exec_mem_limit=20G; sh tools/clickbench-tools/run-clickbench-queries.sh ``` 2、SSB, single thread sequential execution Reproduce： ``` be.conf add `enable_tcmalloc_hook=false` `disable_storage_page_cache=true` `disable_mem_pools=true` `chunk_reserved_bytes_limit=1` vim tools/ssb-tools/bin/run-ssb-queries.sh pre_set "set global parallel_fragment_exec_instance_num=1;" sh tools/ssb-tools/bin/run-ssb-queries.sh ``` 3、Clickbench, jmeter stress test, disable page cache and chunk allocator Reproduce： ``` be.conf add `enable_tcmalloc_hook=false` `disable_storage_page_cache=true` `disable_mem_pools=true` `chunk_reserved_bytes_limit=1` jmeter conf <stringProp name="ThreadGroup.num_threads">30</stringProp> <stringProp name="ThreadGroup.ramp_time">1</stringProp> <boolProp name="ThreadGroup.scheduler">true</boolProp> <stringProp name="ThreadGroup.duration">100</stringProp> <stringProp name="ThreadGroup.delay">0</stringProp> ``` ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
