xinyiZzz opened a new pull request #8476: URL: https://github.com/apache/incubator-doris/pull/8476
# Proposed changes Issue Number: close #7196 (step 2/3) ## Problem Summary: Early Design Documentation: https://shimo.im/docs/DT6JXDRkdTvdyV3G Implement a new way of memory statistics based on TCMalloc New/Delete Hook, MemTracker and TLS, and it is expected that all memory new/delete/malloc/free of the BE process can be counted. specific purpose: 1. Complete and accurate query memory statistics and limits, more details: - Before, exec_mem_limit is actually the memory limit at the Fragment Instance level, not the query. After, all Fragment Instance mem trackers of a query share a common ancestor query mem tracker, which will be a real memory limit for a query. - After attaching to the query through tls, save the queryID, instanceID, and query mem tracker to tls. If the limit is exceeded when consume query mem tracker in the new/delete hook, the query will be canceled. - All threads involved in a query runtime should attach the query through tls at startup, and detach the query when the thread exits, by using preset macros. - Note that this may cause some previously successful queries to fail oom, and the query needs to be rewritten by hint set exec_mem_limit. 2. Clearer tree structure of mem tracker. - The mem tracker is clearly divided into process - query(task) pool - query(task) - fragment instance - exec node - exprs/hash table and others from top to bottom. - When creating a tracker, the current tracker in tls is used as the parent by default, so the hierarchical relationship of mem tracker is equivalent to the hierarchical relationship of code, which avoids complicated mem tracker parameter passing. - Because before, if you want to record the memory consumption of a location in the specified mem tracker, you need to pass the mem tracker layer by layer to the function of this location, which looks messy. - After that, you only need to attach query or switch mem tracker externally, and you can get this mem tracker from tls at any internal location. - This is especially useful for RowBatch and MemPool to avoid a large number of mem trackers being passed as arguments and temporary mem trackers being created. 3. Complete and accurate memory statistics. - Automatic memory statistics based on hook tcmalloc new/delete replaces the previous manual call consume/release mem tracker, which theoretically will not be missed. - The logic of manually calling consume/release of specific memory is still reserved, but these mem trackers are created as virtual trackers, and cosume/release will not sync to parent, which is only used to improve the observability of running details. Note, It is independent of the recording of tcmalloc hook in the thread local tracker, so the same block of memory is recorded independently in these two trackers. - if an independent memory allocator is used in a third-party library, special treatment is required. (TODO) Existing performance problems and solutions: 1. The mem tracker shared between threads has low performance when it is frequently consumed/released in the new/delete hook; The memory consumption of the current thread is cached in tls, and after the cumulative consumption reaches 2M, the consume / release mem tracker will be called once to avoid frequent calls. 2. At this stage, std::shared_ptr is used to save the mem tracker in tls. When a thread frequently switches the mem tracker, the use count of std::shared_ptr is frequently changed and the performance is low; During an attach query, tls caches all mem trackers that have been switched and uncommitted memory consumption, and does not need to reset ptr when switching to the same mem tracker next time. In the future, the mem tracker in tls should be changed to raw pointers to solve this problem from the source (TODO) ## Checklist(Required) 1. Does it affect the original behavior: (Yes) 2. Has unit tests been added: (Yes) 3. Has document been added or modified: (Yes) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments In the next pr, more detailed memory statistics such as exec node, exprs, hash table, etc. will be realized through the mem tracker switch during the thread attach query. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
