xinyiZzz opened a new pull request, #10743:
URL: https://github.com/apache/doris/pull/10743
# Proposed changes
Issue Number: close #10741
## Problem Summary:
#### Usability issues
The new MemTracker is difficult to use, hoping to reduce the cost of use,
mainly:
1. `SCOPED_SWITCH_xxx_TRACKER` related macros for switching TLS trackers are
various and depend on each other, which is difficult to use;
Most of the macros are mainly to improve performance and code robustness,
because MemTracker currently uses std::shared_ptr in TLS to have performance
problems under multi-threading, which is solved by a complex caching mechanism.
2. Creating a new MemTracker is error-prone;
Because the hierarchical relationship between MemTrackers needs to be
considered to avoid statistical errors affecting the accuracy of the overall
statistics;
#### Solution
1. Divide MemTracker into two subclasses,
1) The first type of MemTrackerLimiter, including four layers of
process, task pool, query/load task traker, and instance tracker, is used to
track and limit the memory usage of processes and Query.
- Manual consumption of Tracker is prohibited, only automatic
consumption in TCMalloc Hook is allowed to ensure accuracy.
- PageCache transfers memory ownership only when memory allocation
and release occur.
2) The second type of MemTrackerObserve, all operators and other
MemTrackers, are only used to track the memory usage of the specified code
segment,
- There is no parent-child relationship between MemTrackerObserves.
Both fathers are instance trakcers, but their consumption will not consume
instance trakcers synchronously. Therefore, errors in statistics will not
affect the memory tracking and restrictions of processes and Query;
- When the Hook consumes the thread tls MemTracker, it will consume
the MemTrackerLimiter and all MemTrackerObserves respectively.
2. All MemTrackers use raw pointers to avoid performance problems caused by
frequent switching of MemTrackers in tls;
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
