ByteYue opened a new pull request, #23834:
URL: https://github.com/apache/doris/pull/23834
## Proposed changes
Issue Number: close #xxx
Before this pr, WriteCooldownMetaExecutors use `ThreadPool`, the `shutdown`
function of which would try to execute the following expression `_pool_status =
Status::ServiceUnavailable("The thread pool {} has been shut down.", _name);`
which might invoke `toStringCached` inside stack_trace.cpp. And there is also
one singleton named `cacheInstance` lying inside stack_trace.cpp. And once the
`cacheInstance` is already destructed before WriteCooldownMetaExecutors it
would result in heap use after free as follows.
```
==3940985==ERROR: AddressSanitizer: heap-use-after-free on address
0x6140011f43c8 at pc 0x55861ac91cfc bp 0x7ffc418f47e0 sp 0x7ffc418f47d8
READ of size 8 at 0x6140011f43c8 thread T0
#0 0x55861ac91cfb in std::_Head_base<2ul, unsigned long,
false>::_Head_base(unsigned long const&)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:183:22
#1 0x55861ac91c9e in std::_Tuple_impl<2ul, unsigned
long>::_Tuple_impl(unsigned long const&)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:426:9
#2 0x55861bcfab56 in std::_Tuple_impl<1ul, unsigned long, unsigned
long>::_Tuple_impl(unsigned long const&, unsigned long const&)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:270:9
#3 0x55861bcfaafe in std::_Tuple_impl<0ul, std::array, unsigned long,
unsigned long>::_Tuple_impl(std::array const&, unsigned long const&, unsigned
long const&)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:270:9
#4 0x55861bcf973d in std::tuple, unsigned long, unsigned
long>::tuple(std::array const&, unsigned long const&, unsigned long const&)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:719:4
#5 0x55861bcf92f6 in bool operator<(StackTraceTriple const&,
StackTraceRefTriple const&)
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/common/stack_trace.cpp:343:12
#6 0x55861bcf907e in decltype(auto) std::less::_S_cmp(StackTraceTriple
const&, StackTraceRefTriple const&, std::integral_constant)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_function.h:601:34
#7 0x55861bcf8dc2 in decltype(std::forward(fp) < std::forward(fp0))
std::less::operator()(StackTraceTriple const&, StackTraceRefTriple const&)
const
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_function.h:586:11
#8 0x55861bcf89d3 in std::_Rb_tree_const_iterator, std::allocator>>>
std::_Rb_tree, std::allocator>>, std::_Select1st, std::allocator>>>, std::less,
std::allocator, std::allocator>>>>::_M_lower_bound_tr(StackTraceRefTriple
const&) const
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1337:11
#9 0x55861bcf8506 in std::_Rb_tree_const_iterator, std::allocator>>>
std::_Rb_tree, std::allocator>>, std::_Select1st, std::allocator>>>, std::less,
std::allocator, std::allocator>>>>::_M_find_tr(StackTraceRefTriple const&)
const
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1305:15
#10 0x55861bcf82f6 in std::_Rb_tree_iterator, std::allocator>>>
std::_Rb_tree, std::allocator>>, std::_Select1st, std::allocator>>>, std::less,
std::allocator, std::allocator>>>>::_M_find_tr(StackTraceRefTriple const&)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1297:25
#11 0x55861bcf0bfc in decltype((*this)._M_t._M_find_tr(fp)) std::map,
std::allocator>, std::less, std::allocator,
std::allocator>>>>::find(StackTraceRefTriple const&)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_map.h:1176:16
#12 0x55861bce961a in toStringCached[abi:cxx11](std::array const&,
unsigned long, unsigned long)
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/common/stack_trace.cpp:433:25
#13 0x55861bcea172 in StackTrace::toString[abi:cxx11]() const
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/common/stack_trace.cpp:444:12
#14 0x55861bce1132 in doris::get_stack_trace_by_libunwind[abi:cxx11]()
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/util/stack_util.cpp:84:25
#15 0x55861bce05ff in doris::get_stack_trace[abi:cxx11]()
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/util/stack_util.cpp:51:16
#16 0x558618742e49 in doris::Status doris::Status::Error,
std::allocator> const&>(int, std::basic_string_view>,
std::__cxx11::basic_string, std::allocator> const&)
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/common/status.h:376:39
#17 0x55861be6bffc in doris::Status doris::Status::Error<41, true,
std::__cxx11::basic_string, std::allocator> const&>(std::basic_string_view>,
std::__cxx11::basic_string, std::allocator> const&)
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/common/status.h:361:16
#18 0x55861be5af67 in doris::Status doris::Status::ServiceUnavailable,
std::allocator> const&>(std::basic_string_view>, std::__cxx11::basic_string,
std::allocator> const&)
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/common/status.h:411:5
#19 0x55861be4ea61 in doris::ThreadPool::shutdown()
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/util/threadpool.cpp:277:20
#20 0x55861be4e5f6 in doris::ThreadPool::~ThreadPool()
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/util/threadpool.cpp:249:5
#21 0x5586186a77bc in
std::default_delete::operator()(doris::ThreadPool*) const
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2
#22 0x55861868013b in std::unique_ptr>::~unique_ptr()
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4
#23 0x55861add2e76 in void std::destroy_at>>(std::unique_ptr>*)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15
#24 0x55861add2e46 in void std::_Destroy>>(std::unique_ptr>*)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7
#25 0x55861add2dfa in void
std::_Destroy_aux::__destroy>*>(std::unique_ptr>*, std::unique_ptr>*)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6
#26 0x55861add2d9e in void std::_Destroy>*>(std::unique_ptr>*,
std::unique_ptr>*)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7
#27 0x55861add2c62 in void std::_Destroy>*,
std::unique_ptr>>(std::unique_ptr>*, std::unique_ptr>*, std::allocator>>&)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7
#28 0x55861ada3120 in std::vector>, std::allocator>>>::~vector()
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680:2
#29 0x55861adb9d05 in
doris::WriteCooldownMetaExecutors::~WriteCooldownMetaExecutors()
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/olap/tablet.cpp:163:8
#30 0x7f53c33d48a6 in __run_exit_handlers
/build/glibc-SzIz7B/glibc-2.31/stdlib/exit.c:108:8
#31 0x7f53c33d4a5f in exit
/build/glibc-SzIz7B/glibc-2.31/stdlib/exit.c:139:3
#32 0x7f53c33b2089 in __libc_start_main
/build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:342:3
#33 0x558618595029 in _start
(/mnt/hdd01/ci/branch20-deploy/be/lib/doris_be+0x185b4029) (BuildId:
8455972b9fc62591)
0x6140011f43c8 is located 392 bytes inside of 440-byte region
[0x6140011f4240,0x6140011f43f8)
freed by thread T0 here:
#0 0x558618668d9d in operator delete(void*)
(/mnt/hdd01/ci/branch20-deploy/be/lib/doris_be+0x18687d9d) (BuildId:
8455972b9fc62591)
#1 0x55861bcf371e in __gnu_cxx::new_allocator,
std::allocator>>>>::deallocate(std::_Rb_tree_node, std::allocator>>>*, unsigned
long)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:139:2
#2 0x55861bcf36e0 in std::allocator,
std::allocator>>>>::deallocate(std::_Rb_tree_node, std::allocator>>>*, unsigned
long)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:187:27
#3 0x55861bcf36e0 in std::allocator_traits,
std::allocator>>>>>::deallocate(std::allocator, std::allocator>>>>&,
std::_Rb_tree_node, std::allocator>>>*, unsigned long)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:492:13
#4 0x55861bcf34fa in std::_Rb_tree, std::allocator>>, std::_Select1st,
std::allocator>>>, std::less, std::allocator,
std::allocator>>>>::_M_put_node(std::_Rb_tree_node, std::allocator>>>*)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:565:9
#5 0x55861bcf343f in std::_Rb_tree, std::allocator>>, std::_Select1st,
std::allocator>>>, std::less, std::allocator,
std::allocator>>>>::_M_drop_node(std::_Rb_tree_node, std::allocator>>>*)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:632:2
#6 0x55861bcf32d5 in std::_Rb_tree, std::allocator>>, std::_Select1st,
std::allocator>>>, std::less, std::allocator,
std::allocator>>>>::_M_erase(std::_Rb_tree_node, std::allocator>>>*)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1889:4
#7 0x55861bcf3235 in std::_Rb_tree, std::allocator>>, std::_Select1st,
std::allocator>>>, std::less, std::allocator, std::allocator>>>>::~_Rb_tree()
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:984:9
#8 0x55861bcf3106 in std::map, std::allocator>, std::less,
std::allocator, std::allocator>>>>::~map()
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_map.h:302:22
#9 0x7f53c33d48a6 in __run_exit_handlers
/build/glibc-SzIz7B/glibc-2.31/stdlib/exit.c:108:8
previously allocated by thread T0 here:
#0 0x55861866853d in operator new(unsigned long)
(/mnt/hdd01/ci/branch20-deploy/be/lib/doris_be+0x1868753d) (BuildId:
8455972b9fc62591)
#1 0x55861bcfc6df in __gnu_cxx::new_allocator,
std::allocator>>>>::allocate(unsigned long, void const*)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:121:27
#2 0x55861bcfc634 in std::allocator,
std::allocator>>>>::allocate(unsigned long)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:173:32
#3 0x55861bcfc634 in std::allocator_traits,
std::allocator>>>>>::allocate(std::allocator, std::allocator>>>>&, unsigned
long)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:460:20
#4 0x55861bcfc4d3 in std::_Rb_tree, std::allocator>>, std::_Select1st,
std::allocator>>>, std::less, std::allocator, std::allocator>>>>::_M_get_node()
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:561:16
#5 0x55861bcfba52 in std::_Rb_tree_node, std::allocator>>>*
std::_Rb_tree, std::allocator>>, std::_Select1st, std::allocator>>>, std::less,
std::allocator, std::allocator>>>>::_M_create_node,
std::allocator>>(StackTraceTriple&&, std::__cxx11::basic_string,
std::allocator>&&)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:611:23
#6 0x55861bcfb4a5 in std::pair, std::allocator>>>, bool> std::_Rb_tree,
std::allocator>>, std::_Select1st, std::allocator>>>, std::less,
std::allocator, std::allocator>>>>::_M_emplace_unique,
std::allocator>>(StackTraceTriple&&, std::__cxx11::basic_string,
std::allocator>&&)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:2382:19
#7 0x55861bcf0ffa in std::pair, std::allocator>>>, bool> std::map,
std::allocator>, std::less, std::allocator, std::allocator>>>>::emplace,
std::allocator>>(StackTraceTriple&&, std::__cxx11::basic_string,
std::allocator>&&)
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_map.h:577:16
#8 0x55861bce9956 in toStringCached[abi:cxx11](std::array const&,
unsigned long, unsigned long)
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/common/stack_trace.cpp:439:22
#9 0x55861bcea172 in StackTrace::toString[abi:cxx11]() const
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/common/stack_trace.cpp:444:12
#10 0x55861bce1132 in doris::get_stack_trace_by_libunwind[abi:cxx11]()
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/util/stack_util.cpp:84:25
#11 0x55861bce05ff in doris::get_stack_trace[abi:cxx11]()
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/util/stack_util.cpp:51:16
#12 0x558618742e49 in doris::Status doris::Status::Error,
std::allocator> const&>(int, std::basic_string_view>,
std::__cxx11::basic_string, std::allocator> const&)
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/common/status.h:376:39
#13 0x55861be6bffc in doris::Status doris::Status::Error<41, true,
std::__cxx11::basic_string, std::allocator> const&>(std::basic_string_view>,
std::__cxx11::basic_string, std::allocator> const&)
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/common/status.h:361:16
#14 0x55861be5af67 in doris::Status doris::Status::ServiceUnavailable,
std::allocator> const&>(std::basic_string_view>, std::__cxx11::basic_string,
std::allocator> const&)
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/common/status.h:411:5
#15 0x55861be4ea61 in doris::ThreadPool::shutdown()
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/util/threadpool.cpp:277:20
#16 0x55861aaf338c in doris::TaskWorkerPool::stop()
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/agent/task_worker_pool.cpp:246:19
#17 0x55861aad9315 in doris::TaskWorkerPool::~TaskWorkerPool()
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/agent/task_worker_pool.cpp:131:5
#18 0x55861aad9508 in doris::TaskWorkerPool::~TaskWorkerPool()
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/agent/task_worker_pool.cpp:129:35
#19 0x55861b8cb124 in
std::default_delete::operator()(doris::TaskWorkerPool*) const
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2
#20 0x55861b8c7a1b in std::unique_ptr>::~unique_ptr()
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4
#21 0x55861b8bb011 in doris::AgentServer::~AgentServer()
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/agent/agent_server.cpp:143:30
#22 0x55861b8b195c in
std::default_delete::operator()(doris::AgentServer*) const
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2
#23 0x55861b8b184b in std::unique_ptr>::~unique_ptr()
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4
#24 0x55861b8ac22a in doris::BackendService::~BackendService()
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/service/backend_service.h:68:40
#25 0x55861b8ac258 in doris::BackendService::~BackendService()
/home/zcp/repo_center/doris_branch-2.0/doris/be/src/service/backend_service.h:68:40
#26 0x55861b8b2279 in std::_Sp_counted_ptr::_M_dispose()
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:348:9
#27 0x558618683d11 in
std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:168:6
#28 0x558618683a49 in
std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count()
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:702:11
#29 0x55861b8b049a in std::__shared_ptr::~__shared_ptr()
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1149:31
#30 0x55861b8aab34 in std::shared_ptr::~shared_ptr()
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:122:11
SUMMARY: AddressSanitizer: heap-use-after-free
/var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:183:22
in std::_Head_base<2ul, unsigned long, false>::_Head_base(unsigned long const&)
Shadow bytes around the buggy address:
0x6140011f4100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x6140011f4180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x6140011f4200: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
0x6140011f4280: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x6140011f4300: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x6140011f4380: fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd fa
0x6140011f4400: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x6140011f4480: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x6140011f4500: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x6140011f4580: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x6140011f4600: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==3940985==ABORTING
```
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
