This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 64769a111dd [fix](asan) Restore __asan_handle_no_return on the throw
path bypassed by the static link (#65366)
64769a111dd is described below
commit 64769a111ddde61cd4ecd7a6b3215b7a781bb0f0
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Thu Jul 9 14:00:14 2026 +0800
[fix](asan) Restore __asan_handle_no_return on the throw path bypassed by
the static link (#65366)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #65358 (reproduction & verification branch, do-not-merge),
workaround history: #34813 #34994 #36808 #37134 #48347 #49516 #49878
Problem Summary:
For years the BE has crashed **only in ASAN builds**, on the thrift RPC
retry/`reopen` path (and, before the `#ifndef ADDRESS_SANITIZER` log
guards were added, inside glog `LOG(WARNING)`), always with the same
signature:
```
AddressSanitizer: CHECK failed: asan_thread.cpp:36x
"((ptr[0] == kCurrentStackFrameMagic)) != (0)" (0x0, 0x0)
```
Same failure family as google/glog#978 and google/sanitizers#1010. It
was never root-caused, only worked around with ASAN-only log guards and
code-path forks — and #49516 showed those forks hatch real bugs of their
own.
**Root cause.** libthrift and the other third-party static libs are
built **without** ASAN instrumentation. ASAN cleans the stack shadow on
exception unwinding via `__asan_handle_no_return()`: the compiler emits
it before instrumented throws, and the runtime provides weak
`__cxa_throw` / `_Unwind_RaiseException` interceptors to cover
un-instrumented throws. Under the BE's fully static link
(`-static-libstdc++` / `-static-libgcc` plus a static sanitizer runtime)
those **weak** interceptors lose to libstdc++.a's / libgcc_eh.a's
**strong** definitions at link time, so the cleanup never runs when
un-instrumented code throws (e.g. `TSocket` raising
`TTransportException` when the FE restarts). The unwound instrumented
frames keep their stack-shadow poison; the retry path immediately
re-descends over the same stack range, innocent locals (thrift's
`getSocketInfo()` strings, glog's `struct tm`, ...) land on the stale
poison, the first intercepted write reports a false positive, and ASAN
aborts while walking the already-overwritten frame descriptor — before
it can even print a normal report. RELEASE/DEBUG builds have no shadow
memory, which is why this never happened outside ASAN.
**The fix.** ASAN/ASAN_UT Linux links now pass
```
-Wl,--wrap=__cxa_throw -Wl,--wrap=__cxa_rethrow
-Wl,--wrap=_Unwind_RaiseException
```
and `be/src/common/asan_cxa_throw_wrap.cpp` provides wrappers that call
`__asan_handle_no_return()` before forwarding to the real
implementations — exactly what the bypassed official interceptors do.
`--wrap` rewrites every reference from every input object, including the
un-instrumented third-party archives and libstdc++.a itself. The flags
are attached to `DORIS_LINK_LIBS`, so any executable receiving them also
links `Common`, which carries the wrapper definitions — flags and
definitions cannot drift apart. `_Unwind_RaiseException` is wrapped too
because `std::rethrow_exception` and foreign-language unwinders raise
through it without touching `__cxa_throw`; its wrapper forwards the
return value (it returns when no handler is found). **Production
(RELEASE/DEBUG) builds are byte-identical.**
**Cleanup.** With the throw path fixed, the ASAN-only forks are removed
— 15 `#ifndef ADDRESS_SANITIZER` sites in `thrift_rpc_helper.cpp` (6),
`agent/utils.cpp` (6), `runtime_query_statistics_mgr.cpp` (2, one of
which skipped the whole reopen+retry under ASAN, the failure pattern
#49516 exposed), `pipeline_fragment_context.cpp` (1), plus the leftover
`std::cerr` LOG substitutes. ASAN CI now exercises the exact production
retry/reopen/logging code.
**Verification.** Done on the ASAN pipeline via the dedicated
reproduction branch #65358, which carries a deterministic startup
reproducer (not part of this PR):
| | unfixed (baseline) | with this fix |
|---|---|---|
| interceptor check | `__cxa_throw ... NOT intercepted (bypassed by
static link)` | `WRAPPED by __wrap___cxa_throw` |
| stale shadow poison after an un-instrumented throw | 58/160 probed
frames poisoned | 0/160 |
| driving the historical crash site (`report()` fail → `close()` →
`flush()` → `getSocketInfo()`, swept over 257 stack offsets) |
deterministic startup crash with the historical stack | survives the
full sweep, BE starts and runs the regression |
---
be/CMakeLists.txt | 14 ++++
be/src/agent/utils.cpp | 14 +---
be/src/common/asan_cxa_throw_wrap.cpp | 93 ++++++++++++++++++++++
be/src/exec/pipeline/pipeline_fragment_context.cpp | 4 +-
be/src/runtime/runtime_query_statistics_mgr.cpp | 8 --
be/src/util/thrift_rpc_helper.cpp | 13 ---
6 files changed, 109 insertions(+), 37 deletions(-)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index fa7e716df92..074e16af419 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -791,6 +791,20 @@ if ("${CMAKE_BUILD_TYPE}" STREQUAL "DEBUG" OR
"${CMAKE_BUILD_TYPE}" STREQUAL "RE
set(DORIS_LINK_LIBS ${DORIS_LINK_LIBS} ${MALLOCLIB})
elseif ("${CMAKE_BUILD_TYPE}" STREQUAL "ASAN" OR "${CMAKE_BUILD_TYPE}"
STREQUAL "ASAN_UT")
set(DORIS_LINK_LIBS ${DORIS_LINK_LIBS} ${ASAN_LIBS})
+ if (OS_LINUX)
+ # The fully static link resolves __cxa_throw/__cxa_rethrow/
+ # _Unwind_RaiseException to libstdc++.a's/libgcc_eh.a's strong
+ # definitions, silently bypassing the ASAN runtime's weak interceptors,
+ # so __asan_handle_no_return() never runs when un-instrumented
+ # third-party code (libthrift, ...) throws. The unwound instrumented
+ # frames keep their stack-shadow poison and the next deep call over the
+ # same stack range aborts with the kCurrentStackFrameMagic CHECK
+ # (the thrift retry/reopen crashes; same family as google/glog#978 and
+ # google/sanitizers#1010). Redirect every reference to wrappers that
+ # restore the shadow cleanup. See
be/src/common/asan_cxa_throw_wrap.cpp.
+ set(DORIS_LINK_LIBS ${DORIS_LINK_LIBS} -Wl,--wrap=__cxa_throw
-Wl,--wrap=__cxa_rethrow
+ -Wl,--wrap=_Unwind_RaiseException)
+ endif()
elseif ("${CMAKE_BUILD_TYPE}" STREQUAL "LSAN")
set(DORIS_LINK_LIBS ${DORIS_LINK_LIBS} ${LSAN_LIBS})
elseif ("${CMAKE_BUILD_TYPE}" STREQUAL "UBSAN")
diff --git a/be/src/agent/utils.cpp b/be/src/agent/utils.cpp
index 8064eef7ac7..df8d21718cc 100644
--- a/be/src/agent/utils.cpp
+++ b/be/src/agent/utils.cpp
@@ -91,18 +91,14 @@ Status MasterServerClient::finish_task(const
TFinishTaskRequest& request, TMaste
try {
try {
client->finishTask(*result, request);
- } catch ([[maybe_unused]] TTransportException& e) {
-#ifndef ADDRESS_SANITIZER
+ } catch (TTransportException& e) {
LOG(WARNING) << "master client, retry finishTask: " << e.what();
-#endif
client_status = client.reopen(config::thrift_rpc_timeout_ms);
if (!client_status.ok()) {
-#ifndef ADDRESS_SANITIZER
LOG(WARNING) << "fail to get master client from cache. "
<< "host=" <<
_cluster_info->master_fe_addr.hostname
<< ", port=" << _cluster_info->master_fe_addr.port
<< ", code=" << client_status.code();
-#endif
return Status::RpcError("Master client finish task failed");
}
client->finishTask(*result, request);
@@ -137,19 +133,15 @@ Status MasterServerClient::report(const TReportRequest&
request, TMasterResult*
} catch (TTransportException& e) {
TTransportException::TTransportExceptionType type = e.getType();
if (type !=
TTransportException::TTransportExceptionType::TIMED_OUT) {
-#ifndef ADDRESS_SANITIZER
// if not TIMED_OUT, retry
LOG(WARNING) << "master client, retry finishTask: " <<
e.what();
-#endif
client_status = client.reopen(config::thrift_rpc_timeout_ms);
if (!client_status.ok()) {
-#ifndef ADDRESS_SANITIZER
LOG(WARNING) << "fail to get master client from cache. "
<< "host=" <<
_cluster_info->master_fe_addr.hostname
<< ", port=" <<
_cluster_info->master_fe_addr.port
<< ", code=" << client_status.code();
-#endif
return Status::InternalError("Fail to get master client
from cache");
}
@@ -157,9 +149,7 @@ Status MasterServerClient::report(const TReportRequest&
request, TMasterResult*
} else {
// TIMED_OUT exception. do not retry
// actually we don't care what FE returns.
-#ifndef ADDRESS_SANITIZER
LOG(WARNING) << "fail to report to master: " << e.what();
-#endif
return Status::InternalError("Fail to report to master");
}
}
@@ -193,10 +183,8 @@ Status MasterServerClient::confirm_unused_remote_files(
} catch (TTransportException& e) {
TTransportException::TTransportExceptionType type = e.getType();
if (type !=
TTransportException::TTransportExceptionType::TIMED_OUT) {
-#ifndef ADDRESS_SANITIZER
// if not TIMED_OUT, retry
LOG(WARNING) << "master client, retry finishTask: " <<
e.what();
-#endif
client_status = client.reopen(config::thrift_rpc_timeout_ms);
if (!client_status.ok()) {
diff --git a/be/src/common/asan_cxa_throw_wrap.cpp
b/be/src/common/asan_cxa_throw_wrap.cpp
new file mode 100644
index 00000000000..256329aa335
--- /dev/null
+++ b/be/src/common/asan_cxa_throw_wrap.cpp
@@ -0,0 +1,93 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Fix for the long-standing ASAN-only BE crashes on the thrift RPC
+// retry/reopen path (and, before the LOG guards were added, inside glog's
+// LOG(WARNING)), which always aborted with:
+//
+// AddressSanitizer: CHECK failed: asan_thread.cpp:...
+// "((ptr[0] == kCurrentStackFrameMagic)) != (0)" (0x0, 0x0)
+//
+// (Same failure family as google/glog#978 and google/sanitizers#1010.)
+//
+// Root cause: libthrift and the other third-party static libs are built
+// WITHOUT ASAN instrumentation. ASAN cleans up the stack shadow on exception
+// unwinding via __asan_handle_no_return(), emitted by the compiler before
+// instrumented throws and provided as weak __cxa_throw/_Unwind_RaiseException
+// interceptors in the runtime for un-instrumented throws. Under the BE's
+// fully static link (-static-libstdc++/-static-libgcc plus a static sanitizer
+// runtime) the linker resolves those symbols to libstdc++.a's/libgcc_eh.a's
+// STRONG definitions, silently discarding the weak interceptors. So when
+// un-instrumented code throws (e.g. TSocket raising TTransportException on a
+// FE restart), the unwound instrumented frames keep their stack-shadow
+// poison; the retry path immediately re-descends over the same stack range,
+// innocent locals of thrift/glog/libc land on the stale poison, the first
+// intercepted write reports a false positive, and ASAN aborts while walking
+// the already-overwritten frame descriptor -- before it can even print a
+// normal report. RELEASE/DEBUG builds have no shadow memory, which is why
+// this never happened outside ASAN.
+//
+// ASAN builds therefore link with
+// -Wl,--wrap=__cxa_throw -Wl,--wrap=__cxa_rethrow
+// -Wl,--wrap=_Unwind_RaiseException
+// (see the ASAN branch in be/CMakeLists.txt). --wrap redirects every
+// reference from every input object -- including the members of the
+// un-instrumented third-party archives and of libstdc++.a itself -- to the
+// wrappers below, which restore exactly what the official interceptors do:
+// wipe the stack shadow above the throw point, then forward to the real
+// implementation (__real___cxa_throw resolves to libstdc++'s definition).
+//
+// _Unwind_RaiseException is wrapped as well because not every raise funnels
+// through __cxa_throw: std::rethrow_exception (libstdc++'s eh_ptr.cc) and
+// foreign-language unwinders call it directly. This matches the official
+// ASAN interceptor set for the unwind entry points. Throws via __cxa_throw
+// run the shadow wipe twice (once per wrapper); that is idempotent and is
+// also what happens with the stock interceptors.
+
+#if defined(ADDRESS_SANITIZER) && defined(__linux__)
+
+extern "C" {
+
+void __asan_handle_no_return();
+void __real___cxa_throw(void* thrown_exception, void* tinfo, void
(*dest)(void*));
+void __real___cxa_rethrow();
+int __real__Unwind_RaiseException(void* exception_object);
+
+void __wrap___cxa_throw(void* thrown_exception, void* tinfo, void
(*dest)(void*));
+void __wrap___cxa_rethrow();
+int __wrap__Unwind_RaiseException(void* exception_object);
+
+void __wrap___cxa_throw(void* thrown_exception, void* tinfo, void
(*dest)(void*)) {
+ __asan_handle_no_return();
+ __real___cxa_throw(thrown_exception, tinfo, dest);
+}
+
+void __wrap___cxa_rethrow() {
+ __asan_handle_no_return();
+ __real___cxa_rethrow();
+}
+
+// Unlike the two above this one RETURNS when no handler is found (the caller
+// then calls std::terminate), so the return value must be forwarded.
+int __wrap__Unwind_RaiseException(void* exception_object) {
+ __asan_handle_no_return();
+ return __real__Unwind_RaiseException(exception_object);
+}
+
+} // extern "C"
+
+#endif // ADDRESS_SANITIZER && __linux__
diff --git a/be/src/exec/pipeline/pipeline_fragment_context.cpp
b/be/src/exec/pipeline/pipeline_fragment_context.cpp
index f8bd3120846..5664042dd24 100644
--- a/be/src/exec/pipeline/pipeline_fragment_context.cpp
+++ b/be/src/exec/pipeline/pipeline_fragment_context.cpp
@@ -2535,12 +2535,10 @@ void
PipelineFragmentContext::_coordinator_callback(const ReportStatusRequest& r
try {
try {
(*coord)->reportExecStatus(res, params);
- } catch ([[maybe_unused]]
apache::thrift::transport::TTransportException& e) {
-#ifndef ADDRESS_SANITIZER
+ } catch (apache::thrift::transport::TTransportException& e) {
LOG(WARNING) << "Retrying ReportExecStatus. query id: " <<
print_id(req.query_id)
<< ", instance id: " <<
print_id(req.fragment_instance_id) << " to "
<< req.coord_addr << ", err: " << e.what();
-#endif
rpc_status = coord->reopen();
if (!rpc_status.ok()) {
diff --git a/be/src/runtime/runtime_query_statistics_mgr.cpp
b/be/src/runtime/runtime_query_statistics_mgr.cpp
index 0ac505c7f4b..1533897855c 100644
--- a/be/src/runtime/runtime_query_statistics_mgr.cpp
+++ b/be/src/runtime/runtime_query_statistics_mgr.cpp
@@ -69,7 +69,6 @@ static Status _do_report_exec_stats_rpc(const
TNetworkAddress& coor_addr,
try {
rpc_client->reportExecStatus(res, req);
} catch (const apache::thrift::transport::TTransportException& e) {
-#ifndef ADDRESS_SANITIZER
LOG_WARNING("Transport exception from {}, reason: {}, reopening",
PrintThriftNetworkAddress(coor_addr), e.what());
client_status = rpc_client.reopen(config::thrift_rpc_timeout_ms);
@@ -79,9 +78,6 @@ static Status _do_report_exec_stats_rpc(const
TNetworkAddress& coor_addr,
}
rpc_client->reportExecStatus(res, req);
-#else
- return Status::RpcError("Transport exception when report query
profile, {}", e.what());
-#endif
}
} catch (apache::thrift::TApplicationException& e) {
if (e.getType() == e.UNKNOWN_METHOD) {
@@ -433,13 +429,9 @@ void
RuntimeQueryStatisticsMgr::report_runtime_query_statistics() {
rpc_result[addr] = true;
} catch (apache::thrift::transport::TTransportException& e) {
rpc_status = reopen_coord();
-#ifndef ADDRESS_SANITIZER
LOG_WARNING(
"[report_query_statistics] report to fe {} failed,
reason:{}, try reopen.",
add_str, e.what());
-#else
- std::cerr << "thrift error, reason=" << e.what();
-#endif
if (rpc_status.ok()) {
coord->reportExecStatus(res, params);
rpc_result[addr] = true;
diff --git a/be/src/util/thrift_rpc_helper.cpp
b/be/src/util/thrift_rpc_helper.cpp
index 405e00f8581..05cfc2ffc00 100644
--- a/be/src/util/thrift_rpc_helper.cpp
+++ b/be/src/util/thrift_rpc_helper.cpp
@@ -77,21 +77,16 @@ Status
ThriftRpcHelper::rpc(std::function<TNetworkAddress()> address_provider,
DBUG_EXECUTE_IF("thriftRpcHelper.rpc.error", { timeout_ms = 30000; });
ClientConnection<T> client(_s_exec_env->get_client_cache<T>(), address,
timeout_ms, &status);
if (!status.ok()) {
-#ifndef ADDRESS_SANITIZER
LOG(WARNING) << "Connect frontend failed, address=" << address << ",
status=" << status;
-#endif
return status;
}
try {
try {
callback(client);
} catch (apache::thrift::transport::TTransportException& e) {
- std::cerr << "thrift error, reason=" << e.what();
-#ifndef ADDRESS_SANITIZER
LOG(WARNING) << "retrying call frontend service after "
<< config::thrift_client_retry_interval_ms << " ms,
address=" << address
<< ", reason=" << e.what();
-#endif
std::this_thread::sleep_for(
std::chrono::milliseconds(config::thrift_client_retry_interval_ms));
TNetworkAddress retry_address = address_provider();
@@ -99,37 +94,29 @@ Status
ThriftRpcHelper::rpc(std::function<TNetworkAddress()> address_provider,
return Status::Error<ErrorCode::SERVICE_UNAVAILABLE>("FE
address is not available");
}
if (retry_address.hostname != address.hostname ||
retry_address.port != address.port) {
-#ifndef ADDRESS_SANITIZER
LOG(INFO) << "retrying call frontend service with new
address=" << retry_address;
-#endif
Status retry_status;
ClientConnection<T>
retry_client(_s_exec_env->get_client_cache<T>(), retry_address,
timeout_ms, &retry_status);
if (!retry_status.ok()) {
-#ifndef ADDRESS_SANITIZER
LOG(WARNING) << "Connect frontend failed, address=" <<
retry_address
<< ", status=" << retry_status;
-#endif
return retry_status;
}
callback(retry_client);
} else {
status = client.reopen(timeout_ms);
if (!status.ok()) {
-#ifndef ADDRESS_SANITIZER
LOG(WARNING) << "client reopen failed. address=" << address
<< ", status=" << status;
-#endif
return status;
}
callback(client);
}
}
} catch (apache::thrift::TException& e) {
-#ifndef ADDRESS_SANITIZER
LOG(WARNING) << "call frontend service failed, address=" << address
<< ", reason=" << e.what();
-#endif
std::this_thread::sleep_for(
std::chrono::milliseconds(config::thrift_client_retry_interval_ms * 2));
// just reopen to disable this connection
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
