This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ae6eacaac66 [opt](build) Add DORIS_DEV_DEBUG_INFO=line-tables and
default the UT / macOS compile-check builds to it (#66960)
ae6eacaac66 is described below
commit ae6eacaac66f134204edaa1982ba36774febfefd
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Thu Aug 20 14:13:20 2026 +0800
[opt](build) Add DORIS_DEV_DEBUG_INFO=line-tables and default the UT /
macOS compile-check builds to it (#66960)
### What problem does this PR solve?
Issue Number: close #66715
Related PR: #66510 (the BE build-time series this closes out)
Problem Summary:
BE compiles every TU with `-g -gdwarf-5`, and on this codebase that
debug info is not a rounding error — it is **95% of the bytes the
compiler writes**. Across the 7426 objects of a full Release build:
`__text` 150 MB, `.debug_*` **4.01 GB**. Type-level DWARF describes
every type a TU touched, so it scales with the header closure rather
than with the code.
Line tables alone keep stack traces, `perf` and `addr2line` resolving to
`file:line`; what they drop is variable-level DWARF (`print some_var` in
gdb/lldb).
This PR adds the switch and turns it on where nobody debugs.
**1. The switch** (`DORIS_DEV_DEBUG_INFO`, values `line-tables` /
`full`):
```bash
DORIS_DEV_DEBUG_INFO=line-tables ./build.sh --be
```
`-gline-tables-only` is appended **after** the existing `-g -gdwarf-5`
(clang lets the last debug-info flag win). Appending rather than
replacing is the point: with the switch unset the `if` contributes
nothing, so `build.sh` keeps producing byte-identical command lines. An
unrecognized value is a configure FATAL; `line-tables` requires clang
(gcc has no such flag).
Measured — macOS arm64, clang 20.1.8, `./build.sh --compile-bench -j 6`,
PCH on, ccache off, build dir recreated every run, same commit, three
back-to-back runs (default → line-tables → default, so drift is measured
rather than assumed):
| | default (run 1) | `line-tables` | default (run 3) | effect vs. mean
of defaults |
|---|---|---|---|---|
| cold build wall | 12m02s | **10m34s** | 11m52s | **-11.6%** |
| build phase wall | 11m28s | 10m03s | 11m22s | -11.9% |
| sum of TU wall | 3821s | 3341s | 3781s | -12.1% |
| sum of TU cpu | 3774s | 3281s | 3733s | -12.6% |
Machine drift between the two default runs: **-1.3%**, an order of
magnitude below the effect. Artifacts: objects 4.25 GB → 0.99 GB
(**-76.8%**), build dir 9.4 → 2.8 GB, type-level `.debug_*` 4.01 → 0.51
GB, heaviest TU peak RSS 3798 → 2618 MB (-31%), `doris_be` `__text`
**byte-identical**. Per file: 207 TUs improved, 16 regressed (all ≤
+2.6s, one target, scheduling noise).
**2. `run-be-ut.sh` defaults to `line-tables`.** The UT build has its
own cmake invocation, and it is the build a developer repeats most —
`MAKE_TEST=ON` adds all of `be/test` on top of `be/src`. A UT build is a
compile-and-run loop; when a test fails, the stack trace still resolves.
Sampled on 10 real `be/test` TUs (Debug, PCH on, same command with and
without the flag):
| | full | `line-tables` | delta |
|---|---|---|---|
| compile cpu | 55.0s | 51.5s | **-6.3%** |
| objects | 91.4 MB | 8.1 MB | **-91.1%** |
Smaller time win than the Release path (at `-O0`/`-Og` the DWARF is
bulky but cheap to emit), much larger size win.
`DORIS_DEV_DEBUG_INFO=full sh run-be-ut.sh` restores the full DWARF for
a debugger session. The default only applies under clang, so a
`DORIS_TOOLCHAIN=gcc` UT build is untouched rather than hitting the
clang-only FATAL.
**3. `be-ut-mac.yml` sets it explicitly.** That job builds with
`-DMAKE_TEST=OFF` — it only checks that the BE compiles on macOS, runs
no test and keeps no binary — so its full DWARF is pure cost. It is the
same `build.sh --be` path measured above (-11.6%), and its 5G ccache no
longer has to hold 4.25 GB of objects per build.
**Disclosure / blind spots:**
- All numbers are macOS/clang20; Linux is covered only by this PR's CI.
The gcc path was not exercised locally (no gcc toolchain on the bench
machine); it is guarded statically by `NOT COMPILER_CLANG` and by the
clang-only default in `run-be-ut.sh`.
- **No link-time win on macOS**: ld64 leaves DWARF in the objects and
builds a debug map, so `doris_be` is 336 MB either way (`__text`
byte-identical, 176 B of file difference) and the link is 0.9s → 0.8s.
On Linux ELF the linker *does* copy `.debug_*` into the binary, so the
binary should shrink there — not measured, so no number is claimed.
- The switch reaches BE plus the contrib targets that inherit the
directory options; `clucene` (347 TUs) and `apache-orc` (48 TUs) set
their own flags and keep full debug info.
- Changing the UT default changes the compile command line, so the first
UT build after this lands is a full rebuild and does not reuse existing
ccache entries. One-time cost.
- Interactive UT debugging loses variable inspection by default. That is
the trade this PR makes deliberately for the compile-and-run loop;
`DORIS_DEV_DEBUG_INFO=full` is the way back, and `build.sh` still
defaults to full DWARF.
- Orthogonal to `STRIP_DEBUG_INFO=ON`, which splits debug info off
**after** linking (saves binary/deploy size, saves no compile time — the
DWARF was already generated). This switch avoids generating it at all.
### Release note
None
### Check List (For Author)
- Test
- [x] Manual test (add detailed scripts or steps below)
- Three paired cold benchmark builds (`build.sh --compile-bench -j 6`),
same commit, same machine, back to back, the third quantifying drift.
- Sampled A/B over 10 real `be/test` TUs from the UT compile database,
same command with and without the flag, CPU time and object size.
- Verified the flag reaches the compiler on the `--be` path: 7545 of
7940 `compile_commands.json` entries carry `-gline-tables-only` (the
rest are the two flag-overriding contrib targets).
- Verified it on the UT path (`BUILD_TYPE_UT=Debug run-be-ut.sh`): 8741
of 9136 entries, same 395 contrib entries excepted; and that the new
default produces the same result with no environment variable set, while
`DORIS_DEV_DEBUG_INFO=full` produces zero flagged entries.
- Verified emitted code is unaffected: `doris_be` `__text`
byte-identical between the default and `line-tables` builds.
- Verified the unknown-value path: `DORIS_DEV_DEBUG_INFO=bogus` stops
configure with `Unknown DORIS_DEV_DEBUG_INFO value: bogus (supported:
line-tables, full)`.
- Behavior changed:
- [x] Yes. BE **unit-test** builds (`run-be-ut.sh`) and the macOS
BE-compile CI job now default to line-tables-only debug info: stack
traces, `perf` and `addr2line` are unaffected; variable-level DWARF is
not emitted unless `DORIS_DEV_DEBUG_INFO=full` is set. `build.sh` itself
still defaults to full debug info, and release builds are unchanged.
- Does this need documentation?
- [x] No. `build.sh -h` documents the switch and both values;
`run-be-ut.sh` prints the level it is building with.
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
---
.github/workflows/be-ut-mac.yml | 6 ++++++
be/CMakeLists.txt | 15 +++++++++++++++
build.sh | 3 +++
run-be-ut.sh | 11 +++++++++++
4 files changed, 35 insertions(+)
diff --git a/.github/workflows/be-ut-mac.yml b/.github/workflows/be-ut-mac.yml
index 25298adcec0..d970796dd21 100644
--- a/.github/workflows/be-ut-mac.yml
+++ b/.github/workflows/be-ut-mac.yml
@@ -142,5 +142,11 @@ jobs:
# compiled. Skip the Java extensions and cdc client (not relevant to
a
# macOS C++ build check), and pass -j explicitly because build.sh
# otherwise defaults to only ~nproc/4 jobs.
+ #
+ # Nothing here is ever debugged and no binary is kept, so full DWARF
is
+ # pure cost: line tables alone cut this build by ~12% and shrink the
+ # objects from 4.25GB to 0.99GB, which also keeps the 5G ccache from
+ # thrashing between runs.
DISABLE_BE_JAVA_EXTENSIONS=ON DISABLE_BE_CDC_CLIENT=ON \
+ DORIS_DEV_DEBUG_INFO=line-tables \
./build.sh --be -j "$(nproc)"
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index 093a787cd9f..eacfd27db87 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -399,6 +399,21 @@ add_compile_options(-g
-fno-omit-frame-pointer
$<$<COMPILE_LANGUAGE:CXX>:-Wnon-virtual-dtor>)
+# Faster dev compiles: DORIS_DEV_DEBUG_INFO=line-tables keeps only the line
+# tables (enough for stack traces and perf) instead of full debug info; 'full'
+# asks for the default full DWARF explicitly, which is what run-be-ut.sh needs
+# to be overridable with, since UT builds default to line-tables.
+# It must be appended after the -g/-gdwarf-5 above: clang lets the last
debug-info
+# flag win, and this keeps the default command line unchanged.
+if ("${DORIS_DEV_DEBUG_INFO}" STREQUAL "line-tables")
+ if (NOT COMPILER_CLANG)
+ message(FATAL_ERROR "DORIS_DEV_DEBUG_INFO=line-tables requires clang")
+ endif()
+ add_compile_options(-gline-tables-only)
+elseif (NOT "${DORIS_DEV_DEBUG_INFO}" STREQUAL "" AND NOT
"${DORIS_DEV_DEBUG_INFO}" STREQUAL "full")
+ message(FATAL_ERROR "Unknown DORIS_DEV_DEBUG_INFO value:
${DORIS_DEV_DEBUG_INFO} (supported: line-tables, full)")
+endif()
+
add_compile_options(-Wno-unused-parameter
-Wno-sign-compare)
diff --git a/build.sh b/build.sh
index 7168a355d68..a795dfec2be 100755
--- a/build.sh
+++ b/build.sh
@@ -85,6 +85,7 @@ Usage: $0 <options>
ENABLE_DYNAMIC_ARCH If set ENABLE_DYNAMIC_ARCH=ON, it will enable
dynamic CPU detection in OpenBLAS. Default is ON. Can also use
--enable-dynamic-arch flag.
ARM_MARCH Specify the ARM architecture instruction set.
Default is armv8-a+crc.
STRIP_DEBUG_INFO If set STRIP_DEBUG_INFO=ON, the debug
information in the compiled binaries will be stored separately in the
'be/lib/debug_info' directory. Default is OFF.
+ DORIS_DEV_DEBUG_INFO Debug info level for the BE: 'line-tables'
compiles with -gline-tables-only for faster dev builds (clang only; keeps line
tables for stack traces, drops variable-level DWARF), 'full' is the full debug
info. Default is 'full' here; run-be-ut.sh defaults to 'line-tables'.
DISABLE_BE_JAVA_EXTENSIONS If set DISABLE_BE_JAVA_EXTENSIONS=ON, we will
do not build binary with java-udf,hadoop-hudi-scanner,jdbc-scanner and so on
Default is OFF.
DISABLE_JAVA_CHECK_STYLE If set DISABLE_JAVA_CHECK_STYLE=ON, it will
skip style check of java code in FE.
DISABLE_BUILD_AZURE If set DISABLE_BUILD_AZURE=ON, it will not
build azure into BE.
@@ -762,6 +763,7 @@ echo "Get params:
USE_AVX2 -- ${USE_AVX2}
USE_LIBCPP -- ${USE_LIBCPP}
STRIP_DEBUG_INFO -- ${STRIP_DEBUG_INFO}
+ DORIS_DEV_DEBUG_INFO -- ${DORIS_DEV_DEBUG_INFO}
USE_JEMALLOC -- ${USE_JEMALLOC}
USE_BTHREAD_SCANNER -- ${USE_BTHREAD_SCANNER}
ENABLE_INJECTION_POINT -- ${ENABLE_INJECTION_POINT}
@@ -943,6 +945,7 @@ if [[ "${BUILD_BE}" -eq 1 ]]; then
-DBUILD_FILE_CACHE_MICROBENCH_TOOL="${BUILD_FILE_CACHE_MICROBENCH_TOOL}" \
-DBUILD_INDEX_TOOL="${BUILD_INDEX_TOOL}" \
-DSTRIP_DEBUG_INFO="${STRIP_DEBUG_INFO}" \
+ -DDORIS_DEV_DEBUG_INFO="${DORIS_DEV_DEBUG_INFO}" \
-DDISPLAY_BUILD_TIME="${DISPLAY_BUILD_TIME}" \
-DENABLE_PCH="${ENABLE_PCH}" \
-DENABLE_UNITY_BUILD="${ENABLE_UNITY_BUILD:-ON}" \
diff --git a/run-be-ut.sh b/run-be-ut.sh
index d04af7036ed..d0470b64bb8 100755
--- a/run-be-ut.sh
+++ b/run-be-ut.sh
@@ -307,10 +307,20 @@ if [[ -z "${ENABLE_INJECTION_POINT}" ]]; then
ENABLE_INJECTION_POINT='ON'
fi
+# The UT build is a compile-and-run loop, not a debugger session: line tables
+# keep stack traces, perf and addr2line working, and drop the variable-level
+# DWARF that dominates the object files (sampled be/test TUs: -6% compile cpu,
+# objects -91%). Ask for the full DWARF back with DORIS_DEV_DEBUG_INFO=full.
+# Only default it in under clang -- gcc has no -gline-tables-only.
+if [[ -z "${DORIS_DEV_DEBUG_INFO}" ]] && [[ "${DORIS_TOOLCHAIN}" == 'clang'
]]; then
+ DORIS_DEV_DEBUG_INFO='line-tables'
+fi
+
MAKE_PROGRAM="$(command -v "${BUILD_SYSTEM}")"
echo "-- Make program: ${MAKE_PROGRAM}"
echo "-- Use ccache: ${CMAKE_USE_CCACHE_CXX} and ${CMAKE_USE_CCACHE_C}"
echo "-- Extra cxx flags: ${EXTRA_CXX_FLAGS:-}"
+echo "-- Debug info: ${DORIS_DEV_DEBUG_INFO:-full}"
if [[ "${CMAKE_BUILD_TYPE}" = "ASAN" ]]; then
BUILD_TYPE="ASAN_UT"
@@ -337,6 +347,7 @@ cd "${CMAKE_BUILD_DIR}"
${CMAKE_USE_CCACHE_C:+${CMAKE_USE_CCACHE_C}} \
-DENABLE_PCH="${ENABLE_PCH}" \
-DENABLE_UNITY_BUILD="${ENABLE_UNITY_BUILD:-ON}" \
+ -DDORIS_DEV_DEBUG_INFO="${DORIS_DEV_DEBUG_INFO}" \
-DDORIS_JAVA_HOME="${JAVA_HOME}" \
-DBUILD_AZURE="${BUILD_AZURE}" \
"${BE_EXTRA_CMAKE_ARGS[@]}" \
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
