This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 165efa32a9d [opt](build) Add build-timing and header-closure sweep
tooling (#66616)
165efa32a9d is described below
commit 165efa32a9d54ac575cb8403a4dd7d6273c52de2
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Tue Aug 11 13:56:02 2026 +0800
[opt](build) Add build-timing and header-closure sweep tooling (#66616)
> Split out of **https://github.com/apache/doris/pull/66510**. That PR
carries the
> whole BE build-time batch and its end-to-end measurements — **please
refer to
> #66510 for the complete benefit numbers**. This PR only lands the
measurement
> tooling that batch was built on, so that the later
header/instantiation PRs
> have a gate to point at. It has no dependency on the rest and can be
reviewed
> and merged on its own.
### What problem does this PR solve?
Related PR: #66510
Problem Summary:
Three measurement tools under `build-support/compile-bench/`, plus one
real bug
the tools found. None of them enters the build graph.
#### 1. `closure-sweep.sh` + `syntax_sweep.py --no-pch` — the safety net
for header surgery
Any PR that cuts an include edge risks breaking a TU that was only
compiling
because some other header dragged the symbol in. The existing
`syntax_sweep.py`
could not catch that class of debt, for two reasons:
- **With the PCH active, transitive-dependency debt is invisible.**
Symbols leak
out of `cmake_pch.hxx` into every TU, so a TU that forgot an `#include`
still
compiles. `--no-pch` strips the PCH preamble and runs each TU
`-fsyntax-only` against its **natural** include closure. As a bonus it
keeps
the sweep usable while a header included by `pch.h` has been edited but
the
`.pch` is stale.
- **109 compilation units were never being swept at all.** The scan
filtered
`compile_commands.json` by the `be/src` path prefix, but CMake unity
batches
emit their `.cxx` under the *build* directory, so every unity-covered TU
was
silently skipped. Including them takes the real scan surface to 191
standalone
TUs + 109 unity batches = 300 compilation units.
`closure-sweep.sh` wraps this into a gate: it archives per-tag failure
lists
under `be/compile-bench-results/sweeps/`, records a baseline (existing
debt), and
in diff mode exits 1 on any *new* failure versus that baseline.
#### 2. What the net caught on its first run
`DeleteBitmap::diffset` in `tablet_meta.cpp` uses `std::ranges::views`
but never
includes `<ranges>` — the TU only compiled because the PCH leaked it in
via
simdjson. Exactly the class of debt the `--no-pch` mode exists to find,
so the
one-line fix ships here as the tool's first result.
#### 3. `rebuild_radius.py` — how many objects does touching this header
rebuild?
Counts, per header, the object targets that list it in ninja's
dependency
database, and flags the ones reachable from `pch.h` — those rebuild
every
first-party object no matter how few TUs actually use them, which is
what makes
them worth cutting.
The obvious alternative, `touch <header> && ninja -n | wc -l`, **cannot
work in
this repo**: `CONFIGURE_DEPENDS` globs stop the dry run at `Re-running
CMake`
before it lists a single compile edge. Hence the `ninja -t deps`
approach.
#### 4. `tu-bench.sh` — single-TU compile timing probe
Companion to `--compile-bench` for before/after comparison of one
translation
unit. Looks up the TU's real command in `compile_commands.json`, strips
the
binary PCH load (keeping the textual include of `cmake_pch.hxx` so
forced-header
semantics stay identical), and reports wall seconds, max RSS,
weak-definition
count and total `.text` bytes, appending one TSV line per invocation.
Runs on
both Linux and macOS (`/usr/bin/time -v`, `size -A` and the ELF
weak-definition
nm letters are all GNU-only; the Darwin paths use the Apple
equivalents).
### Release note
None
### Check List (For Author)
- Test
- [x] No need to test or manual test. Explain why:
- [x] Other reason: everything here lives under `build-support/` and is
not part of the build graph — no target compiles, links or runs these
scripts. The single source change, `#include <ranges>` in
`tablet_meta.cpp`, is a missing include for code already in the tree
and is covered by the existing build.
- [x] Manual test — the tools were exercised throughout #66510: the
300-unit
closure sweep ran clean (300/300) against the natural-closure baseline
after each header-surgery wave, and `tu-bench.sh` / `rebuild_radius.py`
produced the per-TU and rebuild-radius numbers quoted there.
- Behavior changed:
- [x] No.
- Does this need documentation?
- [x] No. Each script self-documents via `--help` / a header comment.
### Proactive disclosure
- All three tools were developed and run on macOS / clang 20. They are
POSIX-shell and Python 3 with no exotic dependencies, and `tu-bench.sh`
explicitly branches Linux vs Darwin, but the Linux paths of
`tu-bench.sh` have
had far less mileage than the Darwin ones.
- `rebuild_radius.py` reads ninja's dep database, so it needs a
*populated*
build directory to be meaningful; on a fresh configure it reports
nothing.
- The sweep gate is opt-in — nothing in CI invokes it. Wiring it into CI
would be
a separate discussion (the full 300-unit `-fsyntax-only` pass is not
free).
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
---
be/src/storage/tablet/tablet_meta.cpp | 1 +
build-support/compile-bench/closure-sweep.sh | 93 ++++++++++++++++++
build-support/compile-bench/rebuild_radius.py | 116 ++++++++++++++++++++++
build-support/compile-bench/syntax_sweep.py | 120 +++++++++++++++++------
build-support/compile-bench/tu-bench.sh | 132 ++++++++++++++++++++++++++
5 files changed, 433 insertions(+), 29 deletions(-)
diff --git a/be/src/storage/tablet/tablet_meta.cpp
b/be/src/storage/tablet/tablet_meta.cpp
index 1f7aac050ed..4fd34f1122e 100644
--- a/be/src/storage/tablet/tablet_meta.cpp
+++ b/be/src/storage/tablet/tablet_meta.cpp
@@ -31,6 +31,7 @@
#include <cstdint>
#include <memory>
#include <random>
+#include <ranges>
#include <set>
#include <utility>
diff --git a/build-support/compile-bench/closure-sweep.sh
b/build-support/compile-bench/closure-sweep.sh
new file mode 100755
index 00000000000..c42b0265c7b
--- /dev/null
+++ b/build-support/compile-bench/closure-sweep.sh
@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Safety net for header-closure surgery (Phase 3 of the BE build-speed work).
+#
+# Runs syntax_sweep.py in natural-closure mode (-fsyntax-only, PCH stripped)
+# over every first-party TU and archives the failure list, so that header
+# edits can be gated on "no NEW failures versus a recorded baseline".
+# Baseline failures are pre-existing "compiles only via PCH symbol leak"
+# debt; they matter the moment pch.h stops including the leaking header.
+#
+# Usage:
+# closure-sweep.sh baseline <tag> [syntax_sweep.py args...]
+# Full sweep; archive results under
be/compile-bench-results/sweeps/<tag>/.
+# Exit 0 even if TUs fail (failures are the recorded debt).
+#
+# closure-sweep.sh diff <base-tag> <tag> [syntax_sweep.py args...]
+# Full sweep archived as <tag>, then compared against <base-tag>:
+# prints regressions (new failures) and improvements (fixed ones).
+# Exit 1 iff there is at least one regression.
+set -uo pipefail
+
+ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
+SWEEPS="$ROOT/be/compile-bench-results/sweeps"
+
+usage() {
+ grep '^# closure-sweep.sh' "$0" >&2
+ exit 2
+}
+
+[ $# -ge 2 ] || usage
+mode=$1
+case "$mode" in
+baseline) base=""; tag=$2; shift 2 ;;
+diff) [ $# -ge 3 ] || usage; base=$2; tag=$3; shift 3 ;;
+*) usage ;;
+esac
+
+out="$SWEEPS/$tag"
+mkdir -p "$out"
+
+python3 "$ROOT/build-support/compile-bench/syntax_sweep.py" \
+ --no-pch \
+ --fail-list "$out/fails.list" \
+ --fail-log "$out/fail-errors.log" \
+ "$@" 2>&1 | tee "$out/sweep.log"
+sweep_rc=${PIPESTATUS[0]}
+touch "$out/fails.list" # sweep writes it only on failures
+
+echo
+echo "== sweep '$tag': $(wc -l <"$out/fails.list" | tr -d ' ') failing TU(s),
results in ${out#"$ROOT"/} =="
+
+if [ "$mode" = baseline ]; then
+ exit 0
+fi
+
+basef="$SWEEPS/$base/fails.list"
+if [ ! -f "$basef" ]; then
+ echo "ERROR: baseline '$base' not found at $basef" >&2
+ exit 2
+fi
+# temp files instead of process substitution: keeps the script correct even
+# when invoked as `sh closure-sweep.sh` (POSIX mode has no <(...))
+sort "$basef" >"$out/.base.sorted"
+sort "$out/fails.list" >"$out/.cur.sorted"
+regressions=$(comm -13 "$out/.base.sorted" "$out/.cur.sorted")
+fixed=$(comm -23 "$out/.base.sorted" "$out/.cur.sorted")
+if [ -n "$fixed" ]; then
+ echo "-- fixed vs '$base':"
+ echo "$fixed" | sed 's/^/ /'
+fi
+if [ -n "$regressions" ]; then
+ echo "-- REGRESSIONS vs '$base':"
+ echo "$regressions" | sed 's/^/ /' | tee "$out/regressions.list"
+ exit 1
+fi
+echo "-- no regressions vs '$base'"
+exit 0
diff --git a/build-support/compile-bench/rebuild_radius.py
b/build-support/compile-bench/rebuild_radius.py
new file mode 100644
index 00000000000..ceaa6a6e5a5
--- /dev/null
+++ b/build-support/compile-bench/rebuild_radius.py
@@ -0,0 +1,116 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Report how many objects a header change actually rebuilds.
+
+Reads ninja's dependency database (`ninja -t deps`) and counts, per header,
+the object targets that list it -- the incremental rebuild radius. Headers
+reachable from pch.h are flagged separately: a PCH dependency invalidates
+every first-party object no matter how few TUs name a symbol from it, so
+those are the ones worth cutting first.
+
+Usage:
+ python3 rebuild_radius.py [--build-dir DIR] [header-suffix ...]
+
+Header arguments are matched as path suffixes ("core/field.h"); without
+any, a default set of hot BE headers is reported.
+
+Note that `touch <header> && ninja -n | wc -l` is NOT a usable substitute
+here: CONFIGURE_DEPENDS globs make the dry run stop at "Re-running CMake"
+before it lists a single compile edge.
+"""
+
+import argparse
+import os
+import subprocess
+import sys
+
+DEFAULT_HEADERS = [
+ "storage/olap_common.h",
+ "storage/utils.h",
+ "util/uid_util.h",
+ "util/json/path_in_data.h",
+ "storage/index/zone_map/zone_map_index.h",
+ "exprs/expr_zonemap_filter.h",
+ "storage/index/inverted/inverted_index_iterator.h",
+ "storage/index/inverted/inverted_index_parser.h",
+ "storage/tablet/tablet_schema.h",
+ "util/pretty_printer.h",
+ "exprs/function/function.h",
+ "core/column/column.h",
+ "core/field.h",
+ "core/types.h",
+ "common/status.h",
+ "runtime/runtime_profile.h",
+]
+
+
+def main():
+ repo =
os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+ ap = argparse.ArgumentParser(description=__doc__,
+
formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("--build-dir",
+ default=os.path.join(repo, "be",
"build_Release_compile_bench"))
+ ap.add_argument("headers", nargs="*", help="path suffixes to report on")
+ args = ap.parse_args()
+ headers = args.headers or DEFAULT_HEADERS
+
+ counts = {h: 0 for h in headers}
+ pch_deps = set()
+ objects = 0
+
+ proc = subprocess.Popen(["ninja", "-C", args.build_dir, "-t", "deps"],
+ stdout=subprocess.PIPE, text=True,
errors="replace",
+ bufsize=1 << 20)
+ target, target_is_pch, hits = None, False, set()
+
+ def flush():
+ nonlocal objects
+ if target is None:
+ return
+ if target_is_pch:
+ pch_deps.update(hits)
+ else:
+ for h in hits:
+ counts[h] += 1
+ if target.endswith((".o", ".obj")):
+ objects += 1
+
+ for line in proc.stdout:
+ if line[:1] not in (" ", "\t", "\n"):
+ flush()
+ target = line.split(":")[0].strip()
+ target_is_pch = "cmake_pch" in target or target.endswith(".pch")
+ hits = set()
+ elif line.strip():
+ dep = line.strip()
+ for h in headers:
+ if dep.endswith(h):
+ hits.add(h)
+ flush()
+ proc.wait()
+
+ print(f"{objects:,} object targets in the deps database\n")
+ print(f"{'header':58s} {'dependents':>10s} via PCH (= rebuilds
everything)")
+ for h in headers:
+ flag = "YES" if h in pch_deps else "no"
+ print(f"{h:58s} {counts[h]:>10,} {flag}")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/build-support/compile-bench/syntax_sweep.py
b/build-support/compile-bench/syntax_sweep.py
index 0e5e2b7d1c3..e3f77bfa023 100644
--- a/build-support/compile-bench/syntax_sweep.py
+++ b/build-support/compile-bench/syntax_sweep.py
@@ -20,16 +20,25 @@
Validates include-structure changes (header cuts, forward-declaration swaps)
against every TU without mutating the ninja build state: each compile command
is replayed with -o/-c/-MD/-MT/-MF/-ftime-trace* stripped and -fsyntax-only
-appended, so nothing is written to the build directory. Front-end-only checks
-run in roughly half the time of a real compile and catch every missing-include
-or missing-declaration fallout a cut can cause.
+appended, so nothing is written to the build directory.
+
+Sweep scope: all first-party TUs -- standalone be/src entries AND unity
+batches (CMake unity .cxx live under the build dir and would be silently
+skipped by a be/src path filter; each batch textually includes its member
+.cpp files, so checking the batch checks the members). Third-party contrib
+(openblas/faiss/clucene/orc) and generated sources are excluded.
+
+--no-pch strips the PCH preamble (-include-pch / forced cmake_pch.hxx
+include) so every TU is checked against its NATURAL include closure. This is
+the mode that catches "TU compiles only because the PCH leaks symbols it
+never includes" debt -- mandatory before slimming pch.h, and the only mode
+that works while a header included by pch.h has been edited but the .pch has
+not been rebuilt.
Usage:
python3 syntax_sweep.py [--build-dir DIR] [--jobs N] [--filter SUBSTR]
- [--fail-log FILE]
-
- --filter limits the sweep to TUs whose source path contains SUBSTR
- (e.g. --filter load/memtable for a quick re-check of one subsystem).
+ [--no-pch] [--fail-list FILE] [--fail-log FILE]
+ [--timeout SECS]
Exit code: 0 if every TU passes, 1 otherwise.
"""
@@ -52,23 +61,60 @@ STRIP_WITH_ARG = ("-o", "-MT", "-MF")
STRIP_FLAGS = ("-c", "-MD", "-MMD")
-def mangle(cmd):
+def mangle(cmd, no_pch=False):
args = shlex.split(cmd)
- out, skip = [], False
- for a in args:
- if skip:
- skip = False
- continue
+ out = []
+ i = 0
+ while i < len(args):
+ a = args[i]
if a in STRIP_WITH_ARG:
- skip = True
+ i += 2
continue
if a in STRIP_FLAGS or a.startswith("-ftime-trace"):
+ i += 1
continue
+ if no_pch:
+ if a == "-Winvalid-pch":
+ i += 1
+ continue
+ # -Xclang -include-pch -Xclang <file.pch>
+ # -Xclang -include -Xclang <cmake_pch.hxx>
+ if (a == "-Xclang" and i + 3 < len(args)
+ and args[i + 1] in ("-include-pch", "-include")
+ and args[i + 2] == "-Xclang"
+ and ("cmake_pch" in args[i + 3]
+ or args[i + 3].endswith(".pch"))):
+ i += 4
+ continue
+ # plain -include <cmake_pch.hxx> form
+ if a == "-include" and i + 1 < len(args) and "cmake_pch" in args[i
+ 1]:
+ i += 2
+ continue
out.append(a)
+ i += 1
out.append("-fsyntax-only")
return out
+def display_name(path, src_prefix):
+ if path.startswith(src_prefix):
+ return path[len(src_prefix):]
+ if "/Unity/" in path:
+ # .../src/<dir>/CMakeFiles/<tgt>.dir/Unity/unity_N_cxx.cxx
+ m = re.search(r"/src/([^/]+)/CMakeFiles/[^/]+/Unity/(unity_\d+)", path)
+ if m:
+ return f"{m.group(1)}/[{m.group(2)}]"
+ return path
+
+
+def first_party(e, src_prefix):
+ f = e["file"]
+ if f.startswith(src_prefix):
+ return True
+ # CMake unity batches for first-party targets live under the build dir
+ return "/Unity/" in f and "/src/" in f
+
+
def main():
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
@@ -77,44 +123,60 @@ def main():
ap.add_argument("--jobs", type=int, default=max(2, (os.cpu_count() or 8)
// 2))
ap.add_argument("--filter", default="",
help="only sweep TUs whose path contains this substring")
+ ap.add_argument("--no-pch", action="store_true",
+ help="strip PCH preamble: check each TU's natural include
closure")
+ ap.add_argument("--fail-list", default="",
+ help="write sorted display names of failing TUs to this
file")
ap.add_argument("--fail-log", default="",
help="write full stderr of every failing TU to this file")
+ ap.add_argument("--timeout", type=int, default=600,
+ help="per-TU timeout in seconds (counts as failure)")
args = ap.parse_args()
src_prefix = os.path.join(REPO_ROOT, "be", "src") + os.sep
with open(os.path.join(args.build_dir, "compile_commands.json")) as f:
entries = [e for e in json.load(f)
- if e["file"].startswith(src_prefix) and args.filter in
e["file"]]
- print(f"{len(entries)} TUs to check ({args.jobs} jobs)", flush=True)
+ if first_party(e, src_prefix) and args.filter in e["file"]]
+ print(f"{len(entries)} TUs to check ({args.jobs} jobs"
+ f"{', natural closure / no PCH' if args.no_pch else ''})",
flush=True)
t0 = time.time()
fails = []
done = 0
def run(e):
- p = subprocess.run(mangle(e["command"]), cwd=e["directory"],
- stdout=subprocess.DEVNULL, stderr=subprocess.PIPE,
- text=True)
- return e["file"], p.returncode, p.stderr
+ cmd = mangle(e["command"], no_pch=args.no_pch)
+ name = display_name(e["file"], src_prefix)
+ try:
+ p = subprocess.run(cmd, cwd=e["directory"],
+ stdout=subprocess.DEVNULL,
stderr=subprocess.PIPE,
+ text=True, timeout=args.timeout)
+ return name, p.returncode, p.stderr
+ except subprocess.TimeoutExpired:
+ return name, 124, f"TIMEOUT after {args.timeout}s"
with concurrent.futures.ThreadPoolExecutor(args.jobs) as ex:
- for src, rc, err in ex.map(run, entries):
+ for name, rc, err in ex.map(run, entries):
done += 1
if rc != 0:
- fails.append((src, err))
- print(f"FAIL {src.replace(src_prefix, '')}", flush=True)
- if done % 100 == 0:
- print(f" …{done}/{len(entries)} ({time.time() - t0:.0f}s, "
+ fails.append((name, err))
+ print(f"FAIL {name}", flush=True)
+ if done % 50 == 0:
+ print(f" ...{done}/{len(entries)} ({time.time() - t0:.0f}s, "
f"{len(fails)} failures)", flush=True)
print(f"\n==== {len(fails)} failing TU(s) of {len(entries)} "
f"in {time.time() - t0:.0f}s ====", flush=True)
+ if args.fail_list:
+ with open(args.fail_list, "w") as f:
+ for name, _ in sorted(fails):
+ f.write(name + "\n")
if args.fail_log and fails:
with open(args.fail_log, "w") as f:
- for src, err in fails:
- f.write(f"===== {src}\n{ANSI.sub('', err)}\n")
- for src, err in fails[:15]:
+ for name, err in fails:
+ f.write(f"===== {name}\n{ANSI.sub('', err)}\n")
+ for name, err in fails[:15]:
first = [l for l in ANSI.sub("", err).splitlines() if " error: " in
l][:2]
- print(src.replace(src_prefix, ""))
+ print(name)
for l in first:
print(" ", l[:200])
return 1 if fails else 0
diff --git a/build-support/compile-bench/tu-bench.sh
b/build-support/compile-bench/tu-bench.sh
new file mode 100755
index 00000000000..03d7ecd0dac
--- /dev/null
+++ b/build-support/compile-bench/tu-bench.sh
@@ -0,0 +1,132 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Single-TU compile timing probe (no-PCH baseline).
+#
+# Looks up the TU's real compile command in compile_commands.json, strips the
+# binary PCH (keeps the textual -include of cmake_pch.hxx so the source still
+# sees the same forced headers), redirects -o to an output dir, then reports:
+# wall seconds, max RSS, weak-def count, total .text bytes
+# One line per run is appended to $OUT_DIR/results.tsv; the .o, /usr/bin/time
+# log and (by default) an -ftime-trace JSON stay in $OUT_DIR for analysis.
+#
+# Usage:
+# build-support/compile-bench/tu-bench.sh <label> <source-path> [build-dir]
+#
+# <source-path> absolute, or relative to the repo root. Unity TUs work too:
+# pass the generated .cxx path under the build dir.
+# [build-dir] defaults to be/build_Release_compile_bench
+#
+# Env:
+# TU_BENCH_OUT=<dir> output dir (default: <build-dir>/tu-bench)
+# TU_BENCH_TRACE=OFF disable -ftime-trace (default ON, matching the
+# RESEARCH.md probe baseline numbers)
+#
+# Timing hygiene: run on an idle machine, one TU at a time. Concurrent builds
+# invalidate the numbers (see plan-doc/compile-opt/HANDOFF.md).
+
+set -euo pipefail
+
+if [[ $# -lt 2 ]]; then
+ grep '^#' "$0" | sed 's/^# \{0,1\}//' | head -25
+ exit 1
+fi
+
+LABEL="$1"
+SRC="$2"
+DORIS_HOME="$(cd "$(dirname "$0")/../.." && pwd)"
+BUILD_DIR="${3:-${DORIS_HOME}/be/build_Release_compile_bench}"
+OUT_DIR="${TU_BENCH_OUT:-${BUILD_DIR}/tu-bench}"
+TRACE="${TU_BENCH_TRACE:-ON}"
+
+[[ "${SRC}" = /* ]] || SRC="${DORIS_HOME}/${SRC}"
+CCJSON="${BUILD_DIR}/compile_commands.json"
+[[ -f "${CCJSON}" ]] || { echo "ERROR: ${CCJSON} not found (configure first)"
>&2; exit 1; }
+mkdir -p "${OUT_DIR}"
+
+# Extract the compile command, strip binary PCH, redirect -o. Token-level
+# rewrite in python (shell-splitting the escaped command string is too
fragile).
+OBJ="${OUT_DIR}/${LABEL}.o"
+CMDFILE="${OUT_DIR}/${LABEL}.cmd"
+python3 - "${CCJSON}" "${SRC}" "${OBJ}" > "${CMDFILE}" <<'PYEOF'
+import json, shlex, sys
+ccjson, src, obj = sys.argv[1], sys.argv[2], sys.argv[3]
+entries = [e for e in json.load(open(ccjson)) if e["file"] == src]
+if not entries:
+ sys.exit(f"ERROR: no compile_commands.json entry for {src}")
+e = entries[0]
+toks = shlex.split(e["command"])
+out = []
+i = 0
+while i < len(toks):
+ # binary PCH load: -Xclang -include-pch -Xclang <path>
+ if (toks[i] == "-Xclang" and i + 3 < len(toks)
+ and toks[i + 1] == "-include-pch" and toks[i + 2] == "-Xclang"):
+ i += 4
+ continue
+ if toks[i] == "-o" and i + 1 < len(toks):
+ out += ["-o", obj]
+ i += 2
+ continue
+ out.append(toks[i])
+ i += 1
+print(e["directory"])
+print(" ".join(shlex.quote(t) for t in out))
+PYEOF
+
+WORK_DIR="$(head -1 "${CMDFILE}")"
+CMD="$(tail -1 "${CMDFILE}")"
+
+TRACE_ARGS=""
+if [[ "${TRACE}" == "ON" ]]; then
+ TRACE_ARGS="-ftime-trace=${OUT_DIR}/${LABEL}.json
-ftime-trace-granularity=100"
+fi
+
+TIME_LOG="${OUT_DIR}/${LABEL}.time"
+if [[ "$(uname)" == "Darwin" ]]; then
+ # BSD time: "X.XX real X.XX user X.XX sys" plus "-l" resource lines (RSS
in bytes)
+ (cd "${WORK_DIR}" && /usr/bin/time -l bash -c "${CMD} ${TRACE_ARGS}" 2>
"${TIME_LOG}")
+ WALL_S="$(awk '/ real /{printf "%.2f", $1}' "${TIME_LOG}")"
+ MAX_RSS_KB="$(awk '/maximum resident set size/{printf "%d", $1/1024}'
"${TIME_LOG}")"
+else
+ (cd "${WORK_DIR}" && /usr/bin/time -v bash -c "${CMD} ${TRACE_ARGS}" 2>
"${TIME_LOG}")
+ WALL_RAW="$(grep 'Elapsed (wall clock)' "${TIME_LOG}" | awk '{print $NF}')"
+ # h:mm:ss or m:ss.ss -> seconds
+ WALL_S="$(echo "${WALL_RAW}" | awk -F: '{ s=0; for (i=1; i<=NF; i++) s =
s*60 + $i; printf "%.2f", s }')"
+ MAX_RSS_KB="$(grep 'Maximum resident' "${TIME_LOG}" | awk '{print $NF}')"
+fi
+# Use the toolchain's llvm-nm/llvm-size (sit next to the clang++ in the
command;
+# Apple's /usr/bin/size lacks -A).
+TOOLDIR="$(dirname "$(echo "${CMD}" | awk '{print $1}')")"
+NM="${TOOLDIR}/llvm-nm"
+[[ -x "${NM}" ]] || NM="nm"
+SIZE_TOOL="${TOOLDIR}/llvm-size"
+[[ -x "${SIZE_TOOL}" ]] || SIZE_TOOL="size"
+if [[ "$(uname)" == "Darwin" ]]; then
+ # Mach-O weak definitions don't get a distinct nm letter; use darwin
format.
+ WEAK="$(${NM} -m "${OBJ}" 2>/dev/null | grep -c 'weak external' || true)"
+else
+ WEAK="$(${NM} "${OBJ}" | grep -cE ' [VvWw] ' || true)"
+fi
+TEXT="$(${SIZE_TOOL} -A "${OBJ}" | awk '$1 ~ /^__text$|^\.text/{s+=$2}
END{print s+0}')"
+
+RESULTS="${OUT_DIR}/results.tsv"
+[[ -f "${RESULTS}" ]] || printf
'timestamp\tlabel\twall_s\tmax_rss_kb\tweak_defs\ttext_bytes\tsrc\n' >
"${RESULTS}"
+printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
+ "$(date +%Y-%m-%dT%H:%M:%S)" "${LABEL}" "${WALL_S}" "${MAX_RSS_KB}"
"${WEAK}" "${TEXT}" "${SRC}" \
+ | tee -a "${RESULTS}"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
