Dandandan opened a new pull request, #24339:
URL: https://github.com/apache/datafusion/pull/24339
## Which issue does this PR close?
None. Found while profiling compile times for #24325 / #24326 / #24329 /
#24330,
which removed the trait-solving cost from the four crates on the critical
path
and left LLVM as the dominant remaining cost.
## Rationale for this change
`dev` is the profile behind every `cargo build` and `cargo test`, so its
debug
info is generated over and over. `debug = "line-tables-only"` keeps file and
line
numbers — panics and `RUST_BACKTRACE` output stay just as useful — and drops
the
variable-level DWARF that only an interactive debugger consumes.
Measured per crate, **interleaved** with the baseline so machine drift
cancels
out. The flag is passed to the crate under test only, so cached dependency
artifacts stay valid and nothing else moves between the two measurements:
| crate | `debug = 2` | `line-tables-only` | |
|---|---|---|---|
| `datafusion-physical-plan` | 8.89s | 7.43s | −16% |
| `datafusion-functions-aggregate` | 5.86s | 4.63s | −21% |
| `datafusion-physical-expr` | 4.57s | 3.59s | −21% |
| `datafusion-functions` | 4.64s | 4.04s | −13% |
| `datafusion-expr` | 4.54s | 3.57s | −21% |
| `datafusion-functions-nested` | 4.37s | 3.06s | −30% |
| `datafusion-optimizer` | 3.91s | 3.12s | −20% |
| `datafusion-common` | 3.73s | 3.10s | −17% |
| `datafusion-sql` | 3.57s | 2.43s | −32% |
| `datafusion-datasource-parquet` | 3.27s | 2.44s | −25% |
| `datafusion-datasource` | 1.90s | 1.42s | −25% |
| `datafusion-physical-optimizer` | 1.22s | 0.99s | −19% |
| **sum** | **50.5s** | **39.8s** | **−21%** |
The saving is codegen-side, as you would expect: `datafusion-catalog`, which
spends its time in the trait solver rather than in LLVM, moves only 7.4s →
7.0s.
Artifacts shrink as well — `libdatafusion_physical_plan.rlib` goes from
**141MB
to 100MB**.
## What changes are included in this PR?
One setting on `[profile.dev]`, plus an update to the profile documentation
block
above it, which currently advertises "full debug info" for `dev`.
## Are these changes tested?
`cargo build -p datafusion` and `cargo test -p datafusion-catalog` pass
under the
new setting, and I checked the property that actually matters — that
backtraces
still name a file and line:
```
thread 'main' panicked at
datafusion/core/src/bin/print_functions_docs.rs:38:9
at ./datafusion/core/src/bin/print_functions_docs.rs:38:9
```
## Are there any user-facing changes?
For anyone stepping through DataFusion in a debugger, local variable
inspection
needs `CARGO_PROFILE_DEV_DEBUG=2 cargo build` (or a local override in
`.cargo/config.toml`); the comment in `Cargo.toml` says so. Everything else —
panic locations, backtraces, `#[test]` failures — is unchanged.
Two caveats I want to be explicit about:
- The 21% above is compile *work*, measured per crate. The effect on a full
parallel build's wall clock is smaller, because the critical path through
the
workspace is dominated by frontend time rather than codegen. I do not have
a
trustworthy end-to-end wall-clock number to quote: my full-build runs
varied by
2× between repetitions of the same configuration, so I am only reporting
the
measurements I could interleave.
- This is a policy choice as much as a measurement, so if the project would
rather keep full debug info by default, an alternative with most of the
disk
benefit and none of the debugger cost is to apply it to dependencies only:
`[profile.dev.package."*"] debug = "line-tables-only"`. Happy to switch to
that.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
