andygrove opened a new pull request, #6244:
URL: https://github.com/apache/datafusion-comet/pull/6244
## Which issue does this PR close?
No issue; this is documentation only.
## Rationale for this change
The tuning guide has grown to about 600 lines on a single page, covering
memory, shuffle, Celeborn, scans, joins, aggregation, sorting and row/columnar
transitions. Readers usually come for one of those topics and have to scroll
past the rest to find it. This gives tuning its own section in the user guide
with a page per topic, the same way the compatibility guide was split in #4055.
The overview stays at `user-guide/latest/tuning.html` and lists the other
pages. That is the URL in `CometConf.TUNING_GUIDE`, which the executor memory
warnings and several config descriptions link to, so those links keep working,
including from released versions, without a redirect or a code change.
## What changes are included in this PR?
- `tuning.md` becomes the overview: a short intro, a list of the topic
pages, and the three short sections that apply across topics (Tokio runtime,
metrics overhead and explain plan).
- New pages under `tuning/`:
- `memory.md`: the off-heap memory pool, the executor memory overhead and
sizing it from the memory usage log, batch size, and the spill disk limit.
- `shuffle.md`: enabling Comet shuffle, native and columnar shuffle, the
automatic revert to Spark shuffle, and compression.
- `celeborn.md`: remote shuffle with Celeborn. This was about half of the
shuffle section and only matters to Celeborn users, so the shuffle page now
links to it instead.
- `scans.md`: Parquet filter pushdown and split sizing, and Iceberg data
file concurrency.
- `operators.md`: joins and join runtime filters, adaptive partial
aggregation, and sorting on floating-point values.
- `transitions.md`: reducing row/columnar conversion overhead.
- `index.rst`: a new Tuning caption in the sidebar, after Operating Comet,
listing the pages.
- Links into the tuning guide from seven other user guide and contributor
guide pages now point to the page and anchor that holds the content.
The text is moved as-is. Apart from the overview's intro and page list,
three new page titles and the pointer from the shuffle page to the Celeborn
page, the only edits are relative link and image path fixes and dropping two
unused link definitions, one of which pointed at an `#advanced-memory-tuning`
section that doesn't exist.
Old deep links such as `tuning.html#shuffle` now land at the top of the
overview rather than on the section, because the section has moved to its own
page. Links to the three sections that stayed on the overview still work.
## How are these changes tested?
Documentation only, no code paths touched.
I built the site locally with Sphinx before and after the change, and the
change adds no warnings. MyST warns about any link to a missing page or heading
anchor, so every internal link and anchor resolves. In the built HTML, all 21
links into the tuning pages reach an existing anchor, the diagram loads on the
new memory page, and the sidebar shows the Tuning section with the current page
highlighted. A script also checked that every line of the old page appears
exactly once in the new pages, apart from the edits listed above. `npx
prettier@latest --check 'docs/**/*.md'` passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
