NoahKusaba opened a new pull request, #2438: URL: https://github.com/apache/datafusion-ballista/pull/2438
I've noticed for a while that ballista docs were outdated, incorrect, with drifting duplication for a while. My hope is that if we clean this up we can attract more contributors / interest. This is a vibe cleanup where I first asked Opus 5 to "Scan all documentation and do a full cleanup/correctness fix" Than after confirming it was good, started drilling down into clean refactors to minimize duplication and remove stale docs. AI summary: Documentation had drifted from the code in several places, and a set of GitHub-style anchor links never resolved on the published site. Correctness: - AQE was documented as experimental and disabled by default in three places; `ballista.planner.adaptive.enabled` defaults to true. - Scheduler policy default is push-staged, documented as pull-staged; event-loop buffer default is 1000, documented as 10000. - `ballista-cli` has no `spark-compat` feature, but two guides told users to install and build with it. - Scheduler REST table was missing five endpoints and both health probes; the Prometheus list carried a `failed` metric that does not exist. - Every doc that started a second executor moved only `--bind-port`, which collides with the first executor's gRPC port. - CLI usage blocks were clap 3 output missing `--vcores`, `--color` and `--tui`; regenerated from the built binary. - extending-components built a runtime with `RuntimeConfig`, removed in DataFusion 55, and used helper names that had been renamed. - License badge pointed at datafusion-comet; adopter rows referenced image files that are not in the repo. Site rendering: - Set `myst_heading_anchors`, without which MyST emits no heading ids and all 14 `file.md#heading` cross-references 404. These work when reading the files on GitHub, which is why the breakage went unnoticed. - code-organization.md started its headings at H2. Structure: - Delete docs/developer/, which sat outside the Sphinx tree and so was never published, after porting its Multi-Partition Tasks section and the SchedulerGrpc method table into the contributors guide. Its diagram was dropped rather than ported: it labels arrows `ExecutePartition`, which is not an RPC in the current proto. - Pull the Rust examples from examples/examples/ with `literalinclude` instead of pasting them; all three copies had already drifted from the shipped source. - Replace the hand-maintained config tables in the tuning guide with generated regions, so the existing CI check covers them. - Give ballista-core, ballista-scheduler and ballista-executor real READMEs; each is a docs.rs landing page via include_str! and was four lines long. - Standardise on "vcores" over "task slots" and "concurrency levels". - Collapse cluster-start instructions from five copies to two canonical pages, which is the duplication that produced the port collision above. Verified with prettier, sphinx-build -W, cargo doc under -D warnings, cargo test --doc, and the config-docs drift check. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
