andygrove opened a new pull request, #6265: URL: https://github.com/apache/datafusion-comet/pull/6265
Backport of #6237, #6244 and #6248 to `branch-1.1`. Cherry-picked in merge order from `81f2574d5e92028d6a0c8aebd39e6e0b5b4c2fd2`, `defa2546d8e66b82541e8a451a28541b7a399637` and `bc4be39964cbe9cdb5f2a949740a8164e6b5755b`, without conflicts. The diffs of #6237 and #6244 are byte-identical to upstream. #6248's changed lines are identical too; only its hunk offsets in `CometConf.scala` and `Plugins.scala` differ, because #6155 is not on this branch. ## Which issue does this PR close? None. Part of #5327. ## Rationale for this change All three merged to `main` after `branch-1.1` was cut at 36ab57c68. The 1.1 user guide on the website is copied from this branch: `docs/generate-versions.py` clones `branch-1.1` and takes its `docs/source/user-guide/latest`. Without this backport the 1.1.0 docs would miss: - #6248: the 1.1.0 section of the upgrade guide. It lists the changes since 1.0.0 that need no legacy key but can still change what an existing deployment does: #6191, #6195, #6163 and #6066, all of which are on this branch. - #6237: the diagram of how Comet uses executor memory. - #6244: the tuning guide split into pages for memory, scans, operators, shuffle, Celeborn and transitions. #6248 also needs to land before the release docs are generated, because `dev/generate-release-docs.sh` renders the config reference from `CometConf` on this branch. ## What changes are included in this PR? The three original changes, so see each PR for the details. No adaptations were needed. Besides the docs, #6248 changes the deprecation text of `spark.comet.exec.memoryPool.fraction` and `spark.comet.exec.sortMergeJoinWithJoinFilter.enabled` from "will be removed in a future release" to "in a future major release", both in `CometConf` and in the driver plugin's warning for the fraction. That matches the versioning policy, which removes a covered key only in a major release. #6245, which fixes redirects in `docs/source/conf.py`, is left out. The website is built from `main` and takes only the user guide directory from this branch, so this branch's `conf.py` is not used. ## How are these changes tested? - `sphinx-build -b html` on `docs/source` gives the same 63 warnings on `branch-1.1` with and without this PR. All of them come from the generated pages that a build without Maven skips. The new `tuning/` pages and the diagram are in the output. - The only link into the tuning guide from code, `user-guide/latest/tuning.html` in `CometConf`, still resolves after the split. - `prettier --check` passes on the 15 changed Markdown files. - The Scala changes are two doc strings and one log message, and no changed line is over 100 characters. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
