Fokko commented on code in PR #8919: URL: https://github.com/apache/iceberg/pull/8919#discussion_r1386152391
########## site/dev/common.sh: ########## @@ -0,0 +1,125 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +set -e + +mv_nightly_up () { + mv docs/docs/nightly/ docs/ +} + +mv_nightly_down () { + mv docs/nightly/ docs/docs/ +} + +uninstall_deps () { + if test -f .env; then + deactivate + rm -r .env + fi +} + +install_deps () { + uninstall_deps + python -m venv .env Review Comment: This already assumes that you have `virtualenv` installed. Do we actually need a `venv` for this simple setup? We could also use the system `python3` ########## site/README.md: ########## @@ -54,18 +54,34 @@ The non-versioned site pages are all the `/site/docs/.*md` files and the docs ar │ ├── about.md │ ├── ... │ └── view-spec.md -├── README.md +├── ... +├── Makefile ├── mkdocs.yml (non-versioned) -├── requirements.txt -└── variables.yml +└── requirements.txt ``` ### Building the versioned docs -> [!IMPORTANT] -> This build process is currently missing older versions and the javadoc branches. -> Until these branches are merged, these steps will not work. +The Iceberg versioned docs are committed in the orphan `docs` branch and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The `docs` branch contains the versioned documenation source files at the root. These versions exist at the `/site/docs/docs/<version>` directory once mounted. The `latest` version, is a soft link to the most recent build version in the `docs` branch. There is also a `javadoc` branch that contains all prior static generation versions of the javadocs that is mounted at `/site/docs/javadoc/<version>`. -All previously versioned docs will be committed in `docs-<version>` branches and mounted using [git worktree](https://git-scm.com/docs/git-worktree) at build time. The worktree will pull these versions in following the `/site/docs/docs/<version>` convention. The `latest` version, will be a secondary copy of the most recent build version in the worktree, but pointing to `/site/docs/docs/latest`. There is also a `javadoc` branch that contains all prior static generation versions of the javadocs in a single tag. +The docs are built, run, and released using [make](https://www.gnu.org/software/make/manual/make.html). The [Makefile](Makefile) and the [common shell script](dev/common.sh) support the following command: + +``` +site > make help +[build](dev/build.sh): Clean and build the site locally. +[clean](dev/clean.sh): Clean the local site. +[deploy](dev/deploy.sh): Clean, build, and deploy the Iceberg docs site. +help: Show help for each of the Makefile recipes. +[release](dev/release.sh): Release the current nightly docs as ICEBERG_VERSION (make release ICEBERG_VERSION=<MAJOR.MINOR.PATCH>). +[serve](dev/serve.sh): Clean, build, and run the site locally. +``` Review Comment: The links don't render inside of the code block:  ```suggestion - [build](dev/build.sh): Clean and build the site locally. - [clean](dev/clean.sh): Clean the local site. - [deploy](dev/deploy.sh): Clean, build, and deploy the Iceberg docs site. - [release](dev/release.sh): Release the current nightly docs as ICEBERG_VERSION (make release ICEBERG_VERSION=<MAJOR.MINOR.PATCH>). - [serve](dev/serve.sh): Clean, build, and run the site locally. ``` ########## site/dev/common.sh: ########## @@ -0,0 +1,125 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +set -e + +mv_nightly_up () { + mv docs/docs/nightly/ docs/ +} + +mv_nightly_down () { + mv docs/nightly/ docs/docs/ +} + +uninstall_deps () { + if test -f .env; then + deactivate + rm -r .env + fi +} + +install_deps () { + uninstall_deps + python -m venv .env + source env/bin/activate + pip install -r requirements.txt +} + +assert_not_empty () { + if [ -z "$1" ] + then + echo "No argument supplied" + exit 1 + fi +} + +get_latest_version () { + basename $(ls -d docs/docs/*/ | sort -V | tail -1) +} + +update_version () { + local ICEBERG_VERSION="$1" + assert_not_empty "${ICEBERG_VERSION}" + + sed -i '' -E "s/(^site\_name:[[:space:]]+docs\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml + sed -i '' -E "s/(^[[:space:]]*-[[:space:]]+Javadoc:.*\/javadoc\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml + + sed -i '' -E "s/(^ \- latest: '\!include docs\/docs\/latest\/mkdocs\.yml'/a \- ${ICEBERG_VERSION}: '\!include docs\/docs\/${ICEBERG_VERSION}\/mkdocs\.yml/" ../../mkdocs.yml +} + +create_latest () { + local ICEBERG_VERSION="$1" + assert_not_empty "${ICEBERG_VERSION}" + + rm -rf docs/docs/latest/ + mkdir docs/docs/latest/ + + ln -s "../${ICEBERG_VERSION}/docs" docs/docs/latest/docs + cp "docs/docs/${ICEBERG_VERSION}/mkdocs.yml" docs/docs/latest/ + + cd docs/docs/ + update_version "latest" + cd - +} + + +# https://squidfunk.github.io/mkdocs-material/setup/setting-up-site-search/#search-exclusion +search_exclude_versioned_docs () { + local ICEBERG_VERSION="$1" + assert_not_empty "${ICEBERG_VERSION}" + + cd "${ICEBERG_VERSION}/docs/" + + python3 -c "import os Review Comment: We both use python and python3, we probably want to consolidate this ########## site/dev/common.sh: ########## @@ -0,0 +1,125 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +set -e + +mv_nightly_up () { + mv docs/docs/nightly/ docs/ +} + +mv_nightly_down () { + mv docs/nightly/ docs/docs/ +} + +uninstall_deps () { + if test -d site_env; then + deactivate + rm -r site_env + fi +} + +install_deps () { + uninstall_deps + python -m venv site_env + source site_env/bin/activate + pip install -r requirements.txt +} + +assert_not_empty () { + if [ -z "$1" ] + then + echo "No argument supplied" + exit 1 + fi +} + +get_latest_version () { + basename $(ls -d docs/docs/*/ | sort -V | tail -1) +} + +update_version () { + local ICEBERG_VERSION="$1" + assert_not_empty "${ICEBERG_VERSION}" + + sed -i '' -E "s/(^site\_name:[[:space:]]+docs\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml + sed -i '' -E "s/(^[[:space:]]*-[[:space:]]+Javadoc:.*\/javadoc\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml + + sed -i '' -E "s/ \- latest: '\!include docs\/docs\/latest\/mkdocs\.yml'/a \- ${ICEBERG_VERSION}: '\!include docs\/docs\/${ICEBERG_VERSION}\/mkdocs\.yml/" ../../mkdocs.yml +} + +create_latest () { + local ICEBERG_VERSION="$1" + assert_not_empty "${ICEBERG_VERSION}" + + rm -rf docs/docs/latest/ + mkdir docs/docs/latest/ + + ln -s "../${ICEBERG_VERSION}/docs" docs/docs/latest/docs + cp "docs/docs/${ICEBERG_VERSION}/mkdocs.yml" docs/docs/latest/ + + cd docs/docs/ + update_version "latest" + cd - +} + + +# https://squidfunk.github.io/mkdocs-material/setup/setting-up-site-search/#search-exclusion +search_exclude_versioned_docs () { + local ICEBERG_VERSION="$1" + assert_not_empty "${ICEBERG_VERSION}" + + cd "${ICEBERG_VERSION}/docs/" + + python3 -c "import os +for f in filter(lambda x: x.endswith('.md'), os.listdir()): lines = open(f).readlines(); open(f, 'w').writelines(lines[:2] + ['search:\n', ' exclude: true\n'] + lines[2:]);" + + cd - +} + +pull_versioned_docs () { + mv_nightly_up + + git worktree add docs/docs docs + git worktree add docs/javadoc javadoc Review Comment: My `make serve` gets stuck on this one: ``` + git worktree add docs/docs docs Preparing worktree (checking out 'docs') HEAD is now at 87d2dfbd6 Shift site build to use monorepo and gh-pages + git worktree add docs/javadoc javadoc fatal: invalid reference: javadoc ``` ########## site/dev/common.sh: ########## @@ -0,0 +1,107 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +set -e + +mv_nightly_up () { + mv docs/docs/nightly/ docs/ +} + +mv_nightly_down () { + mv docs/nightly/ docs/docs/ +} + +assert_not_empty () { + if [ -z "$1" ] + then + echo "No argument supplied" + exit 1 + fi +} + +get_latest_version () { + basename $(ls -d docs/docs/*/ | sort -V | tail -1) +} + +update_version () { + local ICEBERG_VERSION="$1" + assert_not_empty "${ICEBERG_VERSION}" + + sed -i '' -E "s/(^site\_name:[[:space:]]+docs\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml + sed -i '' -E "s/(^[[:space:]]*-[[:space:]]+Javadoc:.*\/javadoc\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml +} + +create_latest () { + local ICEBERG_VERSION="$1" + assert_not_empty "${ICEBERG_VERSION}" + + rm -rf docs/docs/latest/ + mkdir docs/docs/latest/ + + ln -s "../${ICEBERG_VERSION}/docs" docs/docs/latest/docs + cp "docs/docs/${ICEBERG_VERSION}/mkdocs.yml" docs/docs/latest/ + + cd docs/docs/ + update_version "latest" + cd - +} + + +# https://squidfunk.github.io/mkdocs-material/setup/setting-up-site-search/#search-exclusion +search_exclude_versioned_docs () { + local ICEBERG_VERSION="$1" + assert_not_empty "${ICEBERG_VERSION}" + + cd "${ICEBERG_VERSION}/docs/" + + python3 -c "import os +for f in filter(lambda x: x.endswith('.md'), os.listdir()): lines = open(f).readlines(); open(f, 'w').writelines(lines[:2] + ['search:\n', ' exclude: true\n'] + lines[2:]);" Review Comment: Nice! 😁 ########## site/Makefile: ########## @@ -0,0 +1,38 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +.PHONY: help +help: # Show help for each of the Makefile recipes. + @grep -E '^[a-zA-Z0-9 -]+:.*#' Makefile | sort | while read -r l; do printf "\033[1;32m$$(echo $$l | cut -f 1 -d':')\033[00m:$$(echo $$l | cut -f 2- -d'#')\n"; done + +.PHONY: serve +serve: # Clean, build, and run the docs site locally. + dev/serve.sh + +.PHONY: build +build: # Clean and build the docs site locally. Review Comment: In Makefile you can also invoke other commands, for example here: https://github.com/apache/iceberg-python/blob/main/Makefile#L24 This way we can pull logic from the scripts themselves, into the Makefile (and therefore we don't need this kind of comments). ########## .github/workflows/site-ci.yml: ########## @@ -0,0 +1,21 @@ +name: ci Review Comment: ```suggestion # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # name: ci ``` ########## site/README.md: ########## @@ -83,59 +99,27 @@ All previously versioned docs will be committed in `docs-<version>` branches and └── ... ``` -### Install - -1. (Optional) Set up venv +To run this, run the `serve` recipe, which runs the `build` recipe and calls `mkdocs serve`. This will run locally at <http://localhost:8000>. ``` -python -m venv mkdocs_env -source mkdocs_env/bin/activate +make serve ``` -1. Install required Python libraries +To clear all build files, run `clean`. ``` -pip install -r requirements.txt -``` - -#### Adding additional versioned documentation - -To build locally with additional docs versions, add them to your working tree. -For now, I'm just adding a single version, and the javadocs directory. - -``` -git worktree add site/docs/docs/1.4.0 docs-1.4.0 -git worktree add site/docs/javadoc javadoc -``` - -## Build - -Run the build command in the root directory, and optionally add `--clean` to force MkDocs to clear previously generated pages. - -``` -mkdocs build [--clean] -``` - -## Run - -Start MkDocs server locally to verify the site looks good. - -``` -mkdocs serve +make clean ``` ## Release process -Deploying a version of the docs is a two step process: - 1. ~~Cut a new release from the current branch revision. This creates a new branch `docs-<version>`.~~ - +Deploying the docs is a two step process: + 1. Release a new version by copying the current nightly directory to a new version directory in the `docs` branch. ``` - .github/bin/deploy_docs.sh -v 1.4.0 + make release ICEBERG_VERSION=${ICEBERG_VERSION} + ``` + 1. Push the generated site to `asf-site`. + ``` + make deploy Review Comment: Will this be done by a person? We could also do this in the CI, as we do with PyIceberg: https://github.com/apache/iceberg-python/blob/main/.github/workflows/python-ci-docs.yml ########## site/.gitignore: ########## @@ -1,13 +1,3 @@ -## Temp remove for first phase Review Comment: Do we need this one at all? We can just rely on the main `.gitignore` ########## site/dev/common.sh: ########## @@ -0,0 +1,125 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +set -e + +mv_nightly_up () { + mv docs/docs/nightly/ docs/ +} + +mv_nightly_down () { + mv docs/nightly/ docs/docs/ +} + +uninstall_deps () { + if test -f .env; then + deactivate + rm -r .env + fi +} + +install_deps () { + uninstall_deps + python -m venv .env + source env/bin/activate Review Comment: ```suggestion source .env/bin/activate ``` ########## site/dev/common.sh: ########## @@ -0,0 +1,107 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +set -e + +mv_nightly_up () { + mv docs/docs/nightly/ docs/ +} + +mv_nightly_down () { + mv docs/nightly/ docs/docs/ +} + +assert_not_empty () { + if [ -z "$1" ] + then + echo "No argument supplied" + exit 1 + fi +} + +get_latest_version () { + basename $(ls -d docs/docs/*/ | sort -V | tail -1) +} + +update_version () { + local ICEBERG_VERSION="$1" + assert_not_empty "${ICEBERG_VERSION}" + + sed -i '' -E "s/(^site\_name:[[:space:]]+docs\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml + sed -i '' -E "s/(^[[:space:]]*-[[:space:]]+Javadoc:.*\/javadoc\/).*$/\1${ICEBERG_VERSION}/" ${ICEBERG_VERSION}/mkdocs.yml +} + +create_latest () { + local ICEBERG_VERSION="$1" + assert_not_empty "${ICEBERG_VERSION}" + + rm -rf docs/docs/latest/ + mkdir docs/docs/latest/ + + ln -s "../${ICEBERG_VERSION}/docs" docs/docs/latest/docs + cp "docs/docs/${ICEBERG_VERSION}/mkdocs.yml" docs/docs/latest/ + + cd docs/docs/ + update_version "latest" + cd - +} + + +# https://squidfunk.github.io/mkdocs-material/setup/setting-up-site-search/#search-exclusion +search_exclude_versioned_docs () { + local ICEBERG_VERSION="$1" + assert_not_empty "${ICEBERG_VERSION}" + + cd "${ICEBERG_VERSION}/docs/" + + python3 -c "import os +for f in filter(lambda x: x.endswith('.md'), os.listdir()): lines = open(f).readlines(); open(f, 'w').writelines(lines[:2] + ['search:\n', ' exclude: true\n'] + lines[2:]);" Review Comment: Nice! 😁 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
