This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new a1364217df7c [SPARK-52562][INFRA] Automatically create the base of
release notes and push
a1364217df7c is described below
commit a1364217df7cdc5cfb9e763b33f0229fdf71df67
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Wed Jun 25 08:42:25 2025 +0900
[SPARK-52562][INFRA] Automatically create the base of release notes and push
### What changes were proposed in this pull request?
This PR proposes to add the automatic release note process. Here is what it
does:
1. Add download link to docs
Inserts the new release version link into `documentation.md`, keeping
versions sorted by recency.
2. Add download link to spark-website
Updates `js/downloads.js` with the new version's metadata for the
downloads page. Replaces existing entry if it's a patch; inserts new entry
otherwise. Uses different package lists for Spark 3 vs. Spark 4.
3. Generate news & release notes
Creates a news post and release notes file as below. Note that I skipped
the short link generation step here.
- For minor/major releases (x.y.0), describes new features
- For patch/maintenance releases (x.y.z, z > 0), mentions stability
fixes and encourages upgrades.
4. Build the Website
Runs Jekyll to generate updated HTML files for the website.
5. Update latest symlink (only for major/minor)
Updates the `site/docs/latest` symlink to point to the new version only
if it's a major or minor release (x.y.0), so maintenance releases don’t affect
the default documentation version.
If the release manager needs to have better release notes, they can create
a separate PR to update this.
### Why are the changes needed?
To make the release process easier.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
I manually tested them in my Mac for now, and checked that they are
compatible with Ubuntu. It has to be tested in the official release later again.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #51260 from HyukjinKwon/SPARK-52562.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit 82ab68045759602b1b5a4e8308915375de03be7f)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
dev/create-release/release-build.sh | 232 +++++++++++++++++++++++++++++++++++-
1 file changed, 231 insertions(+), 1 deletion(-)
diff --git a/dev/create-release/release-build.sh
b/dev/create-release/release-build.sh
index d3f8c509910e..21235e9a8847 100755
--- a/dev/create-release/release-build.sh
+++ b/dev/create-release/release-build.sh
@@ -142,9 +142,239 @@ if [[ "$1" == "finalize" ]]; then
git commit -m "Add docs for Apache Spark $RELEASE_VERSION"
git push origin HEAD:asf-site
cd ..
- rm -rf spark-website
echo "docs uploaded"
+ echo "Uploading release docs to spark-website"
+ cd spark-website
+
+ # TODO: Test it in the actual release
+ # 1. Add download link to documentation.md
+ python3 <<EOF
+import re
+
+release_version = "${RELEASE_VERSION}"
+newline = f' <li><a href="{{{{site.baseurl}}}}/docs/{release_version}/">Spark
{release_version}</a></li>'
+inserted = False
+
+def parse_version(v):
+ return [int(p) for p in v.strip().split(".")]
+
+def vercmp(v1, v2):
+ a = parse_version(v1)
+ b = parse_version(v2)
+ return (a > b) - (a < b)
+
+with open("documentation.md") as f:
+ lines = f.readlines()
+
+with open("documentation.md", "w") as f:
+ for line in lines:
+ match = re.search(r'docs/(\d+\.\d+\.\d+)/', line)
+ if not inserted and match:
+ existing_version = match.group(1)
+ if vercmp(release_version, existing_version) >= 0:
+ f.write(newline + "\n")
+ inserted = True
+ f.write(line)
+ if not inserted:
+ f.write(newline + "\n")
+EOF
+
+ echo "Edited documentation.md"
+
+ # 2. Add download link to js/downloads.js
+ RELEASE_DATE=$(TZ=America/Los_Angeles date +"%m/%d/%Y")
+ IFS='.' read -r rel_maj rel_min rel_patch <<< "$RELEASE_VERSION"
+ NEW_PACKAGES="packagesV14"
+ if [[ "$rel_maj" -ge 4 ]]; then
+ NEW_PACKAGES="packagesV15"
+ fi
+
+ python3 <<EOF
+import re
+
+release_version = "${RELEASE_VERSION}"
+release_date = "${RELEASE_DATE}"
+new_packages = "${NEW_PACKAGES}"
+newline = f'addRelease("{release_version}", new Date("{release_date}"),
{new_packages}, true);'
+
+new_major, new_minor, new_patch = [int(p) for p in release_version.split(".")]
+
+def parse_version(v):
+ return [int(p) for p in v.strip().split(".")]
+
+def vercmp(v1, v2):
+ a = parse_version(v1)
+ b = parse_version(v2)
+ return (a > b) - (a < b)
+
+inserted = replaced = False
+
+with open("js/downloads.js") as f:
+ lines = f.readlines()
+
+with open("js/downloads.js", "w") as f:
+ for line in lines:
+ m = re.search(r'addRelease\("(\d+\.\d+\.\d+)"', line)
+ if m:
+ existing_version = m.group(1)
+ cmp_result = vercmp(release_version, existing_version)
+ ex_major, ex_minor, ex_patch = parse_version(existing_version)
+
+ if cmp_result == 0:
+ f.write(newline + "\n")
+ replaced = True
+ elif not replaced and ex_major == new_major and ex_minor ==
new_minor:
+ f.write(newline + "\n")
+ replaced = True
+ elif not replaced and not inserted and cmp_result > 0:
+ f.write(newline + "\n")
+ f.write(line)
+ inserted = True
+ else:
+ f.write(line)
+ else:
+ f.write(line)
+ if not replaced and not inserted:
+ f.write(newline + "\n")
+EOF
+
+ echo "Edited js/downloads.js"
+
+ # 3. Add news post
+ RELEASE_DATE=$(TZ=America/Los_Angeles date +"%Y-%m-%d")
+
FILENAME="news/_posts/${RELEASE_DATE}-spark-${RELEASE_VERSION//./-}-released.md"
+ mkdir -p news/_posts
+ cat > "$FILENAME" <<EOF
+---
+layout: post
+title: Spark ${RELEASE_VERSION} released
+categories:
+- News
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+ _wpas_done_all: '1'
+---
+We are happy to announce the availability of <a
href="{{site.baseurl}}/releases/spark-release-${RELEASE_VERSION}.html"
title="Spark Release ${RELEASE_VERSION}">Apache Spark ${RELEASE_VERSION}</a>!
Visit the <a
href="{{site.baseurl}}/releases/spark-release-${RELEASE_VERSION}.html"
title="Spark Release ${RELEASE_VERSION}">release notes</a> to read about the
new features, or <a href="{{site.baseurl}}/downloads.html">download</a> the
release today.
+EOF
+
+ echo "Created $FILENAME"
+
+ # 4. Add release notes with Python to extract JIRA version ID
+ RELEASE_DATE=$(TZ=America/Los_Angeles date +"%Y-%m-%d")
+ JIRA_PROJECT_ID=12315420
+ JIRA_URL="https://issues.apache.org/jira/rest/api/2/project/SPARK/versions";
+ JSON=$(curl -s "$JIRA_URL")
+
+ VERSION_ID=$(python3 - <<EOF
+import sys, json
+
+release_version = "${RELEASE_VERSION}"
+json_str = """$JSON"""
+
+try:
+ versions = json.loads(json_str)
+except Exception as e:
+ print(f"Error parsing JSON: {e}", file=sys.stderr)
+ sys.exit(1)
+
+version_id = ""
+for v in versions:
+ if v.get("name") == release_version:
+ version_id = v.get("id", "")
+ break
+
+print(version_id)
+EOF
+ )
+
+ if [[ -z "$VERSION_ID" ]]; then
+ echo "Error: Couldn't find JIRA version ID for $RELEASE_VERSION" >&2
+ fi
+
+
JIRA_LINK="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=${JIRA_PROJECT_ID}&version=${VERSION_ID}";
+
+ IFS='.' read -r rel_maj rel_min rel_patch <<< "$RELEASE_VERSION"
+ if [[ "$rel_patch" -eq 0 ]]; then
+ ACKNOWLEDGE="patches and features to this release."
+ BODY="Apache Spark ${RELEASE_VERSION} is a new feature release. It
introduces new functionality and improvements. We encourage users to try it and
provide feedback."
+ else
+ ACKNOWLEDGE="patches to this release."
+ BODY="Apache Spark ${RELEASE_VERSION} is a maintenance release containing
security and correctness fixes. This release is based on the
branch-${rel_maj}.${rel_min} maintenance branch of Spark. We strongly recommend
all ${rel_maj}.${rel_min} users to upgrade to this stable release."
+ fi
+
+ BODY+="
+
+You can find the list of resolved issues and detailed changes in the [JIRA
release notes](${JIRA_LINK}).
+
+We would like to acknowledge all community members for contributing
${ACKNOWLEDGE}"
+
+
FILENAME="releases/_posts/${RELEASE_DATE}-spark-release-${RELEASE_VERSION}.md"
+ mkdir -p releases/_posts
+ cat > "$FILENAME" <<EOF
+---
+layout: post
+title: Spark Release ${RELEASE_VERSION}
+categories: []
+tags: []
+status: publish
+type: post
+published: true
+meta:
+ _edit_last: '4'
+ _wpas_done_all: '1'
+---
+
+${BODY}
+EOF
+
+ echo "Created $FILENAME"
+
+ # 5. Build the website
+ bundle install
+ bundle exec jekyll build
+
+ # 6. Update latest symlink if minor/major release
+ LINK_PATH="site/docs/latest"
+ TARGET_DIR="site/docs/$RELEASE_VERSION"
+ IFS='.' read -r rel_maj rel_min rel_patch <<< "$RELEASE_VERSION"
+ if [[ "$rel_patch" -eq 0 ]]; then
+ if [[ -L "$LINK_PATH" ]]; then
+ CURRENT_TARGET=$(readlink "$LINK_PATH")
+ else
+ CURRENT_TARGET=""
+ fi
+
+ if [[ "$CURRENT_TARGET" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
+ IFS='.' read -r cur_maj cur_min cur_patch <<< "$CURRENT_TARGET"
+
+ if [[ "$rel_maj" -gt "$cur_maj" ]]; then
+ ln -sfn "$RELEASE_VERSION" "$LINK_PATH"
+ echo "Updated symlink $LINK_PATH -> $RELEASE_VERSION (major version
increased)"
+ elif [[ "$rel_maj" -eq "$cur_maj" && "$rel_min" -gt "$cur_min" ]]; then
+ ln -sfn "$RELEASE_VERSION" "$LINK_PATH"
+ echo "Updated symlink $LINK_PATH -> $RELEASE_VERSION (minor version
increased)"
+ else
+ echo "Symlink $LINK_PATH points to $CURRENT_TARGET with equal or newer
major.minor, no change"
+ fi
+ else
+ echo "No valid existing version target."
+ fi
+ else
+ echo "Patch release detected ($RELEASE_VERSION), not updating symlink"
+ fi
+
+ git add .
+ git commit -m "Add release docs for Apache Spark $RELEASE_VERSION"
+ git push origin HEAD:asf-site
+ cd ..
+ echo "release docs uploaded"
+ rm -rf spark-website
+
# Moves the docs from dev directory to release directory.
echo "Moving Spark docs to the release directory"
svn mv --username "$ASF_USERNAME" --password "$ASF_PASSWORD" -m"Apache Spark
$RELEASE_VERSION" \
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
