This is an automated email from the ASF dual-hosted git repository.
SteNicholas pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/paimon-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new c961ff2 fix(release): make source archives reproducible (#177)
c961ff2 is described below
commit c961ff222129e55e085c353b652632b112fb1d87
Author: Zhang Jiawei <[email protected]>
AuthorDate: Tue Aug 4 15:13:15 2026 +0800
fix(release): make source archives reproducible (#177)
---
.github/workflows/release_candidate.yaml | 4 +
scripts/releasing/README.md | 53 +++--
scripts/releasing/create_source_release.sh | 40 +++-
scripts/releasing/release_rc.sh | 126 ++++++++++--
scripts/releasing/tests/test_release_tools.py | 282 ++++++++++++++++++++++++++
scripts/releasing/verify_release_candidate.sh | 45 ++--
6 files changed, 495 insertions(+), 55 deletions(-)
diff --git a/.github/workflows/release_candidate.yaml
b/.github/workflows/release_candidate.yaml
index 28a4b09..859b257 100644
--- a/.github/workflows/release_candidate.yaml
+++ b/.github/workflows/release_candidate.yaml
@@ -78,9 +78,13 @@ jobs:
--output-dir release/ci
- name: Audit source archive
+ env:
+ # The creator must ignore caller-provided gzip defaults.
+ GZIP: "-9"
run: |
scripts/releasing/verify_release_candidate.sh \
--allow-unsigned \
+ --git-ref HEAD \
--skip-build \
"release/ci/apache-paimon-cpp-${RELEASE_VERSION}-src.tgz"
diff --git a/scripts/releasing/README.md b/scripts/releasing/README.md
index ee558c1..a251bed 100644
--- a/scripts/releasing/README.md
+++ b/scripts/releasing/README.md
@@ -34,9 +34,9 @@ Before starting a release:
- obtain an ASF code-signing key, publish it through the ASF account system,
and make sure it is present in
[Paimon KEYS](https://downloads.apache.org/paimon/KEYS);
-- install `git`, `gpg`, `svn`, `gh`, `python3`, `curl` or `wget`, Java, CMake,
- Ninja, and the toolchain needed by `ci/scripts/build_paimon.sh` (Java is
- required by Apache RAT);
+- install `git`, GNU `gzip`, `gpg`, `svn`, `gh`, `python3`, `curl` or `wget`,
+ Java, CMake, Ninja, and the toolchain needed by
+ `ci/scripts/build_paimon.sh` (Java is required by Apache RAT);
- authenticate `gh` with access to read GitHub Actions runs in
`apache/paimon-cpp`;
- make sure the Apache Git remote points directly to
@@ -44,6 +44,15 @@ Before starting a release:
- prepare and merge a release-preparation PR that updates the release notes and
all version metadata, and passes the normal and release-candidate workflows.
+The source archive uses GNU gzip with fixed options so that macOS and Linux
+produce the same bytes. On macOS, install Homebrew gzip and either put it first
+on `PATH` or select it explicitly:
+
+```bash
+brew install gzip
+export PAIMON_GZIP="$(brew --prefix gzip)/bin/gzip"
+```
+
For example, update all version locations and review the diff:
```bash
@@ -80,10 +89,13 @@ The release scripts use `vVERSION-rcRC` for
release-candidate tags and
Start from the exact clean commit approved for the candidate. Before
publishing,
the wrapper fetches the release branch and requires `HEAD` to be contained in
-its current history. It then creates and verifies a signed RC tag, creates the
-source archive and its checksum/signature, performs the full source-release
-verification, pushes the tag, waits for the tag-triggered release-candidate
-workflow to succeed, and imports the artifacts into ASF `dist/dev`:
+its current history. It then creates and verifies a signed RC tag, pushes the
+tag, and waits for the tag-triggered release-candidate workflow. That workflow
+creates the canonical source archive and checksum, builds and tests the same
+archive with GCC and Clang, and uploads it as a workflow artifact. The wrapper
+downloads those exact bytes, confirms that they are reproducible from the tag,
+signs the archive locally, performs the full source-release verification, and
+imports the three files into ASF `dist/dev`:
```bash
scripts/releasing/release_rc.sh \
@@ -96,12 +108,13 @@ scripts/releasing/release_rc.sh \
The release branch defaults to `main`; use `--release-branch NAME` for a
maintenance release from another Apache branch.
-Use `--prepare-only` to create and verify artifacts without pushing the tag or
-uploading to ASF infrastructure. This local-only mode does not require `HEAD`
-to match the remote release branch. Use `--dry-run` to print identifiers
-without making changes. A resumed run reuses an existing local tag or complete
-artifact set only after validating it. A prepare-only run does not print a vote
-email and must not be used to start a vote.
+Use `--prepare-only` to create and verify preview artifacts without pushing the
+tag or uploading to ASF infrastructure. This local-only mode does not require
+`HEAD` to match the remote release branch. A preview is not authoritative: a
+published run downloads the workflow artifact and rejects an existing local
+archive if its bytes differ. Use `--dry-run` to print identifiers without
+making changes. A prepare-only run does not print a vote email and must not be
+used to start a vote.
The candidate directory contains:
@@ -164,13 +177,16 @@ The verifier checks:
- installation plus compilation and execution of an external CMake consumer.
Pass `--git-ref v0.3.0-rc1` when the Git repository is available to regenerate
-the archive from the signed tag and compare it byte-for-byte.
+the archive from the signed tag with GNU gzip and compare it byte-for-byte.
+This check requires GNU gzip on every platform; it intentionally rejects the
+macOS system gzip instead of treating different compressed bytes as equivalent.
`--allow-unsigned`, `--skip-rat`, `--skip-build`, and `--skip-install` exist
for
CI or local development of the release process. They are not a substitute for
the corresponding checks when voting. The release-candidate workflow creates
-an unsigned archive for deterministic CI validation; official artifacts must
-always be signed by the release manager.
+the unsigned canonical archive for deterministic CI validation. The release
+manager downloads and signs that exact archive; the private signing key remains
+only on the release manager's machine.
## Publish an approved release
@@ -207,8 +223,9 @@ than ASF's general one-hour minimum.
- `bump_version.py`: consistently check or update CMake and documentation
version metadata.
-- `create_source_release.sh`: deterministically create an archive, SHA-512
- checksum, and optional detached signature from an immutable Git ref.
+- `create_source_release.sh`: deterministically create an archive with GNU
+ gzip, a SHA-512 checksum, and an optional detached signature from an
immutable
+ Git ref.
- `validate_source_archive.py`: reject unsafe or non-portable tar members and
compiled files.
- `verify_release_candidate.sh`: perform voter-facing integrity, license,
diff --git a/scripts/releasing/create_source_release.sh
b/scripts/releasing/create_source_release.sh
index 125ab57..67e4d09 100755
--- a/scripts/releasing/create_source_release.sh
+++ b/scripts/releasing/create_source_release.sh
@@ -49,6 +49,9 @@ The script creates:
apache-paimon-cpp-VERSION-src.tgz.asc (when --signing-key is provided)
Existing artifacts are never overwritten.
+
+GNU gzip is required so macOS and Linux produce the same compressed bytes.
+Set PAIMON_GZIP to an explicit GNU gzip executable when it is not on PATH.
EOF
}
@@ -68,6 +71,36 @@ calculate_sha512() {
fi
}
+find_gnu_gzip() {
+ local candidate
+ local resolved
+ local version_line
+ local -a candidates
+
+ if [[ -n "${PAIMON_GZIP:-}" ]]; then
+ candidates=("${PAIMON_GZIP}")
+ else
+ candidates=(gzip ggzip)
+ fi
+
+ for candidate in "${candidates[@]}"; do
+ if [[ -x "${candidate}" ]]; then
+ resolved=${candidate}
+ elif resolved=$(command -v "${candidate}" 2>/dev/null); then
+ :
+ else
+ continue
+ fi
+ version_line=$("${resolved}" --version 2>/dev/null | sed -n '1p' ||
true)
+ if [[ "${version_line}" =~ ^gzip[[:space:]][0-9] ]]; then
+ printf '%s\n' "${resolved}"
+ return 0
+ fi
+ done
+
+ fail "GNU gzip is required for reproducible source archives; on macOS run
'brew install gzip' and set PAIMON_GZIP to the Homebrew gzip executable"
+}
+
while [[ $# -gt 0 ]]; do
case "$1" in
--version)
@@ -128,6 +161,8 @@ DOCS_VERSION=$(
[[ "${DOCS_VERSION}" == "${RELEASE_VERSION}" ]] ||
fail "documentation version ${DOCS_VERSION:-<missing>} does not match
${RELEASE_VERSION}"
+GZIP_BIN=$(find_gnu_gzip)
+
ARTIFACT_NAME="apache-paimon-cpp-${RELEASE_VERSION}-src.tgz"
ARCHIVE_ROOT="paimon-cpp-${RELEASE_VERSION}"
@@ -147,7 +182,10 @@ git -C "${SOURCE_ROOT}" -c tar.umask=0022 archive \
--format=tar \
--prefix="${ARCHIVE_ROOT}/" \
"${GIT_REF}" |
- gzip -n >"${TEMP_DIR}/${ARTIFACT_NAME}"
+ (
+ unset GZIP
+ "${GZIP_BIN}" --no-name --stdout -6
+ ) >"${TEMP_DIR}/${ARTIFACT_NAME}"
SHA512=$(calculate_sha512 "${TEMP_DIR}/${ARTIFACT_NAME}")
printf '%s %s\n' "${SHA512}" "${ARTIFACT_NAME}" \
diff --git a/scripts/releasing/release_rc.sh b/scripts/releasing/release_rc.sh
index f943491..3454863 100755
--- a/scripts/releasing/release_rc.sh
+++ b/scripts/releasing/release_rc.sh
@@ -31,6 +31,8 @@
DIST_DEV_BASE_URL="https://dist.apache.org/repos/dist/dev/paimon";
PREPARE_ONLY=false
DRY_RUN=false
WORKFLOW_DISCOVERY_TIMEOUT_SECONDS=600
+WORKFLOW_RUN_ID=""
+TEMP_DIR=""
usage() {
cat <<'EOF'
@@ -53,8 +55,9 @@ Options:
--dry-run Print the planned release identifiers and exit
-h, --help Show this help
-The script is resumable when the local signed tag or artifacts already exist,
-provided that they match HEAD and pass all verification checks.
+For a published RC, GitHub Actions creates and tests the canonical source
+archive. This script downloads those exact bytes, signs them locally, and
+uploads them to ASF dist/dev. Prepare-only mode creates a local preview.
EOF
}
@@ -117,6 +120,74 @@ wait_for_release_candidate_workflow() {
--exit-status \
--interval 30 ||
fail "Release Candidate workflow run ${run_id} failed"
+ WORKFLOW_RUN_ID=${run_id}
+}
+
+validate_workflow_artifact_directory() {
+ local directory=$1
+ local -a entries
+ local entry
+ local name
+
+ shopt -s dotglob nullglob
+ entries=("${directory}"/*)
+ shopt -u dotglob nullglob
+ [[ ${#entries[@]} -eq 2 ]] ||
+ fail "workflow artifact must contain exactly the archive and checksum"
+ for entry in "${entries[@]}"; do
+ [[ -f "${entry}" ]] ||
+ fail "workflow artifact contains a non-file entry: ${entry}"
+ name=$(basename "${entry}")
+ case "${name}" in
+ "${ARTIFACT_NAME}" | "${ARTIFACT_NAME}.sha512")
+ ;;
+ *)
+ fail "workflow artifact contains an unexpected file: ${name}"
+ ;;
+ esac
+ done
+}
+
+download_and_sign_workflow_artifact() {
+ local workflow_dir="${TEMP_DIR}/source-archive"
+ local source
+ local target
+ local suffix
+
+ [[ -n "${WORKFLOW_RUN_ID}" ]] || fail "release workflow run ID is missing"
+ mkdir -p "${workflow_dir}"
+ gh run download "${WORKFLOW_RUN_ID}" \
+ --repo apache/paimon-cpp \
+ --name source-archive \
+ --dir "${workflow_dir}"
+ validate_workflow_artifact_directory "${workflow_dir}"
+
+ mkdir -p "${OUTPUT_DIR}"
+ for suffix in "" ".sha512"; do
+ source="${workflow_dir}/${ARTIFACT_NAME}${suffix}"
+ target="${OUTPUT_DIR}/${ARTIFACT_NAME}${suffix}"
+ if [[ -e "${target}" ]]; then
+ [[ -f "${target}" ]] || fail "artifact path is not a file:
${target}"
+ cmp "${source}" "${target}" >/dev/null ||
+ fail "existing ${target} differs from workflow run
${WORKFLOW_RUN_ID}"
+ else
+ cp -p "${source}" "${target}"
+ fi
+ done
+
+ ARTIFACT="${OUTPUT_DIR}/${ARTIFACT_NAME}"
+ if [[ -e "${ARTIFACT}.asc" ]]; then
+ [[ -f "${ARTIFACT}.asc" ]] ||
+ fail "artifact signature path is not a file: ${ARTIFACT}.asc"
+ echo "Reusing existing source artifact signature."
+ else
+ echo "Signing workflow source artifact with ${SIGNING_KEY}."
+ gpg --armor \
+ --local-user "${SIGNING_KEY}" \
+ --detach-sign \
+ --output "${ARTIFACT}.asc" \
+ "${ARTIFACT}"
+ fi
}
validate_artifact_directory() {
@@ -224,6 +295,9 @@ EOF
exit 0
fi
+TEMP_DIR=$(mktemp -d)
+trap 'rm -rf "${TEMP_DIR}"' EXIT
+
for command in git gpg python3; do
require_command "${command}"
done
@@ -269,33 +343,33 @@ else
fi
ARTIFACT="${OUTPUT_DIR}/${ARTIFACT_NAME}"
-if [[ -e "${ARTIFACT}" || -e "${ARTIFACT}.asc" || -e "${ARTIFACT}.sha512" ]];
then
- [[ -f "${ARTIFACT}" && -f "${ARTIFACT}.asc" && -f "${ARTIFACT}.sha512" ]]
||
- fail "artifact directory contains an incomplete release candidate"
- echo "Reusing existing artifacts in ${OUTPUT_DIR}."
-else
- "${SCRIPT_DIR}/create_source_release.sh" \
- --version "${VERSION}" \
+if [[ "${PREPARE_ONLY}" == true ]]; then
+ if [[ -e "${ARTIFACT}" || -e "${ARTIFACT}.asc" || -e "${ARTIFACT}.sha512"
]]; then
+ [[ -f "${ARTIFACT}" && -f "${ARTIFACT}.asc" && -f "${ARTIFACT}.sha512"
]] ||
+ fail "artifact directory contains an incomplete release candidate"
+ echo "Reusing existing preview artifacts in ${OUTPUT_DIR}."
+ else
+ "${SCRIPT_DIR}/create_source_release.sh" \
+ --version "${VERSION}" \
+ --git-ref "${RC_TAG}" \
+ --output-dir "${OUTPUT_DIR}" \
+ --signing-key "${SIGNING_KEY}"
+ fi
+
+ "${SCRIPT_DIR}/verify_release_candidate.sh" \
--git-ref "${RC_TAG}" \
- --output-dir "${OUTPUT_DIR}" \
- --signing-key "${SIGNING_KEY}"
-fi
-
-"${SCRIPT_DIR}/verify_release_candidate.sh" \
- --git-ref "${RC_TAG}" \
- --keys-url "https://downloads.apache.org/paimon/KEYS"; \
- "${ARTIFACT}"
+ --keys-url "https://downloads.apache.org/paimon/KEYS"; \
+ "${ARTIFACT}"
+ validate_artifact_directory
-validate_artifact_directory
-
-if [[ "${PREPARE_ONLY}" == true ]]; then
cat <<EOF
Local release preparation completed successfully.
-The signed tag and source artifacts were created and verified locally.
+The signed tag and preview source artifacts were created and verified locally.
No tag was pushed and no artifacts were uploaded to ASF dist/dev.
-Do not start a release vote from this prepare-only run.
+The published workflow artifact is authoritative; do not start a release vote
+from this prepare-only run.
EOF
exit 0
fi
@@ -305,6 +379,14 @@ if svn info "${RC_URL}" >/dev/null 2>&1; then
fi
git push "${REMOTE}" "${RC_TAG}"
wait_for_release_candidate_workflow
+download_and_sign_workflow_artifact
+
+"${SCRIPT_DIR}/verify_release_candidate.sh" \
+ --git-ref "${RC_TAG}" \
+ --keys-url "https://downloads.apache.org/paimon/KEYS"; \
+ "${ARTIFACT}"
+validate_artifact_directory
+
svn import "${OUTPUT_DIR}" "${RC_URL}" \
-m "Add Apache Paimon C++ ${VERSION} RC${RC}"
diff --git a/scripts/releasing/tests/test_release_tools.py
b/scripts/releasing/tests/test_release_tools.py
index d1b4eea..827ddfe 100644
--- a/scripts/releasing/tests/test_release_tools.py
+++ b/scripts/releasing/tests/test_release_tools.py
@@ -20,6 +20,8 @@ import hashlib
import io
import json
import os
+import re
+import shutil
import subprocess
import sys
import tarfile
@@ -33,9 +35,26 @@ RELEASING_DIR = Path(__file__).resolve().parents[1]
ARCHIVE_VALIDATOR = RELEASING_DIR / "validate_source_archive.py"
VERSION_TOOL = RELEASING_DIR / "bump_version.py"
RELEASE_VERIFIER = RELEASING_DIR / "verify_release_candidate.sh"
+SOURCE_RELEASE_CREATOR = RELEASING_DIR / "create_source_release.sh"
+SOURCE_ROOT = RELEASING_DIR.parents[1]
class ReleaseToolTest(unittest.TestCase):
+ def head_release_version(self) -> str:
+ result = subprocess.run(
+ ["git", "-C", str(SOURCE_ROOT), "show", "HEAD:CMakeLists.txt"],
+ universal_newlines=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ check=False,
+ )
+ self.assertEqual(result.returncode, 0, msg=result.stderr)
+ match = re.search(
+ r"^\s*VERSION\s+(\d+\.\d+\.\d+)\s*$", result.stdout, re.MULTILINE
+ )
+ self.assertIsNotNone(match)
+ return match.group(1)
+
def run_tool(
self, tool: Path, *args: str, expected_returncode: int = 0
) -> subprocess.CompletedProcess:
@@ -314,6 +333,269 @@ source_root=$(cd "$(dirname "$0")/../.." && pwd)
result.stderr,
)
+ @unittest.skipUnless(
+ shutil.which("gpg") and shutil.which("gzip"), "gpg and gzip are
required"
+ )
+ def test_verifier_uses_keys_file_for_unsigned_artifact_tag(self) -> None:
+ with tempfile.TemporaryDirectory() as temp:
+ directory = Path(temp)
+ source_root = directory / "source"
+ releasing_dir = source_root / "scripts/releasing"
+ releasing_dir.mkdir(parents=True)
+ for script in (
+ "bump_version.py",
+ "create_source_release.sh",
+ "validate_source_archive.py",
+ "verify_release_candidate.sh",
+ ):
+ shutil.copy2(RELEASING_DIR / script, releasing_dir / script)
+
+ files = {
+ "CMakeLists.txt": "project(paimon\n VERSION 1.2.3\n)\n",
+ "LICENSE": "Apache License\n",
+ "NOTICE": "Apache Paimon\n",
+ "docs/source/conf.py": 'version = "1.2.3"\n',
+ "docs/source/_static/versions.json": (
+ '[{"name": "1.2.3", "version": "1.2.3", '
+ '"url": "https://paimon.apache.org/docs/cpp/"}]\n'
+ ),
+ ".github/.rat-excludes": "",
+ }
+ for name, content in files.items():
+ path = source_root / name
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(content, encoding="utf-8")
+
+ empty_git_config = directory / "empty-gitconfig"
+ empty_git_config.touch()
+ git_env = os.environ.copy()
+ for name in list(git_env):
+ if name in ("GIT_CONFIG_COUNT", "GIT_CONFIG_PARAMETERS") or
re.fullmatch(
+ r"GIT_CONFIG_(KEY|VALUE)_\d+", name
+ ):
+ del git_env[name]
+ git_env["GIT_CONFIG_GLOBAL"] = str(empty_git_config)
+ git_env["GIT_CONFIG_SYSTEM"] = str(empty_git_config)
+
+ subprocess.run(
+ ["git", "init", "-q", str(source_root)], env=git_env,
check=True
+ )
+ subprocess.run(
+ ["git", "-C", str(source_root), "config", "user.name",
"Release Test"],
+ env=git_env,
+ check=True,
+ )
+ subprocess.run(
+ [
+ "git",
+ "-C",
+ str(source_root),
+ "config",
+ "user.email",
+ "[email protected]",
+ ],
+ env=git_env,
+ check=True,
+ )
+ subprocess.run(
+ ["git", "-C", str(source_root), "add", "."],
+ env=git_env,
+ check=True,
+ )
+ subprocess.run(
+ ["git", "-C", str(source_root), "commit", "-q", "-m", "test"],
+ env=git_env,
+ check=True,
+ )
+
+ signing_home = directory / "signing-home"
+ signing_home.mkdir(mode=0o700)
+ signing_env = git_env.copy()
+ signing_env["GNUPGHOME"] = str(signing_home)
+ subprocess.run(
+ [
+ "gpg",
+ "--batch",
+ "--passphrase",
+ "",
+ "--quick-generate-key",
+ "Release Test <[email protected]>",
+ "ed25519",
+ "sign",
+ "0",
+ ],
+ env=signing_env,
+ check=True,
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.DEVNULL,
+ )
+ key_listing = subprocess.run(
+ ["gpg", "--batch", "--with-colons", "--list-secret-keys"],
+ env=signing_env,
+ universal_newlines=True,
+ stdout=subprocess.PIPE,
+ check=True,
+ )
+ fingerprint = next(
+ line.split(":")[9]
+ for line in key_listing.stdout.splitlines()
+ if line.startswith("fpr:") # codespell:ignore fpr
+ )
+ subprocess.run(
+ [
+ "git",
+ "-C",
+ str(source_root),
+ "tag",
+ "-s",
+ "-u",
+ fingerprint,
+ "-m",
+ "test tag",
+ "v1.2.3-rc1",
+ ],
+ env=signing_env,
+ check=True,
+ )
+
+ keys_file = directory / "KEYS"
+ with keys_file.open("w", encoding="utf-8") as output:
+ subprocess.run(
+ ["gpg", "--batch", "--armor", "--export", fingerprint],
+ env=signing_env,
+ universal_newlines=True,
+ stdout=output,
+ check=True,
+ )
+
+ real_gzip = shutil.which("gzip")
+ self.assertIsNotNone(real_gzip)
+ fake_gzip = directory / "gzip"
+ fake_gzip.write_text(
+ "#!/usr/bin/env bash\n"
+ "set -euo pipefail\n"
+ "if [[ ${1:-} == --version ]]; then\n"
+ " echo 'gzip 1.99'\n"
+ " exit 0\n"
+ "fi\n"
+ f'exec "{real_gzip}" -n -c -6\n',
+ encoding="utf-8",
+ )
+ fake_gzip.chmod(0o755)
+ release_env = git_env.copy()
+ release_env["PAIMON_GZIP"] = str(fake_gzip)
+
+ artifact_dir = directory / "release"
+ subprocess.run(
+ [
+ "bash",
+ str(releasing_dir / "create_source_release.sh"),
+ "--version",
+ "1.2.3",
+ "--git-ref",
+ "v1.2.3-rc1",
+ "--output-dir",
+ str(artifact_dir),
+ ],
+ env=release_env,
+ check=True,
+ stdout=subprocess.DEVNULL,
+ )
+ artifact = artifact_dir / "apache-paimon-cpp-1.2.3-src.tgz"
+ result = subprocess.run(
+ [
+ "bash",
+ str(releasing_dir / "verify_release_candidate.sh"),
+ "--allow-unsigned",
+ "--keys-file",
+ str(keys_file),
+ "--git-ref",
+ "v1.2.3-rc1",
+ "--skip-rat",
+ "--skip-build",
+ str(artifact),
+ ],
+ universal_newlines=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ env=release_env,
+ check=False,
+ )
+ self.assertEqual(
+ result.returncode, 0, msg=result.stdout + result.stderr
+ )
+ self.assertIn("Git ref reproducibility: valid", result.stdout)
+
+ def test_source_creator_rejects_non_gnu_gzip(self) -> None:
+ with tempfile.TemporaryDirectory() as temp:
+ directory = Path(temp)
+ fake_gzip = directory / "gzip"
+ fake_gzip.write_text(
+ "#!/usr/bin/env bash\n"
+ "echo 'Apple gzip 999.0'\n",
+ encoding="utf-8",
+ )
+ fake_gzip.chmod(0o755)
+ env = os.environ.copy()
+ env["PAIMON_GZIP"] = str(fake_gzip)
+ result = subprocess.run(
+ [
+ "bash",
+ str(SOURCE_RELEASE_CREATOR),
+ "--version",
+ self.head_release_version(),
+ "--git-ref",
+ "HEAD",
+ "--output-dir",
+ str(directory / "release"),
+ ],
+ universal_newlines=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ env=env,
+ check=False,
+ )
+ self.assertEqual(result.returncode, 1, msg=result.stdout +
result.stderr)
+ self.assertIn("GNU gzip is required", result.stderr)
+
+ def test_source_creator_clears_gzip_environment_options(self) -> None:
+ with tempfile.TemporaryDirectory() as temp:
+ directory = Path(temp)
+ fake_gzip = directory / "gzip"
+ fake_gzip.write_text(
+ "#!/usr/bin/env bash\n"
+ "set -euo pipefail\n"
+ "if [[ ${1:-} == --version ]]; then\n"
+ " echo 'gzip 1.99'\n"
+ " exit 0\n"
+ "fi\n"
+ "[[ -z ${GZIP+x} ]] || { echo 'GZIP was not cleared' >&2; exit
1; }\n"
+ "dd of=/dev/null 2>/dev/null\n",
+ encoding="utf-8",
+ )
+ fake_gzip.chmod(0o755)
+ env = os.environ.copy()
+ env["GZIP"] = "-9"
+ env["PAIMON_GZIP"] = str(fake_gzip)
+ result = subprocess.run(
+ [
+ "bash",
+ str(SOURCE_RELEASE_CREATOR),
+ "--version",
+ self.head_release_version(),
+ "--git-ref",
+ "HEAD",
+ "--output-dir",
+ str(directory / "release"),
+ ],
+ universal_newlines=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ env=env,
+ check=False,
+ )
+ self.assertEqual(result.returncode, 0, msg=result.stdout +
result.stderr)
+
def create_version_tree(self, root: Path) -> None:
(root / "docs/source/_static").mkdir(parents=True)
(root / "CMakeLists.txt").write_text(
diff --git a/scripts/releasing/verify_release_candidate.sh
b/scripts/releasing/verify_release_candidate.sh
index 99e7df0..0f3e76d 100755
--- a/scripts/releasing/verify_release_candidate.sh
+++ b/scripts/releasing/verify_release_candidate.sh
@@ -28,6 +28,8 @@
DIST_DEV_BASE_URL="https://dist.apache.org/repos/dist/dev/paimon";
KEYS_URL=""
KEYS_FILE=""
GIT_REF=""
+VERIFY_GNUPG_HOME=""
+VERIFY_KEYS_FILE=""
RAT_JAR=${RAT_JAR:-}
RAT_VERSION="0.16.1"
ALLOW_UNSIGNED=false
@@ -52,7 +54,7 @@ Download options:
Trust and reproducibility:
--keys-url URL Download KEYS and verify in an isolated GPG home
--keys-file FILE Import this KEYS file into an isolated GPG home
- --git-ref REF Regenerate the archive from REF and compare bytes
+ --git-ref REF Regenerate with GNU gzip and compare archive bytes
Verification options:
--rat-jar FILE Apache RAT executable jar (or set RAT_JAR)
@@ -99,6 +101,24 @@ download_file() {
fi
}
+prepare_verification_keyring() {
+ [[ -z "${VERIFY_GNUPG_HOME}" ]] || return 0
+
+ command -v gpg >/dev/null 2>&1 || fail "gpg is required for signature
verification"
+ VERIFY_GNUPG_HOME="${TEMP_DIR}/gnupg"
+ mkdir -m 700 "${VERIFY_GNUPG_HOME}"
+ if [[ -n "${KEYS_URL}" ]]; then
+ VERIFY_KEYS_FILE="${TEMP_DIR}/KEYS"
+ download_file "${KEYS_URL}" "${VERIFY_KEYS_FILE}"
+ else
+ VERIFY_KEYS_FILE=$(cd "$(dirname "${KEYS_FILE}")" && pwd)/$(basename
"${KEYS_FILE}")
+ fi
+ [[ -f "${VERIFY_KEYS_FILE}" ]] ||
+ fail "KEYS file does not exist: ${VERIFY_KEYS_FILE}"
+ gpg --batch --homedir "${VERIFY_GNUPG_HOME}" \
+ --import "${VERIFY_KEYS_FILE}" >/dev/null
+}
+
while [[ $# -gt 0 ]]; do
case "$1" in
--version)
@@ -249,19 +269,10 @@ echo "SHA-512 checksum: valid"
if [[ -f "${SIGNATURE_FILE}" ]]; then
command -v gpg >/dev/null 2>&1 || fail "gpg is required to verify the
signature"
if [[ -n "${KEYS_URL}" || -n "${KEYS_FILE}" ]]; then
- GNUPG_HOME="${TEMP_DIR}/gnupg"
- mkdir -m 700 "${GNUPG_HOME}"
- if [[ -n "${KEYS_URL}" ]]; then
- KEYS_FILE="${TEMP_DIR}/KEYS"
- download_file "${KEYS_URL}" "${KEYS_FILE}"
- else
- KEYS_FILE=$(cd "$(dirname "${KEYS_FILE}")" && pwd)/$(basename
"${KEYS_FILE}")
- fi
- [[ -f "${KEYS_FILE}" ]] || fail "KEYS file does not exist:
${KEYS_FILE}"
- gpg --batch --homedir "${GNUPG_HOME}" --import "${KEYS_FILE}"
>/dev/null
- gpg --batch --homedir "${GNUPG_HOME}" \
+ prepare_verification_keyring
+ gpg --batch --homedir "${VERIFY_GNUPG_HOME}" \
--verify "${SIGNATURE_FILE}" "${ARTIFACT}"
- echo "OpenPGP signature: valid against ${KEYS_FILE}"
+ echo "OpenPGP signature: valid against ${VERIFY_KEYS_FILE}"
else
gpg --verify "${SIGNATURE_FILE}" "${ARTIFACT}"
echo "OpenPGP signature: valid against the default GPG keyring"
@@ -313,7 +324,13 @@ if [[ -n "${GIT_REF}" ]]; then
fail "Git ref does not resolve to a commit: ${GIT_REF}"
if git -C "${SOURCE_ROOT}" rev-parse --verify "${GIT_REF}^{tag}" \
>/dev/null 2>&1; then
- git -C "${SOURCE_ROOT}" verify-tag "${GIT_REF}"
+ if [[ -n "${KEYS_URL}" || -n "${KEYS_FILE}" ]]; then
+ prepare_verification_keyring
+ GNUPGHOME=${VERIFY_GNUPG_HOME} \
+ git -C "${SOURCE_ROOT}" verify-tag "${GIT_REF}"
+ else
+ git -C "${SOURCE_ROOT}" verify-tag "${GIT_REF}"
+ fi
fi
REPRO_DIR="${TEMP_DIR}/reproduced"
"${SCRIPT_DIR}/create_source_release.sh" \
