This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 795aeb76a468 [chore](ci) Measure five-locale publishing and raise the
temporary build budget (#4484)
795aeb76a468 is described below
commit 795aeb76a468b06fb12fd8f218fa33c31d76b63c
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Fri Oct 23 23:42:33 2026 +0100
[chore](ci) Measure five-locale publishing and raise the temporary build
budget (#4484)
Five-locale publishing can exceed the current 90-minute build limit
before strict verification and site artifact upload finish. Give
cron/manual builds a temporary 180-minute ceiling and retain separate
build/deploy measurements to establish a complete publishing budget with
about 30% margin.
Record the resolved source and trusted workflow SHAs, runner
image/hardware and cache conditions, step timings, build/verification
GNU time RSS, sampled system memory/disk use, and output/artifact sizes
and digests. Keep telemetry outside the published site and retain it
after failures. Read measurement code from the trusted workflow commit
so manual publishing still supports older protected source branches.
Build remains read-only without production secrets; deployment keeps its
existing Production environment and trusted checkout.
Validation:
- 22 Python tests passed, including deployment boundaries, failure
reporting, credential exclusion and API timing/resource windows.
- 22 build/publishing tests passed; one Apache-dependent test skipped.
- Real five-locale publishing fixture passed: 107 output files, 40
redirects and 146 compiled asset references verified.
- Both publishing workflows passed actionlint 1.7.12, YAML parsing and
Bash syntax checks; Linux resource snapshot smoke check passed.
Full active-version CI and a successful complete production artifact
transfer with this instrumentation are still required. The temporary
timeout is not a performance result; accept the final budget only from
complete job durations including telemetry and cleanup. No production
dispatch or deployment was initiated for this change.
---------
Co-authored-by: morningman <[email protected]>
---
.github/scripts/publish_metrics.py | 238 ++++++++++++++++++++++++++++
.github/scripts/test_publish_metrics.py | 121 ++++++++++++++
.github/workflows/cron-deploy-website.yml | 70 +++++++-
.github/workflows/manual-deploy-website.yml | 70 +++++++-
scripts/build-locales.md | 52 ++++++
5 files changed, 549 insertions(+), 2 deletions(-)
diff --git a/.github/scripts/publish_metrics.py
b/.github/scripts/publish_metrics.py
new file mode 100644
index 000000000000..6d109319ad42
--- /dev/null
+++ b/.github/scripts/publish_metrics.py
@@ -0,0 +1,238 @@
+"""Measure publishing jobs without putting reports or credentials in the
site."""
+
+import argparse
+from datetime import datetime
+import hashlib
+import json
+import math
+import os
+from pathlib import Path
+import platform
+import shutil
+import subprocess
+import sys
+import time
+import urllib.request
+
+
+REPORTS = Path(".i18n-build/publish-metrics")
+
+
+def write_json(path, value):
+ path.write_text(json.dumps(value, indent=2) + "\n")
+
+
+def git(*args):
+ return subprocess.check_output(["git", *args]).decode().strip()
+
+
+def snapshot():
+ memory = {}
+ if Path("/proc/meminfo").exists():
+ for line in Path("/proc/meminfo").read_text().splitlines():
+ key, value = line.split(":", 1)
+ if key in ("MemTotal", "MemAvailable", "SwapTotal", "SwapFree"):
+ memory[key + "KiB"] = int(value.split()[0])
+ disks = {}
+ for mount in (str(Path.cwd()), "/mnt"):
+ if Path(mount).exists():
+ usage = shutil.disk_usage(mount)
+ disks[mount] = {"totalBytes": usage.total, "usedBytes": usage.used,
+ "freeBytes": usage.free}
+ return {"time": time.time(), "memory": memory, "disks": disks}
+
+
+def sample():
+ # A timeout/cancellation may prevent finish from running. The runner also
+ # cleans up this process at job end; the deadline bounds standalone use.
+ deadline = time.monotonic() + 4 * 3600
+ while not (REPORTS / "stop").exists() and time.monotonic() < deadline:
+ with (REPORTS / "samples.jsonl").open("a") as out:
+ out.write(json.dumps(snapshot()) + "\n")
+ time.sleep(10)
+
+
+def launch_sampler():
+ subprocess.Popen([sys.executable, str(Path(__file__).resolve()), "sample"],
+ stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
+ stderr=subprocess.DEVNULL, start_new_session=True)
+
+
+def start(job):
+ REPORTS.mkdir(parents=True, exist_ok=True)
+ (REPORTS / "stop").unlink(missing_ok=True)
+ context = {key: os.environ.get(key) for key in (
+ "GITHUB_REPOSITORY", "GITHUB_RUN_ID", "GITHUB_RUN_ATTEMPT",
"GITHUB_SHA",
+ "GITHUB_WORKFLOW_REF", "GITHUB_EVENT_NAME", "RUNNER_OS", "RUNNER_ARCH",
+ "ImageOS", "ImageVersion", "DEPLOY_SOURCE_SHA", "DORIS_ENABLE_JA",
+ "DORIS_ENABLE_KO", "DORIS_ENABLE_RU", "DOCS_VERSIONS")}
+ write_json(REPORTS / "context.json", {
+ "schemaVersion": 1, "job": job, "startedAt": time.time(),
+ "checkoutSha": git("rev-parse", "HEAD"), "context": context,
+ "platform": platform.platform(), "cpuCount": os.cpu_count(),
+ "activeVersions": (["current",
*json.loads(Path("versions.json").read_text())]
+ if job == "build" else None),
+ "checkoutActiveVersions": ["current",
*json.loads(Path("versions.json").read_text())],
+ "cachePolicy": ("fresh runner; no dependency cache; yarn cache clean
before install"
+ if job == "build" else "fresh runner; site downloaded
from this run's artifact"),
+ "initialResources": snapshot(),
+ })
+ launch_sampler()
+
+
+def site_inventory(site):
+ if not site.is_dir():
+ return None
+ files, size = 0, 0
+ for root, _, names in os.walk(site):
+ for name in names:
+ item = Path(root) / name
+ if item.is_file():
+ files += 1
+ size += item.stat().st_size
+ return {"files": files, "bytes": size}
+
+
+def api(url):
+ req = urllib.request.Request(url, headers={
+ "Authorization": "Bearer " + os.environ["GH_TOKEN"],
+ "Accept": "application/vnd.github+json", "X-GitHub-Api-Version":
"2022-11-28"})
+ with urllib.request.urlopen(req, timeout=15) as response:
+ return json.load(response)
+
+
+def job_details(context):
+ env = context["context"]
+ if not os.environ.get("GH_TOKEN"):
+ return None
+ url =
(f"https://api.github.com/repos/{env['GITHUB_REPOSITORY']}/actions/runs/"
+
f"{env['GITHUB_RUN_ID']}/attempts/{env['GITHUB_RUN_ATTEMPT']}/jobs?per_page=100")
+ return next(job for job in api(url)["jobs"] if job["name"] ==
context["job"])
+
+
+def duration(start, end):
+ return (datetime.fromisoformat(end.replace("Z", "+00:00")) -
+ datetime.fromisoformat(start.replace("Z",
"+00:00"))).total_seconds()
+
+
+def tool_version(name):
+ try:
+ result = subprocess.run([name, "--version"], capture_output=True,
text=True, timeout=15)
+ return result.stdout.strip() if result.returncode == 0 else None
+ except (OSError, subprocess.TimeoutExpired):
+ return None
+
+
+def resource_peaks(samples):
+ disks = {}
+ available = []
+ for sample in samples:
+ memory = sample["memory"]
+ if "MemAvailableKiB" in memory:
+ available.append(memory["MemAvailableKiB"])
+ for mount, usage in sample["disks"].items():
+ current = disks.setdefault(mount, {"maxUsedBytes": 0,
"minFreeBytes": usage["freeBytes"]})
+ current["maxUsedBytes"] = max(current["maxUsedBytes"],
usage["usedBytes"])
+ current["minFreeBytes"] = min(current["minFreeBytes"],
usage["freeBytes"])
+ return {"sampleCount": len(samples), "sampleIntervalSeconds": 10,
+ "minSystemMemoryAvailableKiB": min(available) if available else
None,
+ "disks": disks}
+
+
+def finish(budget):
+ (REPORTS / "stop").touch()
+ report = json.loads((REPORTS / "context.json").read_text())
+ report["finishedAt"] = time.time()
+ report["elapsedThroughReportSeconds"] = report["finishedAt"] -
report["startedAt"]
+ report["finalResources"] = snapshot()
+ report["site"] = site_inventory(Path("build"))
+ if report["job"] == "deploy" and Path("build/versions.json").is_file():
+ # Manual deploy code is checked out at the workflow commit, which
+ # may have different versions from the protected source branch.
+ report["activeVersions"] = ["current",
*json.loads(Path("build/versions.json").read_text())]
+ report["trackedDiffSha256"] = hashlib.sha256(
+ subprocess.check_output(["git", "diff", "HEAD",
"--binary"])).hexdigest()
+ report["nodeVersion"] = tool_version("node")
+ report["yarnVersion"] = tool_version("yarn")
+ report["jobBudgetMinutes"] = budget
+ report["steps"] = []
+ samples = [report["initialResources"], report["finalResources"]]
+ if (REPORTS / "samples.jsonl").exists():
+ for line in (REPORTS / "samples.jsonl").read_text().splitlines():
+ try:
+ samples.append(json.loads(line))
+ except json.JSONDecodeError:
+ pass # The sampler may still be finishing its final line.
+ report["sampledResources"] = resource_peaks(samples)
+ try:
+ job = job_details(report)
+ if job:
+ report["runner"] = {key: job.get(key) for key in (
+ "id", "runner_name", "runner_group_name", "labels",
"started_at", "html_url")}
+ for step in job["steps"]:
+ item = {key: step.get(key) for key in (
+ "name", "number", "status", "conclusion", "started_at",
"completed_at")}
+ if item["started_at"] and item["completed_at"]:
+ item["seconds"] = duration(item["started_at"],
item["completed_at"])
+ start =
datetime.fromisoformat(item["started_at"].replace("Z", "+00:00")).timestamp()
+ end =
datetime.fromisoformat(item["completed_at"].replace("Z", "+00:00")).timestamp()
+ item["sampledResources"] = resource_peaks(
+ [sample for sample in samples if start <=
sample["time"] <= end])
+ report["steps"].append(item)
+ # Includes checkout/setup before start, excludes this reporting
+ # step, telemetry upload and cleanup. Not a final job duration.
+ report["elapsedThroughReportSeconds"] = (
+ report["finishedAt"] -
datetime.fromisoformat(job["started_at"].replace("Z", "+00:00")).timestamp())
+ except Exception as error:
+ report["jobApiError"] = type(error).__name__
+ print("::warning::Publish step timing API unavailable; retain the
run's final jobs JSON")
+ artifact_id = os.environ.get("SITE_ARTIFACT_ID")
+ if os.environ.get("GH_TOKEN") and report["context"]["GITHUB_REPOSITORY"]:
+ report["siteArtifact"] = {"uploadId": artifact_id,
+ "uploadDigest":
os.environ.get("SITE_ARTIFACT_DIGEST")}
+ try:
+ env = report["context"]
+ artifacts =
api(f"https://api.github.com/repos/{env['GITHUB_REPOSITORY']}/actions/runs/{env['GITHUB_RUN_ID']}/artifacts?per_page=100")
+ name =
f"site-build-{env['GITHUB_RUN_ID']}-{env['GITHUB_RUN_ATTEMPT']}"
+ artifact = next(item for item in artifacts["artifacts"] if
item["name"] == name)
+ report["siteArtifact"].update({key: artifact.get(key) for key in (
+ "id", "name", "size_in_bytes", "digest", "created_at",
"expires_at")})
+ except Exception as error:
+ report["siteArtifact"]["apiError"] = type(error).__name__
+ measured = report["elapsedThroughReportSeconds"]
+ report["provisionalBudgetWith30PercentMinutes"] = math.ceil(measured * 1.3
/ 60)
+ report["headroomPercentThroughReport"] = round((budget * 60 / measured -
1) * 100, 1)
+ # Keep the GNU time output, existing per-locale/build/optimization reports
+ # and ten-second samples as raw evidence alongside this compact report.
+ write_json(REPORTS / "report.json", report)
+ summary = os.environ.get("GITHUB_STEP_SUMMARY")
+ if summary:
+ with open(summary, "a") as out:
+ out.write(f"### {report['job']} publishing measurements\n\n"
+ f"Source: `{report['context']['DEPLOY_SOURCE_SHA']}`; "
+ f"checkout: `{report['checkoutSha']}`.\n\n"
+ f"Elapsed through reporting: {measured / 60:.1f} min; "
+ f"job limit: {budget} min; provisional 30% budget: "
+ f"{report['provisionalBudgetWith30PercentMinutes']} min.
"
+ "Add telemetry upload and cleanup from the final jobs
API before accepting a budget.\n\n"
+ "| Step | Result | Seconds |\n| --- | --- | ---: |\n")
+ for step in report["steps"]:
+ out.write(f"| {step['name']} | {step['conclusion'] or
step['status']} | {step.get('seconds', '')} |\n")
+ if report["provisionalBudgetWith30PercentMinutes"] > budget:
+ print("::warning::Publish job has less than 30% measured budget
margin")
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("operation", choices=("start", "sample", "finish"))
+ parser.add_argument("--job", choices=("build", "deploy"))
+ parser.add_argument("--budget", type=int, default=180)
+ args = parser.parse_args()
+ if args.operation == "start":
+ if not args.job:
+ parser.error("start requires --job")
+ start(args.job)
+ elif args.operation == "sample":
+ sample()
+ else:
+ finish(args.budget)
diff --git a/.github/scripts/test_publish_metrics.py
b/.github/scripts/test_publish_metrics.py
new file mode 100644
index 000000000000..b775e7e4d52a
--- /dev/null
+++ b/.github/scripts/test_publish_metrics.py
@@ -0,0 +1,121 @@
+import importlib.util
+import json
+import os
+from pathlib import Path
+import subprocess
+import tempfile
+import unittest
+from datetime import datetime, timezone
+import time
+from unittest.mock import patch
+
+
+SCRIPT = Path(__file__).with_name("publish_metrics.py")
+ROOT = SCRIPT.parents[2]
+spec = importlib.util.spec_from_file_location("publish_metrics", SCRIPT)
+metrics = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(metrics)
+
+
+class PublishMetricsTests(unittest.TestCase):
+ def test_resource_summary_preserves_disk_and_memory_extrema(self):
+ samples = [
+ {"memory": {"MemAvailableKiB": 100}, "disks": {"/": {"usedBytes":
800, "freeBytes": 200}}},
+ {"memory": {"MemAvailableKiB": 50}, "disks": {"/": {"usedBytes":
850, "freeBytes": 150}}},
+ ]
+ peaks = metrics.resource_peaks(samples)
+ self.assertEqual(peaks["minSystemMemoryAvailableKiB"], 50)
+ self.assertEqual(peaks["disks"]["/"], {"maxUsedBytes": 850,
"minFreeBytes": 150})
+
self.assertEqual(metrics.resource_peaks([])["minSystemMemoryAvailableKiB"],
None)
+ self.assertEqual(metrics.duration("2026-10-23T14:00:00Z",
"2026-10-23T14:01:30Z"), 90)
+
+ def test_failed_build_still_reports_without_serializing_credentials(self):
+ with tempfile.TemporaryDirectory() as directory:
+ previous = Path.cwd()
+ os.chdir(directory)
+ self.addCleanup(os.chdir, previous)
+ subprocess.run(["git", "init", "-q"], check=True)
+ Path("versions.json").write_text('["4.x", "3.x", "2.1"]')
+ subprocess.run(["git", "add", "versions.json"], check=True)
+ subprocess.run(["git", "-c", "user.name=Fixture", "-c",
"[email protected]",
+ "commit", "-qm", "fixture"], check=True)
+ with patch.dict(os.environ, {"GH_TOKEN": "SECRET_MUST_NOT_APPEAR",
"DEPLOY_SOURCE_SHA": "a" * 40,
+ "GITHUB_REPOSITORY": "fixture/site",
"GITHUB_RUN_ID": "42", "GITHUB_RUN_ATTEMPT": "1",
+ "GITHUB_STEP_SUMMARY":
str(Path(directory) / "summary.md")}):
+ with patch.object(metrics, "git", return_value="a" * 40),
patch.object(metrics, "launch_sampler"):
+ metrics.start("build")
+ with patch.object(metrics, "job_details",
side_effect=RuntimeError("SECRET_MUST_NOT_APPEAR")), \
+ patch.object(metrics, "api",
side_effect=RuntimeError("SECRET_MUST_NOT_APPEAR")):
+ metrics.finish(180)
+ text = (metrics.REPORTS / "report.json").read_text()
+ report = json.loads(text)
+ self.assertNotIn("SECRET_MUST_NOT_APPEAR", text)
+ self.assertEqual(report["jobApiError"], "RuntimeError")
+ self.assertIsNone(report["site"])
+ self.assertTrue((metrics.REPORTS / "stop").exists())
+ self.assertEqual(report["activeVersions"], ["current", "4.x",
"3.x", "2.1"])
+ self.assertIn("final jobs API", Path("summary.md").read_text())
+ now = time.time()
+ def stamp(offset):
+ return datetime.fromtimestamp(now + offset,
timezone.utc).isoformat()
+ job = {"name": "build", "started_at": stamp(-120), "steps": [
+ {"name": "Build", "started_at": stamp(-110),
"completed_at": stamp(-20),
+ "conclusion": "failure", "status": "completed"}]}
+ sample = {"time": now - 90, "memory": {"MemAvailableKiB":
100}, "disks": {}}
+ (metrics.REPORTS /
"samples.jsonl").write_text(json.dumps(sample) + '\n{"partial":')
+ with patch.object(metrics, "job_details", return_value=job),
patch.object(metrics, "api", return_value={
+ "artifacts": [{"id": 9, "name": "site-build-42-1",
"size_in_bytes": 123}]}):
+ metrics.finish(180)
+ report = json.loads((metrics.REPORTS /
"report.json").read_text())
+ self.assertEqual(report["steps"][0]["seconds"], 90)
+
self.assertEqual(report["steps"][0]["sampledResources"]["sampleCount"], 1)
+ self.assertEqual(report["steps"][0]["conclusion"], "failure")
+ self.assertEqual(report["siteArtifact"]["size_in_bytes"], 123)
+ with patch.object(metrics, "git", return_value="b" * 40),
patch.object(metrics, "launch_sampler"):
+ metrics.start("deploy")
+ Path("build").mkdir()
+ Path("build/versions.json").write_text('["3.x"]')
+ with patch.object(metrics, "job_details", return_value=None),
patch.object(metrics, "api", return_value={"artifacts": []}):
+ metrics.finish(180)
+ report = json.loads((metrics.REPORTS /
"report.json").read_text())
+ self.assertEqual(report["activeVersions"], ["current", "3.x"])
+ self.assertEqual(report["checkoutActiveVersions"], ["current",
"4.x", "3.x", "2.1"])
+ os.chdir(previous)
+
+ def test_inventory_counts_hidden_control_files_and_all_locales(self):
+ with tempfile.TemporaryDirectory() as directory:
+ site = Path(directory)
+ (site / "ja").mkdir()
+ (site / ".asf.yaml").write_bytes(b"control")
+ (site / "index.html").write_bytes(b"english")
+ (site / "ja/index.html").write_bytes(b"japanese")
+ self.assertEqual(metrics.site_inventory(site), {"files": 3,
"bytes": 22})
+
+ def test_measurement_code_and_reports_preserve_deployment_boundaries(self):
+ raw = subprocess.check_output(["node", "-e", """
+ const fs=require('fs'),yaml=require('js-yaml');
+ console.log(JSON.stringify(['cron','manual'].map(name=>yaml.load(
+
fs.readFileSync(`.github/workflows/${name}-deploy-website.yml`,'utf8')))));
+ """], cwd=ROOT)
+ for workflow in json.loads(raw):
+ build, deploy = (workflow["jobs"][name] for name in ("build",
"deploy"))
+ self.assertEqual(build["timeout-minutes"], 180)
+ self.assertEqual(build["permissions"], {"contents": "read",
"actions": "read"})
+ self.assertNotIn("environment", build)
+ self.assertNotIn("secrets.", json.dumps(build))
+ trusted = next(step for step in build["steps"] if step.get("name")
== "Checkout trusted measurement code")
+ self.assertEqual(trusted["with"]["ref"], "${{ github.sha }}")
+ self.assertFalse(trusted["with"]["persist-credentials"])
+ self.assertEqual(deploy["environment"], "Production")
+ self.assertEqual(deploy["needs"], ["source", "build"])
+ for job in (build, deploy):
+ report = next(step for step in job["steps"] if
step.get("name") == "Report publishing measurements")
+ self.assertIn("always()", report["if"])
+ upload = next(step for step in job["steps"] if
step.get("name") == "Upload publishing measurements")
+ self.assertIn("always()", upload["if"])
+ self.assertTrue(upload["with"]["include-hidden-files"])
+ self.assertTrue(all(path.startswith(".i18n-build/") for path
in upload["with"]["path"].splitlines()))
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/.github/workflows/cron-deploy-website.yml
b/.github/workflows/cron-deploy-website.yml
index e9ae096a7894..63bea61a6339 100644
--- a/.github/workflows/cron-deploy-website.yml
+++ b/.github/workflows/cron-deploy-website.yml
@@ -43,7 +43,12 @@ jobs:
build:
needs: source
runs-on: ubuntu-latest
- timeout-minutes: 90
+ # Temporary ceiling: validate build + verification + artifact transfer
+ # with 30% margin using the publishing measurement reports.
+ timeout-minutes: 180
+ permissions:
+ contents: read
+ actions: read
env:
DORIS_ENABLE_JA: 'true'
DORIS_ENABLE_KO: 'true'
@@ -61,6 +66,21 @@ jobs:
fetch-depth: 0
filter: blob:none
+ - name: Checkout trusted measurement code
+ uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+ with:
+ ref: ${{ github.sha }}
+ persist-credentials: false
+ path: .publish-tools
+ sparse-checkout: .github/scripts/publish_metrics.py
+ sparse-checkout-cone-mode: false
+
+ - name: Start publishing measurements
+ id: metrics-start
+ env:
+ DEPLOY_SOURCE_SHA: ${{ needs.source.outputs.source_sha }}
+ run: python3 .publish-tools/.github/scripts/publish_metrics.py start
--job build
+
- name: Add Chinese Fonts
run: |
sudo apt-get install ttf-mscorefonts-installer
@@ -103,6 +123,7 @@ jobs:
env:
GITHUB_TOKEN: ${{ github.token }}
run: |
+ /usr/bin/time -v -o .i18n-build/publish-metrics/build-time.txt
bash -e -o pipefail <<'BUILD'
npm install -g yarn
yarn cache clean
export NODE_OPTIONS=--max-old-space-size=8192
@@ -114,16 +135,19 @@ jobs:
cp .asf-site.yaml ./build/.asf.yaml
cp versions.json ./build/
ls build
+ BUILD
- name: Verify localized publishing artifacts
# Publishing builds every enabled locale, so every managed locale
# must be present and pass the strict cross-locale verification.
run: |
+ /usr/bin/time -v -o .i18n-build/publish-metrics/verify-time.txt
bash -e -o pipefail <<'VERIFY'
for LOCALE in $(node scripts/i18n/managed-locales.cjs); do
test -s "build/${LOCALE}/index.html"
node scripts/i18n/verify-static.cjs \
--site build --manifest
".i18n-build/manifests/${LOCALE}.json"
done
+ VERIFY
- name: Check file sizes for asf-site
# GitHub rejects a push holding any file over 100 MiB. Checking
@@ -143,6 +167,7 @@ jobs:
# cp *.pdf ./build/assets/files/
# ls ./build/assets/files/
- name: Upload the site artifact for the isolated deployment job
+ id: site-artifact
uses:
actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: site-build-${{ github.run_id }}-${{ github.run_attempt }}
@@ -152,6 +177,27 @@ jobs:
if-no-files-found: error
retention-days: 1
+ - name: Report publishing measurements
+ if: always() && steps.metrics-start.outcome == 'success'
+ env:
+ GH_TOKEN: ${{ github.token }}
+ SITE_ARTIFACT_ID: ${{ steps.site-artifact.outputs.artifact-id }}
+ SITE_ARTIFACT_DIGEST: ${{
steps.site-artifact.outputs.artifact-digest }}
+ run: python3 .publish-tools/.github/scripts/publish_metrics.py
finish --budget 180
+
+ - name: Upload publishing measurements
+ if: always() && steps.metrics-start.outcome == 'success'
+ uses:
actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+ with:
+ name: publish-build-measurements-${{ github.run_id }}-${{
github.run_attempt }}
+ include-hidden-files: true
+ path: |
+ .i18n-build/publish-metrics/
+ .i18n-build/build-result.json
+ .i18n-build/publish-result.json
+ if-no-files-found: error
+ retention-days: 14
+
deploy:
needs: [source, build]
runs-on: ubuntu-latest
@@ -159,6 +205,7 @@ jobs:
environment: Production
permissions:
contents: write
+ actions: read
env:
DORIS_COMMIT: ${{ needs.source.outputs.source_sha }}
steps:
@@ -168,6 +215,11 @@ jobs:
ref: ${{ github.sha }}
persist-credentials: false
submodules: recursive
+ - name: Start publishing measurements
+ id: metrics-start
+ env:
+ DEPLOY_SOURCE_SHA: ${{ needs.source.outputs.source_sha }}
+ run: python3 .github/scripts/publish_metrics.py start --job deploy
- uses:
actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
with:
name: site-build-${{ github.run_id }}-${{ github.run_attempt }}
@@ -226,3 +278,19 @@ jobs:
user_name: 'github-actions[bot]'
user_email: 'github-actions[bot]@users.noreply.github.com'
commit_message: 'Automated deployment with doris master@${{
env.DORIS_COMMIT }}'
+
+ - name: Report publishing measurements
+ if: always() && steps.metrics-start.outcome == 'success'
+ env:
+ GH_TOKEN: ${{ github.token }}
+ run: python3 .github/scripts/publish_metrics.py finish --budget 180
+
+ - name: Upload publishing measurements
+ if: always() && steps.metrics-start.outcome == 'success'
+ uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
# v4
+ with:
+ name: publish-deploy-measurements-${{ github.run_id }}-${{
github.run_attempt }}
+ include-hidden-files: true
+ path: .i18n-build/publish-metrics/
+ if-no-files-found: error
+ retention-days: 14
diff --git a/.github/workflows/manual-deploy-website.yml
b/.github/workflows/manual-deploy-website.yml
index d0051449e4cd..326b876cc4af 100644
--- a/.github/workflows/manual-deploy-website.yml
+++ b/.github/workflows/manual-deploy-website.yml
@@ -38,7 +38,12 @@ jobs:
build:
needs: source
runs-on: ubuntu-latest
- timeout-minutes: 90
+ # Temporary ceiling: validate build + verification + artifact transfer
+ # with 30% margin using the publishing measurement reports.
+ timeout-minutes: 180
+ permissions:
+ contents: read
+ actions: read
env:
DORIS_ENABLE_JA: 'true'
DORIS_ENABLE_KO: 'true'
@@ -53,6 +58,21 @@ jobs:
# shallow clone would date every page to the same commit.
fetch-depth: 0
+ - name: Checkout trusted measurement code
+ uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+ with:
+ ref: ${{ github.sha }}
+ persist-credentials: false
+ path: .publish-tools
+ sparse-checkout: .github/scripts/publish_metrics.py
+ sparse-checkout-cone-mode: false
+
+ - name: Start publishing measurements
+ id: metrics-start
+ env:
+ DEPLOY_SOURCE_SHA: ${{ needs.source.outputs.source_sha }}
+ run: python3 .publish-tools/.github/scripts/publish_metrics.py start
--job build
+
- name: Add Chinese Fonts
run: |
sudo apt-get install ttf-mscorefonts-installer
@@ -83,6 +103,7 @@ jobs:
env:
GITHUB_TOKEN: ${{ github.token }}
run: |
+ /usr/bin/time -v -o .i18n-build/publish-metrics/build-time.txt
bash -e -o pipefail <<'BUILD'
npm install -g yarn
yarn cache clean
export NODE_OPTIONS=--max-old-space-size=8192
@@ -94,16 +115,19 @@ jobs:
cp .asf-site.yaml ./build/.asf.yaml
cp versions.json ./build/
ls build
+ BUILD
- name: Verify localized publishing artifacts
# Publishing builds every enabled locale, so every managed locale
# must be present and pass the strict cross-locale verification.
run: |
+ /usr/bin/time -v -o .i18n-build/publish-metrics/verify-time.txt
bash -e -o pipefail <<'VERIFY'
for LOCALE in $(node scripts/i18n/managed-locales.cjs); do
test -s "build/${LOCALE}/index.html"
node scripts/i18n/verify-static.cjs \
--site build --manifest
".i18n-build/manifests/${LOCALE}.json"
done
+ VERIFY
- name: Check file sizes for asf-site
# GitHub rejects a push holding any file over 100 MiB. Checking
@@ -115,6 +139,7 @@ jobs:
exit 1
fi
- name: Upload the site artifact for the isolated deployment job
+ id: site-artifact
uses:
actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: site-build-${{ github.run_id }}-${{ github.run_attempt }}
@@ -124,6 +149,27 @@ jobs:
if-no-files-found: error
retention-days: 1
+ - name: Report publishing measurements
+ if: always() && steps.metrics-start.outcome == 'success'
+ env:
+ GH_TOKEN: ${{ github.token }}
+ SITE_ARTIFACT_ID: ${{ steps.site-artifact.outputs.artifact-id }}
+ SITE_ARTIFACT_DIGEST: ${{
steps.site-artifact.outputs.artifact-digest }}
+ run: python3 .publish-tools/.github/scripts/publish_metrics.py
finish --budget 180
+
+ - name: Upload publishing measurements
+ if: always() && steps.metrics-start.outcome == 'success'
+ uses:
actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+ with:
+ name: publish-build-measurements-${{ github.run_id }}-${{
github.run_attempt }}
+ include-hidden-files: true
+ path: |
+ .i18n-build/publish-metrics/
+ .i18n-build/build-result.json
+ .i18n-build/publish-result.json
+ if-no-files-found: error
+ retention-days: 14
+
deploy:
needs: [source, build]
runs-on: ubuntu-latest
@@ -131,6 +177,7 @@ jobs:
environment: Production
permissions:
contents: write
+ actions: read
env:
DORIS_COMMIT: ${{ needs.source.outputs.source_sha }}
steps:
@@ -140,6 +187,11 @@ jobs:
ref: ${{ github.sha }}
persist-credentials: false
submodules: recursive
+ - name: Start publishing measurements
+ id: metrics-start
+ env:
+ DEPLOY_SOURCE_SHA: ${{ needs.source.outputs.source_sha }}
+ run: python3 .github/scripts/publish_metrics.py start --job deploy
- uses:
actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
with:
name: site-build-${{ github.run_id }}-${{ github.run_attempt }}
@@ -207,3 +259,19 @@ jobs:
user_name: 'github-actions[bot]'
user_email: 'github-actions[bot]@users.noreply.github.com'
commit_message: 'Automated deployment with doris branch ${{
needs.source.outputs.branch }}@${{ env.DORIS_COMMIT }}'
+
+ - name: Report publishing measurements
+ if: always() && steps.metrics-start.outcome == 'success'
+ env:
+ GH_TOKEN: ${{ github.token }}
+ run: python3 .github/scripts/publish_metrics.py finish --budget 180
+
+ - name: Upload publishing measurements
+ if: always() && steps.metrics-start.outcome == 'success'
+ uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
# v4
+ with:
+ name: publish-deploy-measurements-${{ github.run_id }}-${{
github.run_attempt }}
+ include-hidden-files: true
+ path: .i18n-build/publish-metrics/
+ if-no-files-found: error
+ retention-days: 14
diff --git a/scripts/build-locales.md b/scripts/build-locales.md
index fc27191b02a0..0177078c7325 100644
--- a/scripts/build-locales.md
+++ b/scripts/build-locales.md
@@ -69,3 +69,55 @@ disabled Japanese, failures, source changes and invalid
output prefixes.
`node scripts/i18n/verify-fixture.cjs --browser` exercises the actual native
three-language build and browser routing. Full active-version and OOM evidence
comes from GitHub CI; local website verification uses `DOCS_VERSIONS=current`.
+
+## Publishing job measurements
+
+The cron and manual publishing workflows retain separate
+`publish-build-measurements-<run>-<attempt>` and
+`publish-deploy-measurements-<run>-<attempt>` artifacts for 14 days. These
reports
+stay outside `build/` and are uploaded after failures as well as successes.
+Hard job timeouts or runner loss can prevent this final upload; retain the
+Actions jobs JSON and logs when that happens.
+
+The build job has a temporary 180-minute limit, matching the existing deploy
+limit. This is a budget change, not a measured speedup. Both job summaries show
+elapsed time through reporting and a provisional `ceil(elapsed * 1.3 / 60)`
+budget in minutes. Accept a budget only after a successful complete run,
+including the final telemetry upload and cleanup steps. A failed or incomplete
+run cannot establish a successful publishing baseline.
+
+The reports contain the resolved source SHA separately from the trusted
+workflow/checkout SHA, runner image and hardware context, active versions,
+locale switches, cache policy, per-step timings, output file counts/bytes and
+site artifact archive size/digest. The build and strict verification commands
+also retain GNU `time -v` output, including maximum RSS. Ten-second samples
+capture system memory availability and workspace/`/mnt` disk use for each step.
+These are sampled extrema, not continuous disk peaks or process RSS. Artifact,
+OSS and asf-site process RSS and exact network bytes are not measured; use
+archive/output sizes and the transfer action logs without calling them billed
+network traffic. GNU time reports command RSS, not the sum of every concurrent
+process's memory.
+
+Compare runs only after checking their source SHA (including generated-input
+diff/build stamps), runner image, CPU/memory, versions, locales and cache
+conditions. The publishing workflows retain their uncached install behavior.
+The manual build reads measurement code from the trusted workflow SHA, so an
+older protected source branch does not need to contain the measurement script.
+The reporting token has only the job's existing contents permission plus
+`actions: read`; no publishing secrets enter the build job or reports.
+
+To preserve the final timing and queue evidence for an existing run:
+
+```bash
+gh api repos/apache/doris-website/actions/runs/RUN_ID > run.json
+gh api repos/apache/doris-website/actions/runs/RUN_ID/attempts/ATTEMPT/jobs >
jobs.json
+gh run download RUN_ID --repo apache/doris-website --pattern
'publish-*-measurements-*'
+```
+
+Use `run.json` creation time, job start/end times and the report's resolved
+source SHA together. The workflow run's head SHA can differ from the source
+resolved by the protected-branch job. Separate queue/dependency waits from
+runner execution; preserve failed/cancelled attempts rather than merging them
+into successful timing samples. After matching the uploaded and downloaded
+artifact identities and successful OSS/asf-site steps, record the complete
+build/deploy durations and require about 30% budget margin for each job.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
