This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 2b158c2eff9 [feat](i18n) Track Japanese translation state and isolate
locale builds (#4207)
2b158c2eff9 is described below
commit 2b158c2eff9a3cc12c872ac673698b007223a3ce
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Mon Oct 12 13:49:38 2026 +0800
[feat](i18n) Track Japanese translation state and isolate locale builds
(#4207)
Japanese files currently count as translated solely because they exist,
so old SQL and parameters can remain visible after their source changes.
This adds versioned translation state and a build-time content view:
matching records render Japanese, reviewed prose-only changes retain
Japanese with a stale notice, and technical, unknown or conflicting
changes render the latest original without modifying the committed
translation.
The same decision controls status notices, body language, source update
dates, canonical URLs, noindex and reciprocal language alternates. The
native Docusaurus route parser supplies page identities and permalinks;
orphan targets and unsupported Chinese-only sources are reported. The
first full three-language CI exhausted RAM/swap during Japanese and
recorded kernel and cgroup OOM kills increasing from 0 to 1. Locale
builds now run sequentially in separate Node processes, retaining native
paths and all configured language alternatives. Only after every locale
succeeds does the complete assembly replace the output; source changes
or child failures preserve the old artifact. Cron/manual use the same
entry point. Japanese and automatic translation remain disabled in
production. A configured Chinese source controls translation freshness
and source dates; unavailable or unsafe translations always use the
native English fallback and retain a separate Chinese-original link when
its route exists.
Validation:
- 20 focused state, heading, resource and SSR tests pass.
- Local Dev builds pass with Japanese disabled and enabled; default
English/Chinese page signatures and browser behavior match the T01
baseline.
- Static checks cover 1,778 Dev/content routes, including sitemap,
search exclusion and reciprocal alternates.
- The real three-language fixture verifies 16 routes covering stale SQL,
unknown and approved prose changes, manual target edits, invalid
metadata, source deletion, separate versions/plugins, Chinese source
policy and project-relative MDX imports. Browser checks cover SPA
transitions, back/refresh, source-link query/hash and metadata updates.
The website browser probe verifies 26 representative pages and 32
current/fallback notice checks across desktop/mobile and all eight
themes, with no new errors.
- TypeScript retains the pre-existing 150 distinct diagnostics, with no
new diagnostics.
The [full active-version
CI](https://github.com/apache/doris-website/actions/runs/38196300628)
passes at `b84251532786727c3e00ee1c1d04c476d2850a05`: all 4.x, 3.x, 2.1
and current content and plugins are retained, with no scope filters. The
original Ubuntu runner, Node v20.20.2, 8 GiB heap and swap allocation
remain unchanged. All three child processes exit successfully; global
and cgroup OOM kill counters stay 0 → 0. Per-build system swap peaks at
407.54 MiB, leaving at least 10.60 GiB swap free, versus exhausted
approximately 11 GiB swap in the failed run. The earlier legacy
English/Chinese baseline in the same job peaks at 8.45 GiB swap. The
cgroup swap peak is cumulative across that baseline and must not be
attributed to the isolated build.
Full artifact validation covers 5,874 non-draft Japanese content routes
(5,898 identities, including 24 drafts), all four real translated
samples, 32 browser pages and 32 notice theme/viewport combinations.
English/Chinese content, headings, links, metadata, search suggestions
and page errors match the baseline. The 19 unsupported-source reports
identify existing Chinese-only 3.x/2.1 files without a native English
route; they are reported explicitly. No new page error is introduced.
All other PR checks pass. Local website builds use only Dev. Visual QA
also corrected Japanese message placeholders to match the repository’s
custom version banner.
Repository review checkpoints:
- Goal: T02 supplies independently usable status/fallback/SEO behavior
and fixes the confirmed cross-locale memory accumulation.
- Scope: the locale adapter reuses the pinned native Docusaurus
implementation; no dependency upgrade, larger runner, version split,
model call or public Japanese enablement.
- Information architecture: current, active versions and all four
docs-plugin instances retain native enumeration.
- References: no source docs, slugs, sidebar entries or existing
redirects are moved. Japanese heading IDs follow current English IDs;
native resource prefixes and configured language alternatives remain
intact.
- Frontend/configuration: default English/Chinese behavior is preserved;
notices use semantic theme tokens. Only Japanese routes use the
temporary content view and translation-aware cache snapshot.
- Validation: local website builds use `DOCS_VERSIONS=current`; the
read-only Japanese workflow validates all active versions and memory in
one job with independent sequential locale processes. Existing unrelated
warnings are retained.
- Other issues: no unresolved implementation issue found in self-review;
public launch still depends on T03–T05.
## Versions
- [x] dev
- [x] 4.x
- [x] 3.x
- [x] 2.1 or older (framework behavior only)
## Languages
- [x] Chinese (compatibility)
- [x] English (compatibility)
- Japanese (opt-in)
## Docs Checklist
- [x] Checked by AI
- [x] Test Cases Built
- [x] Updated required version and language counterparts, or explained
why not
- [x] If only one language changed, confirmed whether source/translation
counterparts need sync
The confirmed OOM run is
[38176555239](https://github.com/apache/doris-website/actions/runs/38176555239).
Retained swap was approximately 0.74 GiB after English, 7.4 GiB after
Chinese, and 11 GiB at the Japanese OOM. The isolated-process validation
is
[38196300628](https://github.com/apache/doris-website/actions/runs/38196300628);
all build, static/browser and memory checks pass. No deployment was
triggered. Japanese remains opt-in, and future content growth still
requires memory revalidation.
---------
Co-authored-by: morningman <[email protected]>
---
.github/workflows/cron-deploy-website.yml | 2 +-
.github/workflows/japanese-poc.yml | 29 ++-
.github/workflows/manual-deploy-website.yml | 2 +-
.gitignore | 2 +
docusaurus.config.js | 74 ++++---
i18n/ja/code.json | 6 +-
package.json | 5 +-
scripts/__tests__/build-locales.test.cjs | 82 +++++++
scripts/build-locale.cjs | 32 +++
scripts/build-locales.cjs | 102 +++++++++
scripts/build-locales.md | 44 ++++
scripts/i18n/__tests__/state.test.cjs | 133 ++++++++++++
scripts/i18n/docs-plugin.cjs | 236 +++++++++++++++++++++
scripts/i18n/poc/README.md | 8 +-
scripts/i18n/poc/measure-build.py | 26 ++-
scripts/i18n/poc/verify-static.cjs | 28 ++-
scripts/i18n/state.cjs | 85 ++++++++
scripts/i18n/status-plugin.cjs | 28 +++
scripts/i18n/structure.cjs | 94 ++++++++
scripts/i18n/verify-fixture.cjs | 183 ++++++++++++++++
scripts/i18n/verify-static.cjs | 85 ++++++++
src/theme/DocItem/Content/index.tsx | 5 +-
.../DocItem/Layout/JapaneseFallback.module.css | 6 +
src/theme/DocItem/Layout/TranslationStatus.tsx | 47 ++++
src/theme/DocItem/Layout/index.tsx | 13 +-
src/theme/DocItem/Metadata/index.tsx | 10 +-
.../NavbarItem/LocaleDropdownNavbarItem/index.tsx | 7 +-
src/theme/SiteMetadata/index.tsx | 163 ++++++++++++++
src/utils/translation.ts | 29 +++
translations/README.md | 125 +++++++++++
translations/config.json | 9 +
translations/state/ja/docs.json | 73 +++++++
32 files changed, 1712 insertions(+), 61 deletions(-)
diff --git a/.github/workflows/cron-deploy-website.yml
b/.github/workflows/cron-deploy-website.yml
index 6007b7495df..2f78e9a62e0 100644
--- a/.github/workflows/cron-deploy-website.yml
+++ b/.github/workflows/cron-deploy-website.yml
@@ -78,7 +78,7 @@ jobs:
yarn
node ./scripts/update_github_info.js
yarn key-features:generate
- PWA_SERVICE_WORKER_URL=https://doris.apache.org/sw.js yarn
docusaurus build --locale en --locale zh-CN
+ PWA_SERVICE_WORKER_URL=https://doris.apache.org/sw.js yarn
build:locales
touch build/.dummy
cp .asf-site.yaml ./build/.asf.yaml
cp versions.json ./build/
diff --git a/.github/workflows/japanese-poc.yml
b/.github/workflows/japanese-poc.yml
index 0027ec691d9..2f19d0fa5c3 100644
--- a/.github/workflows/japanese-poc.yml
+++ b/.github/workflows/japanese-poc.yml
@@ -21,7 +21,11 @@ on:
- 'yarn.lock'
- 'scripts/patch-react-ssr.js'
- 'docusaurus.config.js'
- - 'scripts/i18n/poc/**'
+ - 'scripts/i18n/**'
+ - 'scripts/build-locales.cjs'
+ - 'scripts/build-locale.cjs'
+ - 'scripts/__tests__/build-locales.test.cjs'
+ - 'translations/**'
- '.github/workflows/japanese-poc.yml'
# PR code is untrusted. This job has no deployment environment, secrets,
@@ -35,9 +39,9 @@ concurrency:
jobs:
full-build:
- # Keep this expensive pre-merge probe specific to the PoC branch. Later
- # translation PRs retain their normal checks; manual probes stay available.
- if: github.event_name == 'workflow_dispatch' ||
startsWith(github.head_ref, 'feat/japanese-i18n-poc')
+ # Keep the expensive full matrix specific to the staged implementation
+ # branches. General translation-PR scope detection belongs to T04.
+ if: github.event_name == 'workflow_dispatch' ||
startsWith(github.head_ref, 'feat/japanese-i18n-poc') ||
startsWith(github.head_ref, 'feat/japanese-i18n-status')
runs-on: ubuntu-latest
timeout-minutes: 180
env:
@@ -106,11 +110,18 @@ jobs:
yarn cache clean
yarn --frozen-lockfile
yarn key-features:generate
- node --test scripts/i18n/poc/config.test.cjs
scripts/i18n/poc/react-ssr.test.cjs
- - name: Full sequential English, Chinese and Japanese build
+ node --test scripts/i18n/poc/*.test.cjs
scripts/i18n/__tests__/*.test.cjs scripts/__tests__/build-locales.test.cjs
+ - name: Validate translation states in a real fixture site
+ working-directory: poc-source
+ env:
+ NODE_PATH: ${{ runner.temp }}/poc-browser/node_modules
+ run: |
+ set -o pipefail
+ node scripts/i18n/verify-fixture.cjs --browser | tee
../evidence/translation-fixture.log
+ - name: Full sequential build in independent locale processes
run: |
python3 poc-source/scripts/i18n/poc/measure-build.py \
- --source poc-source --output evidence/poc --japanese
+ --source poc-source --output evidence/poc --japanese
--separate-locales
- name: Validate PoC static artifact and existing locales
env:
NODE_PATH: ${{ runner.temp }}/poc-browser/node_modules
@@ -118,6 +129,10 @@ jobs:
node poc-source/scripts/i18n/poc/verify-static.cjs \
--site evidence/poc/site --output evidence/poc/browser --expect-ja
\
--baseline evidence/baseline/browser/signatures.json
+ node poc-source/scripts/i18n/verify-static.cjs \
+ --site evidence/poc/site --manifest
poc-source/.i18n-build/manifests/ja.json \
+ --output evidence/poc/translation-status.json
+ cp -R poc-source/.i18n-build/manifests evidence/poc/manifests
- name: Upload evidence even when validation fails
if: always()
uses: actions/upload-artifact@v4
diff --git a/.github/workflows/manual-deploy-website.yml
b/.github/workflows/manual-deploy-website.yml
index 0bc5d14e748..59d46636614 100644
--- a/.github/workflows/manual-deploy-website.yml
+++ b/.github/workflows/manual-deploy-website.yml
@@ -55,7 +55,7 @@ jobs:
yarn
node ./scripts/update_github_info.js
yarn key-features:generate
- PWA_SERVICE_WORKER_URL=https://doris.apache.org/sw.js yarn
docusaurus build --locale en --locale zh-CN
+ PWA_SERVICE_WORKER_URL=https://doris.apache.org/sw.js yarn
build:locales
touch build/.dummy
cp .asf-site.yaml ./build/.asf.yaml
cp versions.json ./build/
diff --git a/.gitignore b/.gitignore
index 61baea771f4..ef5a7409a0a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -7,6 +7,8 @@
# Generated files
.docusaurus
.cache-loader
+.i18n-build/
+/.doris-build-*/
/src/generated/key-features.ts
# Misc
diff --git a/docusaurus.config.js b/docusaurus.config.js
index 20bea439fea..fb840d5c459 100644
--- a/docusaurus.config.js
+++ b/docusaurus.config.js
@@ -7,6 +7,8 @@ const { ssrTemplate } = require('./config/ssrTemplate');
const customDocusaurusPlugin = require('./config/custom-docusaurus-plugin');
const REDIRECTS_4X = require('./config/redirects-4.x.json');
const path = require('path');
+const translationDocsPlugin = require('./scripts/i18n/docs-plugin.cjs');
+const translationStatusPlugin = require('./scripts/i18n/status-plugin.cjs');
const ENABLE_JA = process.env.DORIS_ENABLE_JA === 'true';
const BRAND_THEME_BOOTSTRAP = `(function () {
@@ -219,6 +221,38 @@ function getLatestVersion() {
return VERSIONS.includes(DEFAULT_VERSION) ? DEFAULT_VERSION : VERSIONS[0];
}
+const MAIN_DOCS_OPTIONS = {
+ includeCurrentVersion: true,
+ sidebarPath: require.resolve('./sidebars.ts'),
+ ...(ONLY_VERSIONS && { onlyIncludeVersions: ONLY_VERSIONS }),
+ // When filtering versions, lastVersion must be in the
+ // included list. Fall back to the first included version.
+ lastVersion: ONLY_VERSIONS && !ONLY_VERSIONS.includes(getLatestVersion())
+ ? ONLY_VERSIONS[0]
+ : getLatestVersion(),
+ versions: getDocsVersions(),
+ // editUrl: ({ locale, versionDocsDirPath, docPath }) => {
+ // return
`https://github.com/apache/doris-website/edit/master/docs/${locale}/docs/${docPath}`;
+ // // if (versionDocsDirPath === 'versioned_docs/version-dev') {
+ // // return
`https://github.com/apache/doris-website/edit/master/docs/${locale}/docs/${docPath}`;
+ // // }
+ // },
+ showLastUpdateAuthor: false,
+ // Date comes from the injected `last_update` front matter
+ // (markdown.parseFrontMatter). The DocItem layout limits
+ // which versions actually display it (Dev + latest stable).
+ showLastUpdateTime: true,
+ remarkPlugins: [markdownBoldPlugin, require('remark-math')],
+ rehypePlugins: [
+ [
+ require('rehype-katex'),
+ {
+ strict: process.env.CI === 'true' ||
process.env.GITHUB_ACTIONS === 'true' ? false : 'warn',
+ }
+ ]
+ ]
+};
+
/** @type {import('@docusaurus/types').Config} */
const config = {
title: 'Apache Doris',
@@ -322,12 +356,14 @@ const config = {
experimental_faster: true,
},
plugins: [
+ SKIP_DOCS ? null : [translationDocsPlugin, MAIN_DOCS_OPTIONS],
+ ENABLE_JA ? translationStatusPlugin : null,
'docusaurus-plugin-sass',
'docusaurus-plugin-matomo',
// Use custom blog plugin
versionsPlugin,
SKIP_COURSE ? null : [
- 'content-docs',
+ translationDocsPlugin,
/** @type {import('@docusaurus/plugin-content-docs').Options} */
({
id: 'course',
@@ -338,7 +374,7 @@ const config = {
}),
],
SKIP_COMMUNITY ? null : [
- 'content-docs',
+ translationDocsPlugin,
/** @type {import('@docusaurus/plugin-content-docs').Options} */
({
id: 'community',
@@ -351,7 +387,7 @@ const config = {
}),
],
SKIP_RELEASES ? null : [
- 'content-docs',
+ translationDocsPlugin,
/** @type {import('@docusaurus/plugin-content-docs').Options} */
({
id: 'releases',
@@ -443,37 +479,7 @@ const config = {
'classic',
/** @type {import('@docusaurus/preset-classic').Options} */
({
- docs: SKIP_DOCS ? false : {
- includeCurrentVersion: true,
- sidebarPath: require.resolve('./sidebars.ts'),
- ...(ONLY_VERSIONS && { onlyIncludeVersions: ONLY_VERSIONS
}),
- // When filtering versions, lastVersion must be in the
- // included list. Fall back to the first included version.
- lastVersion: ONLY_VERSIONS &&
!ONLY_VERSIONS.includes(getLatestVersion())
- ? ONLY_VERSIONS[0]
- : getLatestVersion(),
- versions: getDocsVersions(),
- // editUrl: ({ locale, versionDocsDirPath, docPath }) => {
- // return
`https://github.com/apache/doris-website/edit/master/docs/${locale}/docs/${docPath}`;
- // // if (versionDocsDirPath ===
'versioned_docs/version-dev') {
- // // return
`https://github.com/apache/doris-website/edit/master/docs/${locale}/docs/${docPath}`;
- // // }
- // },
- showLastUpdateAuthor: false,
- // Date comes from the injected `last_update` front matter
- // (markdown.parseFrontMatter). The DocItem layout limits
- // which versions actually display it (Dev + latest
stable).
- showLastUpdateTime: true,
- remarkPlugins: [markdownBoldPlugin,
require('remark-math')],
- rehypePlugins: [
- [
- require('rehype-katex'),
- {
- strict: process.env.CI === 'true' ||
process.env.GITHUB_ACTIONS === 'true' ? false : 'warn',
- }
- ]
- ]
- },
+ docs: false, // The native docs plugin is wrapped below for
translation state.
blog: SKIP_BLOG ? false : {
blogTitle: 'Apache Doris - Blog | Latest news and events ',
blogDescription:
diff --git a/i18n/ja/code.json b/i18n/ja/code.json
index b6d9a8ec002..fbb01b88a85 100644
--- a/i18n/ja/code.json
+++ b/i18n/ja/code.json
@@ -802,18 +802,18 @@
"description": "The link label to edit the current page"
},
"theme.docs.versions.unreleasedVersionLabel": {
- "message": "これはリリース前のバージョン{versionLabel}の{siteTitle}のドキュメントです。",
+ "message": "このドキュメントは Apache Doris の{unreleased}バージョン向けです。",
"description": "The label used to tell the user that he's browsing an
unreleased doc version"
},
"theme.docs.versions.unreleasedText": {
- "message": "unreleased"
+ "message": "開発中の"
},
"theme.docs.versions.unmaintainedVersionLabel": {
"message": "これはバージョン{versionLabel}の{siteTitle}のドキュメントで現在はメンテナンスされていません",
"description": "The label used to tell the user that he's browsing an
unmaintained doc version"
},
"theme.docs.versions.latestVersionSuggestionLabel": {
- "message": "最新のドキュメントは{latestVersionLink} ({versionLabel}) を見てください",
+ "message": "利用方法は、バージョン {v4xLink} / {v3xLink} / {v21Link}
の正式ドキュメントを参照してください。",
"description": "The label used to tell the user to check the latest
version"
},
"theme.NotFound.title": {
diff --git a/package.json b/package.json
index b3858229ca7..e5e42a44cd9 100644
--- a/package.json
+++ b/package.json
@@ -8,9 +8,10 @@
"start:zh-CN": "docusaurus start --locale zh-CN",
"postinstall": "node scripts/patch-search-tokenize.js && node
scripts/patch-react-ssr.js",
"prestart": "yarn key-features:generate",
- "build": "PWA_SERVICE_WORKER_URL=https://doris.apache.org/sw.js
docusaurus build",
+ "build": "PWA_SERVICE_WORKER_URL=https://doris.apache.org/sw.js node
scripts/build-locales.cjs",
+ "build:locales": "node scripts/build-locales.cjs",
"prebuild": "yarn key-features:generate",
- "build:version": "docusaurus set-versions && docusaurus build",
+ "build:version": "docusaurus set-versions && node
scripts/build-locales.cjs",
"prebuild:version": "yarn key-features:generate",
"pretypecheck": "yarn key-features:generate",
"swizzle": "docusaurus swizzle",
diff --git a/scripts/__tests__/build-locales.test.cjs
b/scripts/__tests__/build-locales.test.cjs
new file mode 100644
index 00000000000..e6a5438a63f
--- /dev/null
+++ b/scripts/__tests__/build-locales.test.cjs
@@ -0,0 +1,82 @@
+const test = require('node:test');
+const assert = require('node:assert/strict');
+const fs = require('node:fs');
+const os = require('node:os');
+const path = require('node:path');
+const {buildLocales} = require('../build-locales.cjs');
+
+function fixture(t, locales = ['en','zh-CN','ja']) {
+ const siteDir =
fs.mkdtempSync(path.join(os.tmpdir(),'doris-locale-build-'));
+ t.after(() => fs.rmSync(siteDir,{recursive:true,force:true}));
+ const outDir = path.join(siteDir,'build');
+ fs.mkdirSync(outDir);
+ fs.writeFileSync(path.join(outDir,'previous.txt'),'previous complete
site');
+ const calls = [];
+ const stamp = () => ({sha:'same-sha',trackedDiffHash:'same-inputs'});
+ const run = async args => {
+ calls.push(args);
+ if (args[0] === 'plan') {
+
fs.writeFileSync(args[3],JSON.stringify({defaultLocale:'en',locales:locales.map(locale
=> ({locale,
+
directory:locale==='en'?'.':locale,baseUrl:locale==='en'?'/':`/${locale}/`}))}));
+ } else {
+ const directory = path.join(args[2],args[3]==='en'?'.':args[3]);
+ fs.mkdirSync(directory,{recursive:true});
+ fs.writeFileSync(path.join(directory,'index.html'),args[3]);
+ fs.writeFileSync(path.join(directory,'.htaccess'),`${args[3]}
controls`);
+ fs.writeFileSync(path.join(directory,'sitemap.xml'),`${args[3]}
sitemap`);
+ }
+ return {pid:100+calls.length};
+ };
+ return {siteDir,outDir,run,stamp,calls};
+}
+
+test('ordered locale assembly retains root controls and exactly one localized
prefix', async t => {
+ const data = fixture(t);
+ const report = await buildLocales(data);
+ assert.deepEqual(report.locales.map(item =>
item.locale),['en','zh-CN','ja']);
+ assert.equal(new Set(report.locales.map(item => item.pid)).size,3);
+
assert.equal(fs.readFileSync(path.join(data.outDir,'.htaccess'),'utf8'),'en
controls');
+
assert.equal(fs.readFileSync(path.join(data.outDir,'sitemap.xml'),'utf8'),'en
sitemap');
+
assert.equal(fs.readFileSync(path.join(data.outDir,'ja/index.html'),'utf8'),'ja');
+ assert(!fs.existsSync(path.join(data.outDir,'ja/ja')));
+ assert(!fs.existsSync(path.join(data.outDir,'previous.txt')));
+ assert.equal(report.success,true);
+});
+
+test('disabled Japanese stays absent from the configured build plan', async t
=> {
+ const data = fixture(t,['en','zh-CN']);
+ await buildLocales(data);
+ assert.deepEqual(data.calls.filter(args => args[0]==='build').map(args =>
args[3]),['en','zh-CN']);
+ assert(!fs.existsSync(path.join(data.outDir,'ja')));
+});
+
+test('a failed locale preserves the previous complete artifact and stops
subsequent builds', async t => {
+ const data = fixture(t);
+ const run = async args => {
+ if (args[3] === 'zh-CN') throw new Error('locale failed');
+ return data.run(args);
+ };
+ await assert.rejects(buildLocales({...data,run}),/locale failed/);
+
assert.equal(fs.readFileSync(path.join(data.outDir,'previous.txt'),'utf8'),'previous
complete site');
+ assert(!fs.existsSync(path.join(data.outDir,'index.html')));
+ assert(!data.calls.some(args => args[3]==='ja'));
+
assert.equal(JSON.parse(fs.readFileSync(path.join(data.siteDir,'.i18n-build/build-result.json'),'utf8')).success,false);
+});
+
+test('a source change cannot publish mixed locale artifacts', async t => {
+ const data = fixture(t);
+ let modified = false;
+ const run = async args => {
+ const result = await data.run(args);
+ if (args[3] === 'en') modified = true;
+ return result;
+ };
+ await
assert.rejects(buildLocales({...data,run,stamp:()=>({sha:'same-sha',trackedDiffHash:modified?'new-inputs':'same-inputs'})}),/Source
changed/);
+
assert.equal(fs.readFileSync(path.join(data.outDir,'previous.txt'),'utf8'),'previous
complete site');
+});
+
+test('an unsafe locale path is rejected before building it', async t => {
+ const data = fixture(t,['en','../escape']);
+ await assert.rejects(buildLocales(data),/Unsafe locale/);
+
assert.equal(fs.readFileSync(path.join(data.outDir,'previous.txt'),'utf8'),'previous
complete site');
+});
diff --git a/scripts/build-locale.cjs b/scripts/build-locale.cjs
new file mode 100644
index 00000000000..2fa37bf65c3
--- /dev/null
+++ b/scripts/build-locale.cjs
@@ -0,0 +1,32 @@
+// One disposable Docusaurus process. Pin this adapter to the installed
version:
+// unlike `build --locale ja`, buildLocale with no single-locale CLI override
+// retains the native /ja/ base URL, output path and all language alternatives.
+const fs = require('node:fs');
+const path = require('node:path');
+const {createRequire} = require('node:module');
+
+async function main() {
+ const [mode, siteDir, outDir, value] = process.argv.slice(2);
+ const fromSite = createRequire(path.join(siteDir, 'package.json'));
+ const version = fromSite('@docusaurus/core/package.json').version;
+ if (version !== '3.6.3') throw new Error(`Revalidate the locale build
adapter for Docusaurus ${version}`);
+ process.env.NODE_ENV = 'production';
+ process.env.BABEL_ENV = 'production';
+ delete process.env.DOCUSAURUS_CURRENT_LOCALE;
+ if (mode === 'plan') {
+ const {loadContext} = fromSite('@docusaurus/core/lib/server/site.js');
+ const {localizePath} = fromSite('@docusaurus/utils');
+ const context = await loadContext({siteDir, outDir});
+ const {i18n, siteConfig} = context;
+ const locales = [i18n.defaultLocale, ...i18n.locales.filter(locale =>
locale !== i18n.defaultLocale)];
+ fs.writeFileSync(value, JSON.stringify({version,
defaultLocale:i18n.defaultLocale,
+ locales:locales.map(locale => ({locale,
+ directory:locale === i18n.defaultLocale ? '.' : locale,
+ baseUrl:localizePath({path:siteConfig.baseUrl, pathType:'url',
i18n:{...i18n,currentLocale:locale}}),
+ }))}, null, 2) + '\n');
+ } else if (mode === 'build') {
+ const {buildLocale} =
fromSite('@docusaurus/core/lib/commands/build/buildLocale.js');
+ await buildLocale({siteDir, locale:value, cliOptions:{outDir}});
+ } else throw new Error(`Unknown locale build mode: ${mode}`);
+}
+main().catch(error => {console.error(error); process.exitCode = 1;});
diff --git a/scripts/build-locales.cjs b/scripts/build-locales.cjs
new file mode 100644
index 00000000000..ca818998a9b
--- /dev/null
+++ b/scripts/build-locales.cjs
@@ -0,0 +1,102 @@
+const fs = require('node:fs');
+const path = require('node:path');
+const {createHash} = require('node:crypto');
+const {spawn, execFileSync} = require('node:child_process');
+
+function sourceStamp(siteDir) {
+ return {
+
sha:execFileSync('git',['rev-parse','HEAD'],{cwd:siteDir,encoding:'utf8'}).trim(),
+
trackedDiffHash:createHash('sha256').update(execFileSync('git',['diff','HEAD','--binary'],{cwd:siteDir,maxBuffer:16*1024*1024})).digest('hex'),
+ };
+}
+
+async function buildLocales({siteDir = process.cwd(), outDir = 'build', run,
stamp = sourceStamp}) {
+ siteDir = path.resolve(siteDir);
+ outDir = path.resolve(siteDir, outDir);
+ if (outDir === siteDir || siteDir.startsWith(outDir + path.sep)) throw new
Error('Build output must not contain the source checkout');
+ fs.mkdirSync(path.dirname(outDir), {recursive:true});
+ const stage = fs.mkdtempSync(path.join(path.dirname(outDir),
'.doris-build-'));
+ const initialStamp = stamp(siteDir);
+ const verifyStamp = () => {
+ if (JSON.stringify(stamp(siteDir)) !== JSON.stringify(initialStamp))
throw new Error('Source changed between locale builds; refusing mixed
artifacts');
+ };
+ const report = {schemaVersion:1, source:initialStamp,
mode:'separate-processes', locales:[]};
+ const reportDir = path.join(siteDir, '.i18n-build');
+ fs.mkdirSync(reportDir, {recursive:true});
+ const reportPath = path.join(reportDir, 'build-result.json');
+ const execute = run ?? ((args) => new Promise((resolve, reject) => {
+ const child =
spawn(process.execPath,[path.join(__dirname,'build-locale.cjs'),
...args],{cwd:siteDir,stdio:'inherit'});
+ console.log(JSON.stringify({event:'locale_process_start',
mode:args[0], locale:args[0] === 'build' ? args[3] : null, pid:child.pid}));
+ const stop = signal => child.kill(signal);
+ const term = () => stop('SIGTERM');
+ const interrupt = () => stop('SIGINT');
+ process.once('SIGTERM', term);
+ process.once('SIGINT', interrupt);
+ child.once('error', reject);
+ child.once('exit', (code, signal) => {
+ process.removeListener('SIGTERM', term);
+ process.removeListener('SIGINT', interrupt);
+ console.log(JSON.stringify({event:'locale_process_end',
mode:args[0], locale:args[0] === 'build' ? args[3] : null, pid:child.pid, code,
signal}));
+ if (code !== 0) reject(new Error(`Locale ${args[3]} failed
(${signal ?? code}); output was not published`));
+ else resolve({pid:child.pid});
+ });
+ }));
+ try {
+ const planFile = path.join(stage, 'plan.json');
+ await execute(['plan',siteDir,path.join(stage,'plan-site'),planFile]);
+ const plan = JSON.parse(fs.readFileSync(planFile,'utf8'));
+ if (!plan.locales.length || plan.locales[0].locale !==
plan.defaultLocale
+ || new Set(plan.locales.map(item => item.locale)).size !==
plan.locales.length) throw new Error('Default locale must be first in a unique
locale plan');
+ for (const item of plan.locales) {
+ if (!/^[a-zA-Z0-9-]+$/.test(item.locale) || item.directory !==
(item.locale === plan.defaultLocale ? '.' : item.locale)) {
+ throw new Error('Unsafe locale output directory');
+ }
+ verifyStamp();
+ const localeOutput = path.join(stage, item.locale, 'site');
+ const started = Date.now();
+ const processInfo = await
execute(['build',siteDir,localeOutput,item.locale]);
+ const resultDir = path.join(localeOutput,item.directory);
+ if (!fs.existsSync(path.join(resultDir,'index.html'))) throw new
Error(`Missing locale homepage: ${item.locale}`);
+ report.locales.push({...item, ...processInfo,
elapsedMs:Date.now()-started, resultDir});
+ }
+ verifyStamp();
+ const assembled = path.join(stage,'assembled');
+ fs.renameSync(report.locales[0].resultDir, assembled);
+ for (const item of report.locales.slice(1)) {
+ const destination = path.join(assembled,item.directory);
+ fs.rmSync(destination,{recursive:true,force:true});
+ fs.renameSync(item.resultDir,destination);
+ }
+ verifyStamp();
+ // Prepare every locale before touching the old complete artifact. Root
+ // controls come from the default locale; localized files stay beneath
+ // their own prefix. A failed locale cannot leave a half-built output.
+ const backup = path.join(stage,'previous-output');
+ const hadOutput = fs.existsSync(outDir);
+ if (hadOutput) fs.renameSync(outDir,backup);
+ try { fs.renameSync(assembled,outDir); }
+ catch (error) {
+ if (hadOutput) fs.renameSync(backup,outDir);
+ throw error;
+ }
+ report.success = true;
+ report.output = outDir;
+ for (const item of report.locales) item.output =
path.join(outDir,item.directory);
+ return report;
+ } catch (error) {
+ report.success = false;
+ report.error = error.message;
+ throw error;
+ } finally {
+ fs.writeFileSync(reportPath,JSON.stringify(report,null,2)+'\n');
+ fs.rmSync(stage,{recursive:true,force:true});
+ }
+}
+
+if (require.main === module) {
+ const args = process.argv.slice(2);
+ const option = (name, fallback) => args.includes(name) ?
args[args.indexOf(name)+1] : fallback;
+ buildLocales({siteDir:option('--site-dir',process.cwd()),
outDir:option('--out-dir','build')})
+ .catch(error => {console.error(error);process.exitCode = 1;});
+}
+module.exports = {buildLocales};
diff --git a/scripts/build-locales.md b/scripts/build-locales.md
new file mode 100644
index 00000000000..82348c215a7
--- /dev/null
+++ b/scripts/build-locales.md
@@ -0,0 +1,44 @@
+# Sequential locale builds
+
+Use `yarn build` or `yarn build:locales` for complete website builds. The
launcher
+reads the actual enabled locales from the site configuration and builds each in
+an independent Node process, sequentially. Japanese remains disabled unless
+`DORIS_ENABLE_JA=true`; no translation or deployment service is called.
+
+```bash
+# Local development: only the actual Dev docs version.
+DOCS_VERSIONS=current DORIS_ASSET_ORIGIN='' yarn build:locales
+DORIS_ENABLE_JA=true DOCS_VERSIONS=current DORIS_ASSET_ORIGIN='' \
+ yarn build:locales --out-dir .i18n-build/preview
+```
+
+The adapter calls the installed Docusaurus 3.6.3 `buildLocale` implementation
+without the single-locale CLI path override. Every child therefore retains all
+configured language alternatives, its native base URL (`/`, `/zh-CN/`, `/ja/`)
+and its localized output directory. Moving a root-path single-locale build into
+`ja/` would produce incorrect resource URLs and is not used.
+
+The default locale is built first. All child outputs are prepared in one
+adjacent temporary directory. After every child succeeds, the launcher moves
+those outputs into the complete tree: root control files belong to the default
+locale, and later locales only populate their own prefix. A failed child or a
+changed Git commit/tracked source diff leaves the previous complete output
+untouched. Cron and manual deployment upload only after this command succeeds.
+Assembly uses renames, avoiding a second full copy of all generated site files.
+
+The source stamp, per-locale PID, base URL and timings are recorded in
+`.i18n-build/build-result.json`. The memory observer can run the same entry
point
+with `--separate-locales` and records system/cgroup OOM counters, memory and
swap
+along with those process boundaries. A whole-job OOM may kill the runner before
+artifact upload; periodic JSON log records preserve the diagnosis.
+
+This adapter is pinned to Docusaurus 3.6.3. Revalidate its internal build API,
+paths and artifact assembly before upgrading. Simultaneous builds in the same
+checkout are unsupported because Docusaurus shares `.docusaurus`; use separate
+checkouts.
+
+`node --test scripts/__tests__/build-locales.test.cjs` verifies assembly,
+disabled Japanese, failures, source changes and invalid output prefixes.
+`node scripts/i18n/verify-fixture.cjs --browser` exercises the actual native
+three-language build and browser routing. Full active-version and OOM evidence
+comes from GitHub CI; local website verification uses `DOCS_VERSIONS=current`.
diff --git a/scripts/i18n/__tests__/state.test.cjs
b/scripts/i18n/__tests__/state.test.cjs
new file mode 100644
index 00000000000..66fec3d4572
--- /dev/null
+++ b/scripts/i18n/__tests__/state.test.cjs
@@ -0,0 +1,133 @@
+const test = require('node:test');
+const assert = require('node:assert/strict');
+const {hash, identity, decide} = require('../state.cjs');
+const {structure, alignHeadings} = require('../structure.cjs');
+
+test('Japanese version messages match the custom banner interpolation
contract', () => {
+ const dictionary = require('../../../i18n/ja/code.json');
+ const variables = key =>
[...dictionary[key].message.matchAll(/\{(\w+)\}/g)].map(match =>
match[1]).sort();
+ assert.deepEqual(variables('theme.docs.versions.unreleasedVersionLabel'),
['unreleased']);
+
assert.deepEqual(variables('theme.docs.versions.latestVersionSuggestionLabel'),
['v21Link', 'v3xLink', 'v4xLink']);
+});
+
+const page = {pluginId: 'default', version: 'current', docId: 'intro', locale:
'ja', sourceLocale: 'en',
+ sourcePath: 'docs/intro.md', targetPath:
'i18n/ja/docusaurus-plugin-content-docs/current/intro.md'};
+const original = '## Usage\n\nUse Doris.\n\n```sql\nSELECT 1;\n```\n';
+const target = '## 使用方法 {#usage}\n\nDoris を使用します。\n\n```sql\nSELECT 1;\n```\n';
+async function setup() {
+ const technicalHash = (await structure(original, '.md')).technicalHash;
+ const record = {...page, sourceHash: hash(original), targetHash:
hash(target), sourceTechnicalHash: technicalHash,
+ sourceCommit: 'a'.repeat(40), reviewStatus: 'machine', ownership:
'machine',
+ translatedAt: '2026-10-11T19:57:11Z', glossaryVersion: '0'};
+ return {page, source: original, target, record, technicalHash};
+}
+
+test('missing and unrecorded files cannot claim to be current Japanese
translations', async () => {
+ const data = await setup();
+ assert.equal(decide({...data, target: undefined}).status, 'untranslated');
+ const state = decide({...data, record: undefined});
+ assert.equal(state.status, 'unverified');
+ assert.equal(state.contentLocale, 'en');
+ assert.equal(state.indexable, false);
+ assert.equal(decide({...data, page:{...data.page, sourceLocale:'zh-CN'},
target:undefined}).contentLocale, 'en');
+});
+
+test('freshness does not change machine/human review identity or ownership',
async () => {
+ const data = await setup();
+ assert.equal(decide(data).status, 'current');
+ for (const reviewStatus of ['machine', 'human', 'unverified']) {
+ const state = decide({...data, record: {...data.record, reviewStatus,
ownership: 'human'}});
+ assert.equal(state.status, 'current');
+ assert.equal(state.reviewStatus, reviewStatus);
+ assert.equal(state.useTarget, true);
+ assert.equal(state.indexable, reviewStatus !== 'unverified');
+ }
+});
+
+test('unclassified prose updates select the new English source', async () => {
+ const data = await setup();
+ const source = original.replace('Use Doris.', 'Try Doris.');
+ const state = decide({...data, source, technicalHash: (await
structure(source, '.md')).technicalHash});
+ assert.equal(state.status, 'stale_unknown');
+ assert.equal(state.useTarget, false);
+});
+
+test('a prose approval must bind both source revisions and the current
target', async () => {
+ const data = await setup();
+ const source = original.replace('Use Doris.', 'Try Doris.');
+ const safeSourceChange = {kind: 'prose', fromSourceHash: hash(original),
toSourceHash: hash(source),
+ targetHash: hash(target), reviewedBy: 'reviewer', reviewedAt:
'2026-10-11T20:00:00Z'};
+ const args = {...data, source, technicalHash: (await structure(source,
'.md')).technicalHash,
+ record: {...data.record, safeSourceChange}};
+ const safe = decide(args);
+ assert.equal(safe.status, 'stale_prose');
+ assert.equal(safe.useTarget, true);
+ assert.equal(safe.indexable, false);
+ for (const [key, value] of Object.entries({fromSourceHash: 'old',
toSourceHash: 'other', targetHash: 'changed', reviewedBy: '', reviewedAt:
'invalid'})) {
+ assert.equal(decide({...args, record: {...args.record,
safeSourceChange: {...safeSourceChange, [key]: value}}}).status,
'stale_unknown');
+ }
+});
+
+test('SQL, parameter values, links, headings and MDX attributes invalidate
prose exceptions', async () => {
+ const data = await setup();
+ for (const source of [original.replace('SELECT 1', 'SELECT 2'), original +
'\nTimeout: 10\n',
+ original + '\n[link](https://example.com)\n',
original.replace('Usage', 'Installation')]) {
+ const safeSourceChange = {kind: 'prose', fromSourceHash:
hash(original), toSourceHash: hash(source),
+ targetHash: hash(target), reviewedBy: 'reviewer', reviewedAt:
'2026-10-11T20:00:00Z'};
+ const state = decide({...data, source, technicalHash: (await
structure(source, '.md')).technicalHash,
+ record: {...data.record, safeSourceChange}});
+ assert.equal(state.status, 'stale_technical');
+ assert.equal(state.contentLocale, 'en');
+ }
+ const mdx = original + '\n<Widget timeout={10} />\n';
+ assert.notEqual((await structure(mdx, '.mdx')).technicalHash,
+ (await structure(mdx.replace('timeout={10}', 'timeout={20}'),
'.mdx')).technicalHash);
+});
+
+test('manual target changes and record identity/path conflicts select English
without overwriting the target', async () => {
+ const data = await setup();
+ assert.equal(decide({...data, target: target + '\n人による変更\n'}).status,
'target_modified');
+ for (const record of [{...data.record, version: '4.x'}, {...data.record,
pluginId: 'community'},
+ {...data.record, sourcePath: '../outside.md'}, {...data.record,
sourceLocale: 'zh-CN'},
+ {...data.record, translatedAt: 'not a date'}, {...data.record,
reviewStatus: 'approved'}]) {
+ assert.equal(decide({...data, record}).status, 'state_conflict');
+ }
+ assert.notEqual(identity(page), identity({...page, version: '4.x'}));
+ assert.notEqual(identity(page), identity({...page, pluginId:
'community'}));
+});
+
+test('source hashing includes SQL, parameters, MDX, metadata and whitespace,
normalizing only CRLF', () => {
+ assert.equal(hash(original), hash(original.replace(/\n/g, '\r\n')));
+ assert.notEqual(hash(original), hash(original.replace('SELECT 1', 'SELECT
2')));
+ assert.notEqual(hash(original), hash(original + '\n'));
+});
+
+test('heading alignment follows native duplicate IDs, explicit IDs, setext and
fenced code', async () => {
+ const source = '---\ntitle: Heading\n---\n## Usage\n## Usage\n## Fixed
{#CaseID}\n\nSetext\n------\n\n```md\n## Not a heading\n```\n';
+ const japanese = '---\ntitle: 見出し\n---\n## 使用方法\n## 使用方法\n##
固定\n\nセクション\n------\n\n```md\n## Not a heading\n```\n';
+ const sourceStructure = await structure(source, '.md');
+ assert.deepEqual(sourceStructure.headings.map(h => h.id), ['usage',
'usage-1', 'CaseID', 'setext']);
+ const aligned = await alignHeadings(sourceStructure, japanese, '.md');
+ assert.deepEqual((await structure(aligned, '.md')).headings.map(h =>
h.id), ['usage', 'usage-1', 'CaseID', 'setext']);
+ assert(aligned.startsWith('---\ntitle: 見出し\n---\n'));
+ assert(aligned.includes('## Not a heading\n```'));
+ await assert.rejects(alignHeadings(sourceStructure, japanese.replace('##
固定\n', ''), '.md'), /heading structure/);
+});
+
+test('MDX imports, expressions and explicit IDs preserve heading order and raw
offsets', async () => {
+ const source = "import Card from '@site/Card';\n\n# Getting
started\n\nWelcome.\n\n## Usage\n\n<Card timeout={10}/>\n\n## Usage\n";
+ const japanese = "import Card from '@site/Card';\n\n# 入門
{#getting-started}\n\nようこそ。\n\n## 使用方法 {#usage}\n\n<Card timeout={10}/>\n\n##
使用方法 {#usage-1}\n";
+ const parsed = await structure(source, '.mdx');
+ assert.deepEqual(parsed.headings.map(h => h.id), ['getting-started',
'usage', 'usage-1']);
+ const aligned = await alignHeadings(parsed, japanese, '.mdx');
+ assert.deepEqual((await structure(aligned, '.mdx')).headings.map(h =>
h.id), ['getting-started', 'usage', 'usage-1']);
+ assert.equal((aligned.match(/<Card timeout=\{10\}\/>/g) ?? []).length, 1);
+ const prose = source.replace('Welcome.', 'Try Doris.');
+ assert.equal((await structure(prose, '.mdx')).technicalHash,
parsed.technicalHash);
+ const override = '---\nmdx:\n format: mdx\n---\n';
+ assert.deepEqual((await structure(override + japanese,
'.md')).headings.map(h => h.id), ['getting-started', 'usage', 'usage-1']);
+ assert.deepEqual((await structure(override + source,
'.md')).headings.map(h => h.id), ['getting-started', 'usage', 'usage-1']);
+ const setext = await structure('English heading\n---------------\n',
'.mdx');
+ const setextTarget = await alignHeadings(setext,
'日本語の見出し\n-------------\n', '.mdx');
+ assert.deepEqual((await structure(setextTarget, '.mdx')).headings.map(h =>
h.id), ['english-heading']);
+});
diff --git a/scripts/i18n/docs-plugin.cjs b/scripts/i18n/docs-plugin.cjs
new file mode 100644
index 00000000000..6ce15e46674
--- /dev/null
+++ b/scripts/i18n/docs-plugin.cjs
@@ -0,0 +1,236 @@
+const fs = require('node:fs/promises');
+const path = require('node:path');
+const native = require('@docusaurus/plugin-content-docs');
+const {readVersionsMetadata} =
require('@docusaurus/plugin-content-docs/lib/versions/index.js');
+const {processDocMetadata} =
require('@docusaurus/plugin-content-docs/lib/docs.js');
+const {Globby, getPluginI18nPath, normalizeUrl, readLastUpdateData,
aliasedSitePath, resolveMarkdownLinkPathname} = require('@docusaurus/utils');
+const {applyTrailingSlash} = require('@docusaurus/utils-common');
+const {hash, relative, identity, loadConfig, loadRecords, sourceLocaleFor,
decide} = require('./state.cjs');
+const {structure, alignHeadings} = require('./structure.cjs');
+
+async function exists(file) { try { await fs.access(file); return true; }
catch { return false; } }
+async function read(file) { return await exists(file) ? fs.readFile(file,
'utf8') : undefined; }
+function localizedContext(context, locale) {
+ const defaultLocale = context.i18n.defaultLocale;
+ const rootBaseUrl = context.i18n.currentLocale === defaultLocale ?
context.baseUrl
+ : context.baseUrl.replace(`/${context.i18n.currentLocale}/`, '/');
+ const baseUrl = locale === context.i18n.currentLocale ? context.baseUrl
+ : normalizeUrl([rootBaseUrl, locale === defaultLocale ? '' : locale,
'/']);
+ return {...context, baseUrl, localizationDir: path.join(context.siteDir,
'i18n', locale),
+ i18n: {...context.i18n, currentLocale: locale}};
+}
+
+async function metadata(context, options, versionMetadata, source, filePath,
content) {
+ return processDocMetadata({context, options: {...options,
showLastUpdateTime: false, showLastUpdateAuthor: false},
+ versionMetadata, env: process.env.NODE_ENV, tagsFile: undefined,
+ docFile: {source, content, filePath, contentPath:
path.dirname(filePath)}});
+}
+
+// Keep the native plugin name, validation, route parser, links and sidebars.
+// The adapter only changes its Japanese localization directory and adds state.
+module.exports = async function translationDocsPlugin(context, options) {
+ const settings = context.i18n.locales.includes('ja') ?
loadConfig(context.siteDir).locales.ja : undefined;
+ if (!settings) return native.default(context, options);
+ const locale = 'ja';
+ const records = loadRecords(context.siteDir, locale);
+ const contexts = Object.fromEntries(['en', 'zh-CN', locale].map(lang =>
[lang, localizedContext(context, lang)]));
+ const versions = Object.fromEntries(await
Promise.all(Object.entries(contexts).map(async ([lang, ctx]) =>
+ [lang, await readVersionsMetadata({context: ctx, options})])));
+ // Match i18n/<locale>'s depth so valid project-relative MDX imports still
+ // reach the same @site files. Native plugin namespaces isolate instances.
+ const viewRoot = path.join(context.siteDir, '.i18n-build', locale);
+ const viewPluginDir = getPluginI18nPath({localizationDir: viewRoot,
pluginName: 'docusaurus-plugin-content-docs', pluginId: options.id});
+ const originalPluginDir = getPluginI18nPath({localizationDir:
contexts.ja.localizationDir,
+ pluginName: 'docusaurus-plugin-content-docs', pluginId: options.id});
+ if (context.i18n.currentLocale === locale) {
+ await fs.rm(viewPluginDir, {recursive: true, force: true});
+ await fs.mkdir(viewPluginDir, {recursive: true});
+ if (await exists(originalPluginDir)) await fs.cp(originalPluginDir,
viewPluginDir, {recursive: true});
+ }
+ const pages = [];
+ const reports = [];
+ const route = permalink => applyTrailingSlash(permalink, {trailingSlash:
context.siteConfig.trailingSlash, baseUrl: context.baseUrl});
+ for (const [index, englishVersion] of versions.en.entries()) {
+ const filenames = (await Globby(options.include, {cwd:
englishVersion.contentPath, ignore: options.exclude})).sort();
+ const registered = new Set(filenames);
+ const seenIds = new Set();
+ const versionDir = englishVersion.versionName === 'current' ?
'current' : `version-${englishVersion.versionName}`;
+ const targetDir = path.join(originalPluginDir, versionDir);
+ for (const filename of filenames) {
+ const englishFile = path.join(englishVersion.contentPath,
filename);
+ const englishContent = await fs.readFile(englishFile, 'utf8');
+ const englishDoc = await metadata(contexts.en, options,
englishVersion, filename, englishFile, englishContent);
+ seenIds.add(englishDoc.id);
+ const page = {pluginId: options.id, version:
englishVersion.versionName, docId: englishDoc.id, locale};
+ const requestedSourceLocale = sourceLocaleFor(settings, page);
+ const chineseFile =
path.join(versions['zh-CN'][index].contentPathLocalized, filename);
+ const chineseContent = await read(chineseFile);
+ const sourceLocale = requestedSourceLocale === 'zh-CN' &&
chineseContent !== undefined ? 'zh-CN' : 'en';
+ const sourceFile = sourceLocale === 'en' ? englishFile :
chineseFile;
+ const source = sourceLocale === 'en' ? englishContent :
chineseContent;
+ const targetFile = path.join(targetDir, filename);
+ const target = await read(targetFile);
+ Object.assign(page, {sourceLocale, sourcePath:
relative(context.siteDir, sourceFile),
+ englishPath: relative(context.siteDir, englishFile),
targetPath: relative(context.siteDir, targetFile)});
+ const record = records.get(identity(page));
+ const maintainCase =
context.siteConfig.markdown.anchors.maintainCase;
+ const sourceStructure = target !== undefined
+ ? await structure(source, path.extname(filename),
maintainCase).catch(() => null) : null;
+ let state = decide({page, source, target, record, technicalHash:
sourceStructure?.technicalHash});
+ if (requestedSourceLocale !== sourceLocale) {
+ state = {...state, status: 'source_missing', useTarget: false,
indexable: false, contentLocale: 'en'};
+ reports.push({kind: 'source_divergence', ...page,
requestedSourceLocale});
+ }
+ // Parse all counterparts with the framework, including custom IDs,
+ // number prefixes and localized slugs. Never invent a missing
node.
+ const chineseDoc = await metadata(contexts['zh-CN'], options,
versions['zh-CN'][index], filename,
+ chineseContent === undefined ? englishFile : chineseFile,
chineseContent ?? englishContent);
+ const nativeSourceDoc = sourceLocale === 'en' ? englishDoc :
chineseDoc;
+ let localizedContent;
+ if (state.useTarget) {
+ const targetDoc = await metadata(contexts.ja, options,
versions.ja[index], filename, targetFile, target).catch(() => null);
+ if (!targetDoc || targetDoc.id !== englishDoc.id ||
targetDoc.slug !== englishDoc.slug || targetDoc.draft !== englishDoc.draft
+ || targetDoc.unlisted !== englishDoc.unlisted) {
+ state = {...state, status: 'state_conflict', useTarget:
false, indexable: false, contentLocale: 'en'};
+ } else {
+ try { localizedContent = await alignHeadings(sourceLocale
=== 'en' ? sourceStructure
+ : await structure(englishContent,
path.extname(filename), maintainCase), target, path.extname(filename),
maintainCase); }
+ catch { state = {...state, status: 'heading_conflict',
useTarget: false, indexable: false, contentLocale: 'en'}; }
+ }
+ }
+ // A Chinese source can supply content only for an English
registered
+ // node. Its routing metadata must still match that node.
+ if (sourceLocale === 'zh-CN' && (chineseDoc.id !== englishDoc.id
|| chineseDoc.slug !== englishDoc.slug
+ || chineseDoc.draft !== englishDoc.draft)) {
+ state = {...state, status: 'source_route_conflict', useTarget:
false, indexable: false, contentLocale: 'en'};
+ reports.push({kind: 'source_divergence', ...page, reason:
'source_route_conflict'});
+ }
+ const selectedContent = state.useTarget ? localizedContent :
englishContent;
+ const japaneseDoc = await metadata(contexts.ja, options,
versions.ja[index], filename, targetFile, selectedContent);
+ const indexable = state.indexable && !japaneseDoc.draft &&
!japaneseDoc.unlisted && !versions.ja[index].noIndex;
+ state = {...state, indexable, draft: japaneseDoc.draft, unlisted:
japaneseDoc.unlisted,
+ permalink: route(japaneseDoc.permalink), englishPermalink:
route(englishDoc.permalink),
+ sourcePermalink: route(nativeSourceDoc.draft ?
englishDoc.permalink : nativeSourceDoc.permalink),
+ alternates: {en: route(englishDoc.permalink),
...(!chineseDoc.draft && {'zh-CN': route(chineseDoc.permalink)}),
+ ...(indexable ? {ja: route(japaneseDoc.permalink)} : {})}};
+ pages.push(state);
+ if (context.i18n.currentLocale === locale) {
+ const viewFile = path.join(viewPluginDir, versionDir,
filename);
+ if (state.useTarget) {
+ await fs.mkdir(path.dirname(viewFile), {recursive: true});
+ await fs.writeFile(viewFile, selectedContent);
+ } else {
+ // Omitting it uses the native per-file English fallback.
+ await fs.rm(viewFile, {force: true});
+ }
+ }
+ }
+ for (const target of await Globby(options.include, {cwd: targetDir,
ignore: options.exclude})) {
+ if (!registered.has(target)) reports.push({kind: 'orphan_target',
pluginId: options.id,
+ version: englishVersion.versionName, targetPath:
relative(context.siteDir, path.join(targetDir, target))});
+ }
+ for (const target of await Globby(options.include, {cwd:
versions['zh-CN'][index].contentPathLocalized, ignore: options.exclude})) {
+ if (!registered.has(target)) reports.push({kind:
'unsupported_source_route', pluginId: options.id,
+ version: englishVersion.versionName, sourceLocale: 'zh-CN',
+ sourcePath: relative(context.siteDir,
path.join(versions['zh-CN'][index].contentPathLocalized, target))});
+ }
+ for (const record of records.values()) {
+ if (record.pluginId === options.id && record.version ===
englishVersion.versionName && !seenIds.has(record.docId)) {
+ reports.push({kind: 'source_deleted', ...record});
+ }
+ }
+ }
+ const pluginContext = context.i18n.currentLocale === locale ? {...context,
localizationDir: viewRoot} : context;
+ const plugin = await native.default(pluginContext, options);
+ const byId = new Map(pages.map(page => [identity(page), page]));
+ const sourceAliases = new Map();
+ const adaptedLoaders = new WeakSet();
+ let loaded = false;
+ return {...plugin,
+ getPathsToWatch() {
+ return [...plugin.getPathsToWatch().filter(file =>
!file.startsWith(viewRoot)), `${originalPluginDir}/**/*`,
+ ...versions['zh-CN'].map(version =>
`${version.contentPathLocalized}/**/*`),
+ path.join(context.siteDir, 'translations/**/*.json')];
+ },
+ async loadContent() {
+ // A dev-server content reload must re-evaluate hashes and rebuild
+ // the view, just like a new static build. Do not watch our own
view.
+ if (loaded) return (await translationDocsPlugin(context,
options)).loadContent();
+ loaded = true;
+ const content = await plugin.loadContent();
+ if (context.i18n.currentLocale === locale) {
+ for (const version of content.loadedVersions) {
+ for (const doc of [...version.docs, ...version.drafts]) {
+ const state = byId.get(identity({pluginId: options.id,
version: version.versionName, docId: doc.id, locale}));
+ if (!state || state.permalink !==
route(doc.permalink)) throw new Error(`Translation view/metadata mismatch:
${doc.id}`);
+ doc.translation = state;
+ // A translation commit is not a technical update. Ask
the
+ // same parser/map/git machinery about the actual
source.
+ const sourceFile = path.join(context.siteDir,
state.contentLocale === 'en' ? state.englishPath : state.sourcePath);
+ const parsed = await
context.siteConfig.markdown.parseFrontMatter({filePath: sourceFile,
+ fileContent: await fs.readFile(sourceFile, 'utf8'),
+ defaultParseFrontMatter:
require('@docusaurus/utils').DEFAULT_PARSE_FRONT_MATTER});
+ // Even plugins that hide the date need the real source
+ // timestamp for sitemap lastmod when their file is in
+ // the temporary view (which has no Git history).
+ const sourceOptions = state.useTarget
+ ? {...options, showLastUpdateTime: true} : options;
+ const lastUpdate = await
readLastUpdateData(sourceFile, sourceOptions, parsed.frontMatter.last_update);
+ doc.lastUpdatedAt = lastUpdate.lastUpdatedAt;
+ doc.lastUpdatedBy = lastUpdate.lastUpdatedBy;
+ }
+ }
+ }
+ return {...content, translationManifest: {schemaVersion: 1,
pluginId: options.id, locale: context.i18n.currentLocale, pages, reports}};
+ },
+ translateContent({content, translationFiles}) {
+ // Docusaurus's native translateLoadedContent returns only
versions.
+ return {...plugin.translateContent({content, translationFiles}),
translationManifest: content.translationManifest};
+ },
+ async contentLoaded(args) {
+ await plugin.contentLoaded(args);
+ sourceAliases.clear();
+ if (context.i18n.currentLocale === locale) {
+ for (const version of args.content.loadedVersions) {
+ for (const doc of version.docs) {
+ const state = doc.translation;
+ const viewPath = path.join(viewPluginDir,
path.relative(originalPluginDir, path.join(context.siteDir, state.targetPath)));
+ for (const filename of [state.englishPath,
state.sourcePath, state.targetPath, relative(context.siteDir, viewPath)]) {
+
sourceAliases.set(aliasedSitePath(path.join(context.siteDir, filename),
context.siteDir), doc.permalink);
+ }
+ }
+ }
+ }
+ },
+ configureWebpack(...args) {
+ const result = plugin.configureWebpack(...args);
+ if (context.i18n.currentLocale === locale) {
+ // Native MDX cache keys cannot see a changed counterpart route
+ // or state file read through fs. Bind the Japanese filesystem
+ // cache to this exact content/route decision snapshot.
+ if (args[0]?.cache?.type === 'filesystem') result.cache =
{...args[0].cache,
+ version:
`${args[0].cache.version}:translations:${hash(JSON.stringify(pages))}`};
+ for (const rule of result.module.rules) {
+ for (const item of rule.use) {
+ if (adaptedLoaders.has(item.options)) continue;
+ adaptedLoaders.add(item.options);
+ const nativeResolve = item.options.resolveMarkdownLink;
+ // Native initialization captures the resolver in its
+ // processors. Recreate them with the adapted resolver.
+ delete item.options.processors;
+ item.options.resolveMarkdownLink = params => {
+ const version = versions.ja.find(candidate =>
[candidate.contentPath, path.join(viewPluginDir,
+ candidate.versionName === 'current' ?
'current' : `version-${candidate.versionName}`)]
+ .some(dir =>
params.sourceFilePath.startsWith(dir + path.sep)));
+ return version &&
resolveMarkdownLinkPathname(params.linkPathname, {...params, siteDir:
context.siteDir,
+ sourceToPermalink: sourceAliases,
contentPaths: {...version, contentPathLocalized: path.join(viewPluginDir,
+ version.versionName === 'current' ?
'current' : `version-${version.versionName}`)}}) || nativeResolve(params);
+ };
+ }
+ }
+ }
+ return result;
+ },
+ };
+};
+module.exports.validateOptions = native.validateOptions;
diff --git a/scripts/i18n/poc/README.md b/scripts/i18n/poc/README.md
index d9962402237..6a0fa3e4658 100644
--- a/scripts/i18n/poc/README.md
+++ b/scripts/i18n/poc/README.md
@@ -5,14 +5,16 @@ translation service runs in the browser or in the website
deployment workflows.
The existing production workflows still build only English and Chinese.
For local debugging, generate the usual inputs, then build all three locales in
-one process. Limiting documentation versions is useful for debugging, but does
+independent Node processes in one sequential job. The T02 run confirmed a
+system OOM when earlier language heaps accumulated; the original same-process
+command remains available only for historical comparison. Limiting
documentation versions is useful for debugging, but does
not provide full CI memory evidence:
```bash
yarn key-features:generate
DORIS_ENABLE_JA=true DORIS_ASSET_ORIGIN= DOCS_VERSIONS=current \
PWA_SERVICE_WORKER_URL=https://doris.apache.org/sw.js \
- yarn docusaurus build --locale en --locale zh-CN --locale ja
+ yarn build:locales
```
An empty `DORIS_ASSET_ORIGIN` makes JS and CSS load from the output's own
origin.
@@ -84,3 +86,5 @@ The homepage's owned news announcements and HTML illustration
captions are
translated. Japanese search, complete translation status/SEO, and public launch
remain later tasks; passing this PoC does not authorize enabling Japanese in
production.
+
+Use `measure-build.py --separate-locales --japanese` for the isolated full
three-language probe. It preserves the complete version/plugin scope and
records each child PID. Production cron/manual use the same launcher while
Japanese remains disabled.
diff --git a/scripts/i18n/poc/measure-build.py
b/scripts/i18n/poc/measure-build.py
index d64772f336a..8b48a6fc575 100644
--- a/scripts/i18n/poc/measure-build.py
+++ b/scripts/i18n/poc/measure-build.py
@@ -16,6 +16,7 @@ parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--source', type=Path, required=True)
parser.add_argument('--output', type=Path, required=True)
parser.add_argument('--japanese', action='store_true')
+parser.add_argument('--separate-locales', action='store_true')
args = parser.parse_args()
source, output = args.source.resolve(), args.output.resolve()
output.mkdir(parents=True, exist_ok=True)
@@ -27,10 +28,14 @@ if any(env[key] for key in restricted):
env['DORIS_ENABLE_JA'] = 'true' if args.japanese else 'false'
env['DORIS_ASSET_ORIGIN'] = ''
locales = ['en', 'zh-CN'] + (['ja'] if args.japanese else [])
-command = ['yarn', 'docusaurus', 'build']
-for locale in locales:
- command += ['--locale', locale]
-command += ['--out-dir', str(output / 'site')]
+if args.separate_locales:
+ command = ['node', str(Path(__file__).resolve().parents[2] /
'build-locales.cjs'),
+ '--site-dir', str(source), '--out-dir', str(output / 'site')]
+else:
+ command = ['yarn', 'docusaurus', 'build']
+ for locale in locales:
+ command += ['--locale', locale]
+ command += ['--out-dir', str(output / 'site')]
def run_text(command):
return subprocess.check_output(command, cwd=source, text=True).strip()
@@ -84,6 +89,7 @@ metadata = {
'runner_name': env.get('RUNNER_NAME'),
'uname': run_text(['uname', '-a']),
'command': command,
+ 'build_mode': 'separate-processes' if args.separate_locales else
'in-process',
'node_options': env.get('NODE_OPTIONS'),
'pwa_service_worker_url': env.get('PWA_SERVICE_WORKER_URL'),
'scope_variables': {key: env[key] for key in restricted},
@@ -97,6 +103,7 @@ metadata = {
'kernel_oom_kills_before': kernel_oom_kills(),
'started_at_utc': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()),
'language_events': [],
+ 'process_events': [],
}
(output / 'environment.json').write_text(json.dumps(metadata, indent=2) + '\n')
start = time.monotonic()
@@ -164,6 +171,14 @@ with (output / 'build.log').open('w') as log:
log.write(line)
log.flush()
plain_line = re.sub(r'\x1b\[[0-9;]*m', '', line)
+ if plain_line.lstrip().startswith('{'):
+ try:
+ process_event = json.loads(plain_line)
+ if process_event.get('event') in ('locale_process_start',
'locale_process_end'):
+ metadata['process_events'].append(process_event)
+ emit(process_event)
+ except json.JSONDecodeError:
+ pass
match = re.search(r'\[INFO\] \[([^]]+)\] Creating an optimized
production build', plain_line)
event = None
if match:
@@ -184,6 +199,9 @@ stop.set()
sampler.join()
metadata.update({'exit_code': exit_code, 'elapsed_s': round(time.monotonic() -
start, 3), 'sampled_peaks': peaks, 'cgroup_after': cgroup_snapshot(),
'meminfo_after': memory_info()})
metadata['kernel_oom_kills_after'] = kernel_oom_kills()
+assembly_report = source / '.i18n-build/build-result.json'
+if args.separate_locales and assembly_report.exists():
+ (output / 'build-assembly.json').write_text(assembly_report.read_text())
metadata['interpretation'] = 'Process RSS sums can double-count shared pages.
System used memory includes the whole runner. cgroup peaks can include earlier
steps: compare before/after. Max RSS alone is not whole-job memory. Missing
metrics are null, never an asserted zero.'
(output / 'result.json').write_text(json.dumps(metadata, indent=2) + '\n')
emit({'exit_code': exit_code, 'elapsed_s': metadata['elapsed_s'],
'sampled_peaks': peaks,
diff --git a/scripts/i18n/poc/verify-static.cjs
b/scripts/i18n/poc/verify-static.cjs
index a596db778bf..87acf766501 100644
--- a/scripts/i18n/poc/verify-static.cjs
+++ b/scripts/i18n/poc/verify-static.cjs
@@ -207,8 +207,15 @@ let browser;
}
if (route.startsWith('/ja/')) {
assert.equal(state.lang, 'ja');
+ const banner = (await
page.locator('.theme-doc-version-banner').allTextContents()).join(' ');
+
assert(!/\{(?:unreleased|versionLabel|siteTitle|latestVersionLink|v4xLink|v3xLink|v21Link)\}/.test(banner),
+ `Unexpanded version-banner placeholders: ${route}`);
const fallback = route.includes('before-you-start-the-poc') ||
route.endsWith('/feedback/');
- assert.equal(state.notice, fallback, route);
+ const translationStatus = await
page.locator('[data-translation-status]').getAttribute('data-translation-status').catch(()
=> null);
+ if (translationStatus) {
+ assert.equal(translationStatus, fallback ? 'untranslated' :
'current', route);
+ assert.equal(state.notice, true, route);
+ } else assert.equal(state.notice, fallback, route); // Original
T01 baseline.
const body = page.locator('.theme-doc-markdown');
if (fallback) {
assert(await
page.locator('.theme-doc-markdown[lang="en"]').count(), `English body language
missing: ${route}`);
@@ -281,9 +288,28 @@ let browser;
await page.screenshot({ path: path.join(output,
`ja-${width}-${theme}.png`), fullPage: true });
results.checks.push({ name: 'Japanese homepage theme and
viewport', theme, width, dimensions });
await page.close();
+ for (const [name, route] of [
+ ['current',
'/ja/docs/dev/getting-started/what-is-apache-doris/'],
+ ['fallback',
'/ja/docs/dev/getting-started/before-you-start-the-poc/'],
+ ]) {
+ const doc = await pageFor(route, { width, height: 1000 });
+ assert.equal(await
doc.locator('html').getAttribute('data-brand-theme'), theme);
+ const notice = doc.locator('[aria-label="翻訳状況"]');
+ if (await notice.count()) {
+ const box = await notice.boundingBox();
+ assert(box && box.x >= 0 && box.x + box.width <= width
+ 1,
+ `Translation notice overflow: ${name} ${width}
${theme}`);
+ const colors = await notice.evaluate(node =>
({color:getComputedStyle(node).color,
background:getComputedStyle(node).backgroundColor}));
+ assert.notEqual(colors.color, colors.background,
`Invisible translation notice: ${theme}`);
+ await doc.screenshot({path:path.join(output,
`ja-doc-${name}-${width}-${theme}.png`)});
+ results.checks.push({name:'Translation notice theme
and viewport', state:name, width, theme, box, colors});
+ }
+ await doc.close();
+ }
}
}
checked('Japanese homepage desktop/mobile and all eight brand themes');
+ checked('Japanese current/fallback notices fit desktop/mobile and all
eight brand themes');
}
for (const prefix of ['', '/zh-CN']) for (const width of [1440, 390]) {
const page = await pageFor(`${prefix}/`, { width, height: 1000 });
diff --git a/scripts/i18n/state.cjs b/scripts/i18n/state.cjs
new file mode 100644
index 00000000000..c0bff70fa27
--- /dev/null
+++ b/scripts/i18n/state.cjs
@@ -0,0 +1,85 @@
+const fs = require('node:fs');
+const path = require('node:path');
+const {createHash} = require('node:crypto');
+
+const hash = value => createHash('sha256').update(value.replace(/\r\n/g,
'\n')).digest('hex');
+const identity = ({pluginId, version, docId, locale}) =>
JSON.stringify([pluginId, version, docId, locale]);
+const relative = (root, file) => path.relative(root,
file).split(path.sep).join('/');
+
+function readJson(file, fallback) {
+ return fs.existsSync(file) ? JSON.parse(fs.readFileSync(file, 'utf8')) :
fallback;
+}
+
+function loadConfig(root) {
+ const config = readJson(path.join(root, 'translations/config.json'),
{schemaVersion: 1, locales: {}});
+ if (config.schemaVersion !== 1) throw new Error('Unsupported translation
config schemaVersion');
+ for (const [locale, settings] of Object.entries(config.locales)) {
+ if (!/^[a-z]{2}(?:-[A-Za-z0-9]+)*$/.test(locale) || !['en',
'zh-CN'].includes(settings.sourceLocale)) {
+ throw new Error(`Invalid translation locale/sourceLocale:
${locale}`);
+ }
+ if (!Array.isArray(settings.sourceOverrides)) throw new Error(`Missing
sourceOverrides for ${locale}`);
+ const keys = new Set();
+ for (const override of settings.sourceOverrides) {
+ const key = identity({...override, locale});
+ if (keys.has(key) || !['en',
'zh-CN'].includes(override.sourceLocale)) {
+ throw new Error(`Conflicting source override: ${key}`);
+ }
+ keys.add(key);
+ }
+ }
+ return config;
+}
+
+function loadRecords(root, locale) {
+ const data = readJson(path.join(root,
`translations/state/${locale}/docs.json`), {schemaVersion: 1, pages: []});
+ if (data.schemaVersion !== 1 || !Array.isArray(data.pages)) throw new
Error('Invalid translation state schema');
+ const records = new Map();
+ for (const record of data.pages) {
+ const key = identity(record);
+ if (record.locale !== locale || records.has(key)) throw new
Error(`Duplicate or invalid translation identity: ${key}`);
+ records.set(key, record);
+ }
+ return records;
+}
+
+function sourceLocaleFor(settings, page) {
+ return settings.sourceOverrides.find(override => identity({...override,
locale: page.locale}) === identity(page))?.sourceLocale
+ ?? settings.sourceLocale;
+}
+
+// Freshness, review identity, and ownership are independent. A prose exception
+// requires a human decision bound to BOTH revisions and the unchanged target.
+// Even that exception cannot authorize changes in protected technical content.
+function decide({page, source, target, record, technicalHash}) {
+ const sourceHash = hash(source);
+ const targetHash = target === undefined ? null : hash(target);
+ const result = (status, useTarget = false, indexable = false) => ({
+ ...page, status, contentLocale: useTarget ? page.locale : 'en',
+ useTarget, indexable, sourceHash, targetHash,
+ reviewStatus: record?.reviewStatus ?? 'unverified',
+ translatedAt: record?.translatedAt ?? null,
+ });
+ if (target === undefined) return result('untranslated');
+ if (!record) return result('unverified');
+ if (identity(record) !== identity(page) || record.sourcePath !==
page.sourcePath
+ || record.targetPath !== page.targetPath || record.sourceLocale !==
page.sourceLocale
+ || !/^[a-f0-9]{64}$/.test(record.sourceHash ?? '')
+ || !/^[a-f0-9]{64}$/.test(record.sourceTechnicalHash ?? '')
+ || !/^[a-f0-9]{64}$/.test(record.targetHash ?? '')
+ || !['machine', 'human', 'unverified'].includes(record.reviewStatus)
+ || !['machine', 'human'].includes(record.ownership)
+ || typeof record.glossaryVersion !== 'string'
+ || !/^[a-f0-9]{40}$/.test(record.sourceCommit ?? '')
+ || !Number.isFinite(Date.parse(record.translatedAt))) return
result('state_conflict');
+ if (record.targetHash !== targetHash) return result('target_modified');
+ if (record.sourceHash === sourceHash) return result('current', true,
record.reviewStatus !== 'unverified');
+ if (record.sourceTechnicalHash !== technicalHash) return
result('stale_technical');
+ const approval = record.safeSourceChange;
+ if (approval?.kind === 'prose' && approval.fromSourceHash ===
record.sourceHash
+ && approval.toSourceHash === sourceHash && approval.targetHash ===
targetHash
+ && typeof approval.reviewedBy === 'string' &&
approval.reviewedBy.trim()
+ && Number.isFinite(Date.parse(approval.reviewedAt))) return
result('stale_prose', true);
+ return result('stale_unknown');
+}
+
+module.exports = {hash, identity, relative, loadConfig, loadRecords,
sourceLocaleFor, decide};
diff --git a/scripts/i18n/status-plugin.cjs b/scripts/i18n/status-plugin.cjs
new file mode 100644
index 00000000000..e05db9fc6f5
--- /dev/null
+++ b/scripts/i18n/status-plugin.cjs
@@ -0,0 +1,28 @@
+const fs = require('node:fs/promises');
+const path = require('node:path');
+
+module.exports = function translationStatusPlugin(context) {
+ return {name: 'doris-translation-status',
+ async allContentLoaded({allContent, actions}) {
+ const manifests =
Object.values(allContent['docusaurus-plugin-content-docs'] ?? {})
+ .map(content => content.translationManifest).filter(Boolean);
+ const pages = manifests.flatMap(manifest => manifest.pages);
+ // Only current translations need a cross-locale route group in the
+ // browser. Missing/stale pages use the existing en/zh behavior
with
+ // ja removed; their detailed status lives in the doc's own
metadata.
+ const alternates = {};
+ for (const page of pages) {
+ if (page.alternates.ja) {
+ for (const permalink of Object.values(page.alternates))
alternates[permalink] = page.alternates;
+ }
+ }
+ const prefixes =
Object.values(allContent['docusaurus-plugin-content-docs'] ?? {})
+ .flatMap(content => content.loadedVersions.map(version =>
version.path));
+ actions.setGlobalData({alternates, prefixes});
+ const output = path.join(context.siteDir, '.i18n-build/manifests',
`${context.i18n.currentLocale}.json`);
+ await fs.mkdir(path.dirname(output), {recursive: true});
+ await fs.writeFile(output, JSON.stringify({schemaVersion: 1,
locale: context.i18n.currentLocale,
+ pages, reports: manifests.flatMap(manifest =>
manifest.reports)}, null, 2) + '\n');
+ },
+ };
+};
diff --git a/scripts/i18n/structure.cjs b/scripts/i18n/structure.cjs
new file mode 100644
index 00000000000..5aa2c484527
--- /dev/null
+++ b/scripts/i18n/structure.cjs
@@ -0,0 +1,94 @@
+const matter = require('gray-matter');
+const {hash} = require('./state.cjs');
+const headingsPlugin =
require('@docusaurus/mdx-loader/lib/remark/headings').default;
+const {escapeMarkdownHeadingIds} = require('@docusaurus/utils');
+const {getFormat} = require('@docusaurus/mdx-loader/lib/format.js');
+
+function withoutPositions(value) {
+ if (Array.isArray(value)) return value.map(withoutPositions);
+ if (value && typeof value === 'object') return
Object.fromEntries(Object.entries(value)
+ .filter(([key]) => !['position', 'start', 'end', 'loc',
'range'].includes(key))
+ .map(([key, child]) => [key, withoutPositions(child)]));
+ return value;
+}
+
+async function structure(raw, extension, anchorsMaintainCase = false) {
+ const [{unified}, {default: parse}, {default: gfm}, {default: mdx},
{visit}] = await Promise.all([
+ import('unified'), import('remark-parse'), import('remark-gfm'),
import('remark-mdx'), import('unist-util-visit'),
+ ]);
+ // Use native format detection, including front-matter MDX overrides.
+ const {data, content} = matter(raw, {});
+ const format = getFormat({filePath: `doc${extension}`, frontMatterFormat:
data.mdx?.format, markdownConfigFormat: 'detect'});
+ const processor = unified().use(parse).use(gfm);
+ if (format === 'mdx') processor.use(mdx);
+ const parseContent = format === 'mdx' ? escapeMarkdownHeadingIds(content)
: content;
+ const offsets = [];
+ let originalOffset = 0;
+ for (let i = 0; i < parseContent.length; i++) {
+ offsets[i] = originalOffset;
+ if (parseContent[i] === content[originalOffset]) originalOffset++;
+ else if (parseContent[i] !== '\\') throw new Error('Unexpected heading
preprocessing change');
+ }
+ offsets[parseContent.length] = content.length;
+ const tree = processor.parse(parseContent);
+ const headings = [];
+ // Use the installed framework's heading transform, including its duplicate
+ // slug order and explicit-ID behavior, instead of a second slug algorithm.
+ await headingsPlugin({anchorsMaintainCase})(tree);
+ visit(tree, 'heading', node => {
+ headings.push({depth: node.depth, id: node.data.id, position: {
+ start: {...node.position.start, offset:
offsets[node.position.start.offset]},
+ end: {...node.position.end, offset:
offsets[node.position.end.offset]},
+ }});
+ });
+ const protectedNodes = [];
+ visit(tree, node => {
+ if (['code', 'inlineCode', 'html', 'mdxjsEsm', 'mdxFlowExpression',
'mdxTextExpression'].includes(node.type)) {
+ protectedNodes.push([node.type, node.lang ?? null, node.meta ??
null, node.value]);
+ }
+ if (['link', 'image', 'definition', 'linkReference',
'imageReference'].includes(node.type)) {
+ protectedNodes.push([node.type, node.url ?? null, node.identifier
?? null]);
+ }
+ if (node.type.startsWith('mdxJsx')) protectedNodes.push([node.type,
node.name, withoutPositions(node.attributes)]);
+ if (node.type === 'text') {
+ const numbers = node.value.match(/\d+(?:\.\d+)*/g);
+ if (numbers) protectedNodes.push(['numbers', numbers]);
+ }
+ });
+ // Only title, description and sidebar_label are prose. Treat every other
+ // front-matter change conservatively, including explicit last_update.
+ const technicalMetadata = Object.fromEntries(Object.entries(data)
+ .filter(([key]) => !['title', 'description',
'sidebar_label'].includes(key))
+ .sort(([a], [b]) => a.localeCompare(b)));
+ return {
+ content, headings, format,
+ technicalHash: hash(JSON.stringify([technicalMetadata, protectedNodes,
headings.map(({depth, id}) => ({depth, id}))])),
+ };
+}
+
+async function alignHeadings(sourceStructure, target, extension, maintainCase
= false) {
+ const parsed = await structure(target, extension, maintainCase);
+ if (parsed.headings.length !== sourceStructure.headings.length
+ || parsed.headings.some((heading, i) => heading.depth !==
sourceStructure.headings[i].depth)) {
+ throw new Error('Translation heading structure differs from its
current source');
+ }
+ let content = parsed.content;
+ for (let i = parsed.headings.length - 1; i >= 0; i--) {
+ const heading = parsed.headings[i];
+ const {start, end} = heading.position;
+ const original = content.slice(start.offset, end.offset);
+ const id = sourceStructure.headings[i].id;
+ // ATX and setext headings both receive the source ID in their text.
+ const lines = original.split('\n');
+ const textLine = /^\s{0,3}(?:=+|-+)\s*$/.test(lines[lines.length - 1])
? lines.length - 2 : lines.length - 1;
+ const marker = parsed.format === 'mdx' && textLine !== lines.length -
1 ? ` \\{#${id}}` : ` {#${id}}`;
+ lines[textLine] =
lines[textLine].replace(/\s*\\?\{#[^}]+\}\s*(?:#+\s*)?$/,
'').replace(/\s+#+\s*$/, '') + marker;
+ content = content.slice(0, start.offset) + lines.join('\n') +
content.slice(end.offset);
+ }
+ // Keep original front matter bytes (and raw targetHash). Only the ignored
+ // view is changed; committed Japanese documents are never rewritten.
+ const prefix = target.slice(0, target.length - parsed.content.length);
+ return prefix + content;
+}
+
+module.exports = {structure, alignHeadings};
diff --git a/scripts/i18n/verify-fixture.cjs b/scripts/i18n/verify-fixture.cjs
new file mode 100644
index 00000000000..03e50961304
--- /dev/null
+++ b/scripts/i18n/verify-fixture.cjs
@@ -0,0 +1,183 @@
+// Build a small, real Docusaurus site to exercise stale content, metadata and
+// orphan routing without changing any document in the working tree.
+const fs = require('node:fs/promises');
+const path = require('node:path');
+const {spawnSync} = require('node:child_process');
+const assert = require('node:assert/strict');
+const {hash} = require('./state.cjs');
+const {structure} = require('./structure.cjs');
+const {verify} = require('./verify-static.cjs');
+const root = path.resolve(__dirname, '../..');
+const fixture = path.join(root, '.i18n-build/fixture');
+const sourceBase = '---\ntitle: Example\nlast_update:\n date:
2026-01-01\n---\n## Usage\n\nUse Doris.\n\n```sql\nSELECT 1;\n```\n\n[Current
translation](./machine.md)\n';
+const japanese = sourceBase.replace('title: Example', 'title: 例').replace('##
Usage', '## 使用方法')
+ .replace('Use Doris.', 'Doris を使用します。').replace('date: 2026-01-01', 'date:
2026-10-11');
+
+async function write(file, content) {
+ await fs.mkdir(path.dirname(path.join(fixture, file)), {recursive: true});
+ await fs.writeFile(path.join(fixture, file), content);
+}
+
+async function verifyBrowser(site) {
+ const http = require('node:http');
+ const {chromium} = require('playwright');
+ const server = http.createServer(async (request, response) => {
+ try {
+ const pathname = decodeURIComponent(new URL(request.url,
'http://local').pathname);
+ const file = path.join(site, pathname, path.extname(pathname) ? ''
: 'index.html');
+ response.setHeader('Content-Type',
{'.js':'application/javascript', '.css':'text/css', '.html':'text/html;
charset=utf-8'}[path.extname(file)] ?? 'application/octet-stream');
+ response.end(await fs.readFile(file));
+ } catch { response.writeHead(404); response.end('Not found'); }
+ });
+ await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));
+ const browser = await chromium.launch({headless:true});
+ try {
+ const page = await browser.newPage();
+ const errors = [];
+ page.on('pageerror', error => errors.push(error.message));
+ const origin = `http://127.0.0.1:${server.address().port}`;
+ await page.goto(origin + '/ja/docs/dev/technical/?from=fixture#usage',
{waitUntil:'networkidle'});
+ assert.equal(await
page.locator('.theme-doc-markdown').getAttribute('lang'), 'en');
+ assert.equal((await page.locator('.theme-doc-markdown pre
code').textContent()).trim(), 'SELECT 2;');
+ assert.equal(await page.locator('[data-translation-status]
a[hreflang="en"]').getAttribute('href'),
'/docs/dev/technical/?from=fixture#usage');
+ await page.locator('.theme-doc-markdown
a[href="/ja/docs/dev/machine/"]').click();
+ await page.waitForSelector('[data-translation-status="current"]');
+ assert.equal(await
page.locator('.theme-doc-markdown').getAttribute('lang'), 'ja');
+ await page.waitForFunction(() =>
!document.querySelector('meta[name="robots"]'));
+ assert.equal(await page.locator('meta[name="robots"]').count(), 0);
+ await page.goBack();
+ await
page.waitForSelector('[data-translation-status="stale_technical"]');
+ await page.waitForFunction(() =>
document.querySelector('link[rel="canonical"]').getAttribute('href') ===
'https://doris.apache.org/docs/dev/technical/');
+ assert.equal(await
page.locator('.theme-doc-markdown').getAttribute('lang'), 'en');
+ assert.equal(await
page.locator('link[rel="canonical"]').getAttribute('href'),
'https://doris.apache.org/docs/dev/technical/');
+ assert((await
page.locator('meta[name="robots"]').getAttribute('content')).includes('noindex'));
+ await page.reload({waitUntil:'networkidle'});
+ assert.equal(await
page.locator('[data-translation-status]').getAttribute('data-translation-status'),
'stale_technical');
+ for (const [docId, status, language] of [['prose','stale_prose','ja'],
['human','current','ja'], ['unverified','unverified','en']]) {
+ await page.goto(origin + `/ja/docs/dev/${docId}/`,
{waitUntil:'networkidle'});
+ assert.equal(await
page.locator('[data-translation-status]').getAttribute('data-translation-status'),
status);
+ assert.equal(await
page.locator('.theme-doc-markdown').getAttribute('lang'), language);
+ }
+ assert.deepEqual(errors, []);
+ return {passed:true, checks:['latest stale SQL', 'SPA
fallback-to-translation', 'back', 'refresh', 'query/hash English link',
'prose/human/unverified hydration']};
+ } finally { await browser.close(); await new Promise(resolve =>
server.close(resolve)); }
+}
+async function main() {
+ await fs.rm(fixture, {recursive: true, force: true});
+ await fs.mkdir(fixture, {recursive: true});
+ await fs.symlink(path.join(root, 'node_modules'), path.join(fixture,
'node_modules'));
+ const pages = [];
+ for (const name of ['untranslated', 'machine', 'human', 'unverified',
'prose', 'technical', 'unknown',
+ 'modified', 'heading', 'bad-metadata', 'deleted', 'chinese-source',
'chinese-draft', 'chinese-untranslated']) {
+ const sourcePath = `docs/${name}.md`;
+ const targetPath =
`i18n/ja/docusaurus-plugin-content-docs/current/${name}.md`;
+ const source = ['chinese-source',
'chinese-untranslated'].includes(name) ? sourceBase.replace('Use Doris.', '使用
Doris。')
+ : name === 'chinese-draft' ? sourceBase.replace('title: Example',
'title: Example\ndraft: true') : sourceBase;
+ const sourceLocale = name.startsWith('chinese-') ? 'zh-CN' : 'en';
+ const record = {pluginId: 'default', version: 'current', docId: name,
locale: 'ja', sourceLocale,
+ sourcePath: sourceLocale === 'en' ? sourcePath :
`i18n/zh-CN/docusaurus-plugin-content-docs/current/${name}.md`,
+ targetPath, sourceCommit: 'a'.repeat(40), sourceHash: hash(source),
+ sourceTechnicalHash: (await structure(source,
'.md')).technicalHash, targetHash: hash(japanese),
+ reviewStatus: name === 'human' ? 'human' : 'machine', ownership:
name === 'human' ? 'human' : 'machine',
+ translatedAt: '2026-10-11T19:57:11Z', glossaryVersion: '0'};
+ if (name !== 'deleted') await write(sourcePath, name === 'technical' ?
sourceBase.replace('SELECT 1', 'SELECT 2')
+ : ['prose', 'unknown'].includes(name) ? sourceBase.replace('Use
Doris.', 'Try Doris.') : sourceBase);
+ if (name !== 'deleted') await
write(`i18n/zh-CN/docusaurus-plugin-content-docs/current/${name}.md`,
+ name === 'machine' ? source.replace('title: Example', 'title:
Example\nslug: /localized-machine') : source);
+ if (!['untranslated', 'chinese-untranslated'].includes(name)) await
write(targetPath, name === 'modified' ? japanese + '\n手動編集\n'
+ : name === 'heading' ? japanese.replace('## 使用方法\n', '')
+ : name === 'bad-metadata' ? japanese.replace('title: 例', 'id:
invalid/id\ntitle: 例') : japanese);
+ if (name === 'heading') record.targetHash = hash(japanese.replace('##
使用方法\n', ''));
+ if (name === 'bad-metadata') record.targetHash =
hash(japanese.replace('title: 例', 'id: invalid/id\ntitle: 例'));
+ if (name === 'prose') record.safeSourceChange = {kind: 'prose',
fromSourceHash: record.sourceHash,
+ toSourceHash: hash(sourceBase.replace('Use Doris.', 'Try
Doris.')), targetHash: record.targetHash,
+ reviewedBy: 'fixture-reviewer', reviewedAt:
'2026-10-11T20:00:00Z'};
+ if (!['untranslated', 'unverified',
'chinese-untranslated'].includes(name)) pages.push(record);
+ }
+ await
write('i18n/zh-CN/docusaurus-plugin-content-docs/current/chinese-only.md',
sourceBase);
+ await write('versions.json', '["4.x"]\n');
+ await write('versioned_docs/version-4.x/machine.md',
sourceBase.replace('SELECT 1', 'SELECT 4'));
+ await write('versioned_sidebars/version-4.x-sidebars.json', '{}\n');
+ await
write('i18n/ja/docusaurus-plugin-content-docs/version-4.x/machine.md',
japanese);
+ pages.push({...pages.find(page => page.docId === 'machine'), version:
'4.x',
+ sourcePath: 'versioned_docs/version-4.x/machine.md', targetPath:
'i18n/ja/docusaurus-plugin-content-docs/version-4.x/machine.md'});
+ await write('community/machine.md', sourceBase);
+ await
write('i18n/ja/docusaurus-plugin-content-docs-community/current/machine.md',
japanese);
+ pages.push({...pages.find(page => page.docId === 'machine'), pluginId:
'community',
+ sourcePath: 'community/machine.md', targetPath:
'i18n/ja/docusaurus-plugin-content-docs-community/current/machine.md'});
+ await write('translations/config.json', JSON.stringify({schemaVersion: 1,
locales: {ja: {sourceLocale: 'en',
+ sourceOverrides: ['chinese-source', 'chinese-only', 'chinese-draft',
'chinese-untranslated'].map(docId => ({pluginId: 'default', version: 'current',
docId, sourceLocale: 'zh-CN'}))}}}));
+ await write('translations/state/ja/docs.json',
JSON.stringify({schemaVersion: 1, pages}, null, 2));
+ const relativeSource = '---\ntitle: Relative import\nlast_update:\n date:
2026-01-01\n---\nimport Card from \'../src/components/Card.jsx\';\n\n##
Usage\n\n<Card />\n';
+ const relativeTarget = relativeSource.replace('Relative import',
'相対パスのインポート')
+ .replace("'../src/components/Card.jsx'",
"'../../../../src/components/Card.jsx'").replace('## Usage', '## 使用方法');
+ await write('docs/relative.mdx', relativeSource);
+ await write('i18n/ja/docusaurus-plugin-content-docs/current/relative.mdx',
relativeTarget);
+ await write('src/components/Card.jsx', "import React from 'react'; export
default function Card(){return <p>SHARED_COMPONENT</p>;}\n");
+ pages.push({...pages.find(page => page.docId === 'machine' && page.version
=== 'current' && page.pluginId === 'default'),
+ docId:'relative', sourcePath:'docs/relative.mdx',
targetPath:'i18n/ja/docusaurus-plugin-content-docs/current/relative.mdx',
+ sourceHash:hash(relativeSource), targetHash:hash(relativeTarget),
sourceTechnicalHash:(await structure(relativeSource,'.mdx')).technicalHash});
+ await write('translations/state/ja/docs.json',
JSON.stringify({schemaVersion: 1, pages}, null, 2));
+ // Reuse the production components, including the single alternate emitter.
+ for (const file of ['src/utils/translation.ts', 'src/utils/japanese.ts',
'src/theme/SiteMetadata/index.tsx',
+ 'src/theme/DocItem/Metadata/index.tsx',
'src/theme/DocItem/Content/index.tsx',
+ 'src/theme/DocItem/Layout/TranslationStatus.tsx',
'src/theme/DocItem/Layout/JapaneseFallback.module.css']) {
+ await write(file, await fs.readFile(path.join(root, file), 'utf8'));
+ }
+ await write('src/theme/DocItem/Layout/index.tsx', `import React from
'react';
+import Original from '@theme-original/DocItem/Layout';
+import {useDoc} from '@docusaurus/plugin-content-docs/client';
+import {docTranslation} from '@site/src/utils/translation';
+import TranslationStatus from './TranslationStatus';
+export default function Layout(props) {
+ const state = docTranslation(useDoc().metadata);
+ return <>{state && <TranslationStatus state={state}/>}<Original
{...props}/></>;
+}\n`);
+ await write('src/pages/index.tsx', "import React from 'react'; export
default function Home(){return <main>Fixture</main>;}\n");
+ await write('docusaurus.config.js', `module.exports = {
+ title: 'Fixture', url: 'https://doris.apache.org', baseUrl: '/',
trailingSlash: true,
+ i18n: {defaultLocale: 'en', locales: ['en','zh-CN','ja'], localeConfigs: {
+ en: {htmlLang:'en-US'}, 'zh-CN': {htmlLang:'zh-Hans-CN'},
ja:{htmlLang:'ja'}}},
+ plugins: [[require(${JSON.stringify(path.join(root,
'scripts/i18n/docs-plugin.cjs'))}), {
+ path:'docs', sidebarPath:false, lastVersion:'4.x',
versions:{current:{path:'dev'},'4.x':{path:'4.x'}},showLastUpdateTime:true}],
+ [require(${JSON.stringify(path.join(root, 'scripts/i18n/docs-plugin.cjs'))}),
{id:'community', path:'community',
routeBasePath:'community',sidebarPath:false}],
+ require(${JSON.stringify(path.join(root, 'scripts/i18n/status-plugin.cjs'))}),
+ function fixtureCache(){return
{name:'fixture-cache',configureWebpack:()=>({cache:false})}}],
+ presets: [['classic',{docs:false,blog:false}]],
+};\n`);
+ const result = spawnSync(process.execPath, [path.join(root,
'scripts/build-locales.cjs'),
+ '--site-dir', fixture], {stdio: 'inherit',
+ env: {...process.env, NODE_OPTIONS: '--max-old-space-size=4096'}});
+ assert.equal(result.status, 0, 'Docusaurus fixture build failed');
+ const manifest = path.join(fixture, '.i18n-build/manifests/ja.json');
+ const verification = verify({site: path.join(fixture, 'build'), manifest});
+ const data = JSON.parse(await fs.readFile(manifest, 'utf8'));
+ const stateFor = (docId, version = 'current', pluginId = 'default') =>
data.pages.find(page => page.docId === docId && page.version === version &&
page.pluginId === pluginId);
+ for (const [name, status] of
Object.entries({untranslated:'untranslated',machine:'current',human:'current',unverified:'unverified',
+ prose:'stale_prose',
technical:'stale_technical',unknown:'stale_unknown',modified:'target_modified',heading:'heading_conflict',
+
'bad-metadata':'state_conflict','chinese-source':'current','chinese-draft':'source_route_conflict','chinese-untranslated':'untranslated'}))
{
+ assert.equal(stateFor(name).status, status);
+ }
+ assert.equal(stateFor('machine','4.x').status, 'stale_technical');
+ assert.equal(stateFor('machine','current','community').status, 'current');
+ assert.equal(stateFor('relative').status, 'current');
+ assert((await fs.readFile(path.join(fixture,
'build/ja/docs/dev/relative/index.html'),
'utf8')).includes('SHARED_COMPONENT'));
+ assert(data.reports.some(report => report.kind === 'source_deleted' &&
report.docId === 'deleted'));
+ assert(data.reports.some(report => report.kind ===
'unsupported_source_route' && report.sourcePath.endsWith('chinese-only.md')));
+ await assert.rejects(fs.access(path.join(fixture,
'build/ja/docs/dev/deleted/index.html')));
+ await assert.rejects(fs.access(path.join(fixture,
'build/ja/docs/dev/chinese-only/index.html')));
+ const html = await fs.readFile(path.join(fixture,
'build/ja/docs/dev/technical/index.html'), 'utf8');
+ const $ = require('cheerio').load(html);
+ assert.equal($('.theme-doc-markdown pre code').text().trim(), 'SELECT 2;',
'Latest SQL not rendered');
+ const metadataDir = path.join(fixture,
'.docusaurus/docusaurus-plugin-content-docs/default');
+ const metadata = await Promise.all((await
fs.readdir(metadataDir)).filter(file => file.endsWith('.json'))
+ .map(async file => JSON.parse(await fs.readFile(path.join(metadataDir,
file), 'utf8'))));
+ const machine = metadata.find(doc => doc.id === 'machine' && doc.version
=== 'current' && doc.translation);
+ assert.equal(machine.lastUpdatedAt, new Date('2026-01-01').getTime(),
'Translation date replaced source content date');
+ assert.equal(machine.translation.translatedAt, '2026-10-11T19:57:11Z');
+ assert.equal(hash(await fs.readFile(path.join(fixture,
'i18n/ja/docusaurus-plugin-content-docs/current/technical.md'), 'utf8')),
hash(japanese));
+ const browser = process.argv.includes('--browser') ? await
verifyBrowser(path.join(fixture, 'build')) : undefined;
+ console.log(JSON.stringify({fixture, ...verification, browser}, null, 2));
+}
+if (require.main === module) main().catch(error => {console.error(error);
process.exitCode = 1;});
+module.exports = {verifyBrowser};
diff --git a/scripts/i18n/verify-static.cjs b/scripts/i18n/verify-static.cjs
new file mode 100644
index 00000000000..43472db438f
--- /dev/null
+++ b/scripts/i18n/verify-static.cjs
@@ -0,0 +1,85 @@
+const assert = require('node:assert/strict');
+const fs = require('node:fs');
+const path = require('node:path');
+const cheerio = require('cheerio');
+
+function verify({site, manifest}) {
+ const data = JSON.parse(fs.readFileSync(manifest, 'utf8'));
+ const sitemapFile = path.join(site, 'ja/sitemap.xml');
+ const sitemap = fs.existsSync(sitemapFile) ?
cheerio.load(fs.readFileSync(sitemapFile, 'utf8'), {xmlMode: true}) : null;
+ const sitemapPaths = new Set(sitemap ? sitemap('loc').map((i, node) => new
URL(sitemap(node).text()).pathname).get() : []);
+ const indexedPaths = new Set();
+ function readIndexes(dir) {
+ for (const entry of fs.readdirSync(dir, {withFileTypes: true})) {
+ const file = path.join(dir, entry.name);
+ if (entry.isDirectory()) readIndexes(file);
+ else if (/^search-index.*\.json$/.test(entry.name)) {
+ for (const index of JSON.parse(fs.readFileSync(file, 'utf8')))
{
+ for (const doc of index.documents ?? []) {
+ if (doc.u) indexedPaths.add(new URL(doc.u,
'https://doris.apache.org').pathname.replace(/\/$/, '') + '/');
+ }
+ }
+ }
+ }
+ }
+ readIndexes(path.join(site, 'ja'));
+ let verified = 0;
+ for (const page of data.pages) {
+ const filename = path.join(site, decodeURIComponent(page.permalink),
'index.html');
+ if (!fs.existsSync(filename)) {
+ assert(page.draft, `Missing non-draft static page:
${page.permalink}`);
+ continue;
+ }
+ const $ = cheerio.load(fs.readFileSync(filename, 'utf8'));
+ const body = $('.theme-doc-markdown,.course-doc-content');
+ assert.equal(body.attr('lang'), page.contentLocale, `Body language:
${page.permalink}`);
+
assert.equal($('[data-translation-status]').attr('data-translation-status'),
page.status, `Notice: ${page.permalink}`);
+ assert.equal($('link[rel="canonical"]').length, 1, `Duplicate
canonical: ${page.permalink}`);
+ const canonical = page.useTarget ? page.permalink : page.contentLocale
=== 'en' ? page.englishPermalink : page.sourcePermalink;
+ assert.equal($('link[rel="canonical"]').attr('href'),
`https://doris.apache.org${canonical}`);
+ const alternates = $('link[rel="alternate"][hreflang]');
+ const japanese = alternates.filter('[hreflang="ja"]');
+ assert.equal(japanese.length, page.alternates.ja ? 1 : 0, `Japanese
alternate: ${page.permalink}`);
+ if (!page.indexable) {
+
assert($('meta[name="robots"]').attr('content').includes('noindex'),
page.permalink);
+ assert(!sitemapPaths.has(page.permalink), `Fallback/stale page in
sitemap: ${page.permalink}`);
+ assert(!indexedPaths.has(page.permalink), `Fallback/stale page in
local search index: ${page.permalink}`);
+ for (const permalink of Object.values(page.alternates)) {
+ const file = path.join(site, permalink, 'index.html');
+ if (fs.existsSync(file)) {
+ const counterpart = cheerio.load(fs.readFileSync(file,
'utf8'));
+
assert.equal(counterpart('link[rel="alternate"][hreflang="ja"]').length, 0,
+ `Fallback/stale Japanese alternate in ${permalink}`);
+ }
+ }
+ } else {
+ assert(!($('meta[name="robots"]').attr('content') ??
'').includes('noindex'), page.permalink);
+ assert(sitemapPaths.has(page.permalink), `Current page missing
from sitemap: ${page.permalink}`);
+ const ids = root => root('.theme-doc-markdown
h1,.theme-doc-markdown h2,.theme-doc-markdown h3,.theme-doc-markdown
h4,.theme-doc-markdown h5,.theme-doc-markdown h6').map((i, node) =>
root(node).attr('id') ?? '').get();
+ const english = cheerio.load(fs.readFileSync(path.join(site,
page.englishPermalink, 'index.html'), 'utf8'));
+ assert.deepEqual(ids($), ids(english), `Source heading IDs:
${page.permalink}`);
+ for (const [lang, permalink] of Object.entries(page.alternates)) {
+ const counterpart =
cheerio.load(fs.readFileSync(path.join(site, permalink, 'index.html'), 'utf8'));
+ const links = counterpart('link[rel="alternate"][hreflang]');
+ for (const [otherLang, otherPath] of
Object.entries(page.alternates)) {
+ const htmlLang = otherLang === 'en' ? 'en-US' : otherLang
=== 'zh-CN' ? 'zh-Hans-CN' : otherLang;
+
assert.equal(links.filter(`[hreflang="${htmlLang}"]`).attr('href'),
`https://doris.apache.org${otherPath}`,
+ `Non-reciprocal alternates ${lang} -> ${otherLang}:
${permalink}`);
+
assert.equal(links.filter(`[hreflang="${htmlLang}"]`).length, 1, `Duplicate
hreflang: ${permalink}`);
+ }
+ }
+ }
+ verified++;
+ }
+ return {verified, indexedPaths: indexedPaths.size, states:
Object.fromEntries([...new Set(data.pages.map(page => page.status))]
+ .map(status => [status, data.pages.filter(page => page.status ===
status).length])), reports: data.reports};
+}
+
+if (require.main === module) {
+ const args = process.argv.slice(2);
+ const option = name => args[args.indexOf(name) + 1];
+ const result = verify({site: path.resolve(option('--site')), manifest:
path.resolve(option('--manifest'))});
+ if (args.includes('--output')) fs.writeFileSync(option('--output'),
JSON.stringify(result, null, 2) + '\n');
+ console.log(JSON.stringify({verified: result.verified, states:
result.states, reports: result.reports.length}));
+}
+module.exports = {verify};
diff --git a/src/theme/DocItem/Content/index.tsx
b/src/theme/DocItem/Content/index.tsx
index 41c6ea0df67..e746053c589 100644
--- a/src/theme/DocItem/Content/index.tsx
+++ b/src/theme/DocItem/Content/index.tsx
@@ -7,6 +7,7 @@ import MDXContent from '@theme/MDXContent';
import type {Props} from '@theme/DocItem/Content';
import useDocusaurusContext from '@docusaurus/useDocusaurusContext';
import { isJapaneseFallback } from '@site/src/utils/japanese';
+import {docTranslation} from '@site/src/utils/translation';
/**
Title can be declared inside md content or declared through
@@ -33,7 +34,9 @@ export default function DocItemContent({children}: Props):
JSX.Element {
const {pluginId} = useDocsVersion();
const {metadata} = useDoc();
const {i18n: {currentLocale}} = useDocusaurusContext();
- const contentLanguage = isJapaneseFallback(metadata.source, currentLocale) ?
'en' : undefined;
+ const contentLanguage = currentLocale === 'ja'
+ ? docTranslation(metadata)?.contentLocale ??
(isJapaneseFallback(metadata.source, currentLocale) ? 'en' : 'ja')
+ : undefined;
if (pluginId === 'course') {
return <div className="course-doc-content"
lang={contentLanguage}><MDXContent>{children}</MDXContent></div>;
diff --git a/src/theme/DocItem/Layout/JapaneseFallback.module.css
b/src/theme/DocItem/Layout/JapaneseFallback.module.css
index a6897c08c5f..5f9e34e6a67 100644
--- a/src/theme/DocItem/Layout/JapaneseFallback.module.css
+++ b/src/theme/DocItem/Layout/JapaneseFallback.module.css
@@ -12,3 +12,9 @@
color: inherit;
text-decoration: underline;
}
+
+.translationDate {
+ display: block;
+ margin-top: 0.35rem;
+ font-size: 0.875rem;
+}
diff --git a/src/theme/DocItem/Layout/TranslationStatus.tsx
b/src/theme/DocItem/Layout/TranslationStatus.tsx
new file mode 100644
index 00000000000..8f2aa3094d3
--- /dev/null
+++ b/src/theme/DocItem/Layout/TranslationStatus.tsx
@@ -0,0 +1,47 @@
+import React, {type JSX} from 'react';
+import Link from '@docusaurus/Link';
+import {useLocation} from '@docusaurus/router';
+import useIsBrowser from '@docusaurus/useIsBrowser';
+import type {TranslationState} from '@site/src/utils/translation';
+import styles from './JapaneseFallback.module.css';
+
+function description(state: TranslationState): string {
+ if (state.status === 'untranslated') return 'このページはまだ日本語に翻訳されていません。';
+ if (state.status === 'stale_prose') return '日本語訳は原文の最新の表現を反映していません。';
+ if (state.status === 'source_missing' || state.status ===
'source_route_conflict') {
+ return '翻訳元を確認できないため、この日本語訳は表示していません。';
+ }
+ if (state.status === 'unverified' || state.status === 'state_conflict' ||
state.status === 'target_modified'
+ || state.status === 'heading_conflict' || state.reviewStatus ===
'unverified') {
+ return '日本語訳の内容や更新状況を確認できていません。';
+ }
+ if (state.status !== 'current') return '日本語訳は原文の最新の変更を反映していません。';
+ return state.reviewStatus === 'human' ? 'この日本語訳は人が校正しています。' :
'このページは機械翻訳されています。';
+}
+
+export default function TranslationStatus({state}: {state: TranslationState}):
JSX.Element {
+ const {search, hash} = useLocation();
+ // SSR has no request query/hash. Render the same initial href on both
+ // sides, then update it after hydration (React does not patch a mismatched
+ // attribute merely because the initial client location has a suffix).
+ const isBrowser = useIsBrowser();
+ const suffix = isBrowser ? search + hash : '';
+ const target = state.englishPermalink;
+ return (
+ <aside className={styles.notice} lang="ja" aria-label="翻訳状況"
data-translation-status={state.status}>
+ <strong>{description(state)}</strong>{' '}
+ {!state.useTarget && '現在のバージョンの英語原文を表示しています。 '}
+ <Link to={`pathname://${target}${suffix}`} autoAddBaseUrl={false}
hrefLang="en">
+ 英語版を開く
+ </Link>
+ {state.sourcePermalink !== state.englishPermalink && <>{' '}
+ <Link to={`pathname://${state.sourcePermalink}${suffix}`}
autoAddBaseUrl={false} hrefLang="zh-CN">
+ 中国語原文を開く
+ </Link>
+ </>}
+ {state.translatedAt && <span className={styles.translationDate}>
+ {'翻訳日: '}<time
dateTime={state.translatedAt}>{state.translatedAt.slice(0, 10)}</time>
+ </span>}
+ </aside>
+ );
+}
diff --git a/src/theme/DocItem/Layout/index.tsx
b/src/theme/DocItem/Layout/index.tsx
index 6e1706fed19..9953d2629a6 100644
--- a/src/theme/DocItem/Layout/index.tsx
+++ b/src/theme/DocItem/Layout/index.tsx
@@ -19,6 +19,8 @@ import type { Props } from '@theme/DocItem/Layout';
import { DocsEdit } from '../../../components/Icons/docs-edit';
import MobileSidebarDrawer from './MobileSidebarDrawer';
import JapaneseFallback from './JapaneseFallback';
+import TranslationStatus from './TranslationStatus';
+import {docTranslation} from '@site/src/utils/translation';
import { isJapaneseFallback as usesEnglishFallback } from
'@site/src/utils/japanese';
import styles from './styles.module.css';
@@ -63,7 +65,13 @@ export default function DocItemLayout({ children }: Props):
JSX.Element {
// metadata.source is the file actually selected by the content plugin,
// including its per-file fallback. Do not infer translation from the URL.
const isJapaneseFallback = usesEnglishFallback(metadata.source,
currentLocale);
- const docContent = isJapaneseFallback ? (
+ const translation = currentLocale === 'ja' ? docTranslation(metadata) :
undefined;
+ const docContent = translation ? (
+ <>
+ <TranslationStatus state={translation} />
+ <DocItemContent>{children}</DocItemContent>
+ </>
+ ) : isJapaneseFallback ? (
<>
<JapaneseFallback permalink={metadata.permalink} />
<DocItemContent>{children}</DocItemContent>
@@ -122,7 +130,8 @@ export default function DocItemLayout({ children }: Props):
JSX.Element {
{isNew ? (
<></>
) : (
- <Link to={sourceToEditUrl(metadata.source)}
className={`mr-6 ${styles.footerBtn}`}>
+ <Link to={sourceToEditUrl(translation ?
`@site/${translation.useTarget ? translation.targetPath
+ : translation.contentLocale === 'en' ?
translation.englishPath : translation.sourcePath}` : metadata.source)}
className={`mr-6 ${styles.footerBtn}`}>
<DocsEdit /> <span className="ml-2">{isZH ?
'编辑本页' : 'Edit this page'}</span>
</Link>
)}
diff --git a/src/theme/DocItem/Metadata/index.tsx
b/src/theme/DocItem/Metadata/index.tsx
index 7eb0d1ce69e..294b6ecaa6d 100644
--- a/src/theme/DocItem/Metadata/index.tsx
+++ b/src/theme/DocItem/Metadata/index.tsx
@@ -1,15 +1,23 @@
import React from 'react';
import {PageMetadata} from '@docusaurus/theme-common';
import {useDoc} from '@docusaurus/plugin-content-docs/client';
+import {docTranslation} from '@site/src/utils/translation';
+import useDocusaurusContext from '@docusaurus/useDocusaurusContext';
export default function DocItemMetadata(): JSX.Element {
const {metadata, frontMatter, assets} = useDoc();
+ const translation = docTranslation(metadata);
+ const {siteConfig: {url}} = useDocusaurusContext();
return (
<PageMetadata
title={metadata.title}
description={metadata.description}
keywords={frontMatter.keywords}
image={assets.image ?? frontMatter.image}
- />
+ >
+ {translation && !translation.indexable && <meta name="robots"
content="noindex, follow" />}
+ {translation && !translation.useTarget && <link rel="canonical"
href={url + translation.englishPermalink} />}
+ {translation && !translation.useTarget && <meta property="og:url"
content={url + translation.englishPermalink} />}
+ </PageMetadata>
);
}
diff --git a/src/theme/NavbarItem/LocaleDropdownNavbarItem/index.tsx
b/src/theme/NavbarItem/LocaleDropdownNavbarItem/index.tsx
index a0ba559aee7..d495a0d1b94 100644
--- a/src/theme/NavbarItem/LocaleDropdownNavbarItem/index.tsx
+++ b/src/theme/NavbarItem/LocaleDropdownNavbarItem/index.tsx
@@ -6,6 +6,7 @@ import { useLocation } from '@docusaurus/router';
import DropdownNavbarItem from '@theme/NavbarItem/DropdownNavbarItem';
import type { LinkLikeNavbarItemProps } from '@theme/NavbarItem';
import type { Props } from '@theme/NavbarItem/LocaleDropdownNavbarItem';
+import {useTranslationRoutes} from '@site/src/utils/translation';
import './styles.scss';
export default function LocaleDropdownNavbarItem({
@@ -19,6 +20,7 @@ export default function LocaleDropdownNavbarItem({
i18n: { currentLocale, locales, localeConfigs },
} = useDocusaurusContext();
const alternatePageUtils = useAlternatePageUtils();
+ const {group} = useTranslationRoutes();
const { search, hash } = useLocation();
// Locales listed here render in the dropdown with a "Coming Soon" badge
and
@@ -26,10 +28,11 @@ export default function LocaleDropdownNavbarItem({
const COMING_SOON_LOCALES = new Set<string>([]);
const localeItems = locales.map((locale): LinkLikeNavbarItemProps => {
- const baseTo = `pathname://${alternatePageUtils.createUrl({
+ const nativeTo = alternatePageUtils.createUrl({
locale,
fullyQualified: false,
- })}`;
+ });
+ const baseTo = `pathname://${(locale === 'ja' || currentLocale ===
'ja') && group?.[locale] ? group[locale] : nativeTo}`;
// preserve ?search#hash suffix on locale switches
const to = `${baseTo}${search}${hash}${queryString}`;
const isComingSoon = COMING_SOON_LOCALES.has(locale);
diff --git a/src/theme/SiteMetadata/index.tsx b/src/theme/SiteMetadata/index.tsx
new file mode 100644
index 00000000000..ef5970c5866
--- /dev/null
+++ b/src/theme/SiteMetadata/index.tsx
@@ -0,0 +1,163 @@
+/**
+ * Copyright (c) Facebook, Inc. and its affiliates.
+ *
+ * This source code is licensed under the MIT license found in the
+ * LICENSE file in the root directory of this source tree.
+ */
+
+import React, {type JSX} from 'react';
+import Head from '@docusaurus/Head';
+import {useTranslationRoutes} from '@site/src/utils/translation';
+import useDocusaurusContext from '@docusaurus/useDocusaurusContext';
+import useBaseUrl from '@docusaurus/useBaseUrl';
+import {PageMetadata, useThemeConfig} from '@docusaurus/theme-common';
+import {
+ DEFAULT_SEARCH_TAG,
+ useAlternatePageUtils,
+ keyboardFocusedClassName,
+} from '@docusaurus/theme-common/internal';
+import {useLocation} from '@docusaurus/router';
+import {applyTrailingSlash} from '@docusaurus/utils-common';
+import SearchMetadata from '@theme/SearchMetadata';
+
+// TODO move to SiteMetadataDefaults or theme-common ?
+// Useful for i18n/SEO
+// See
https://developers.google.com/search/docs/advanced/crawling/localized-versions
+// See https://github.com/facebook/docusaurus/issues/3317
+function AlternateLangHeaders(): JSX.Element {
+ const {
+ siteConfig: {url: siteUrl},
+ i18n: {currentLocale, defaultLocale, localeConfigs},
+ } = useDocusaurusContext();
+ const alternatePageUtils = useAlternatePageUtils();
+ const {group, managed} = useTranslationRoutes();
+ const availableLocales = Object.entries(localeConfigs).filter(([locale]) =>
+ group ? !!group[locale] : !managed || locale !== 'ja');
+
+ const currentHtmlLang = localeConfigs[currentLocale]!.htmlLang;
+
+ // HTML lang is a BCP 47 tag, but the Open Graph protocol requires
+ // using underscores instead of dashes.
+ // See https://ogp.me/#optional
+ // See https://en.wikipedia.org/wiki/IETF_language_tag)
+ const bcp47ToOpenGraphLocale = (code: string): string =>
+ code.replace('-', '_');
+
+ // Note: it is fine to use both "x-default" and "en" to target the same url
+ // See https://www.searchviu.com/en/multiple-hreflang-tags-one-url/
+ return (
+ <Head>
+ {availableLocales.map(([locale, {htmlLang}]) => (
+ <link
+ key={locale}
+ rel="alternate"
+ href={group ? siteUrl + group[locale] :
alternatePageUtils.createUrl({
+ locale,
+ fullyQualified: true,
+ })}
+ hrefLang={htmlLang}
+ />
+ ))}
+ <link
+ rel="alternate"
+ href={group ? siteUrl + group[defaultLocale] :
alternatePageUtils.createUrl({
+ locale: defaultLocale,
+ fullyQualified: true,
+ })}
+ hrefLang="x-default"
+ />
+
+ <meta
+ property="og:locale"
+ content={bcp47ToOpenGraphLocale(currentHtmlLang)}
+ />
+ {availableLocales.map(([, config]) => config)
+ .filter((config) => currentHtmlLang !== config.htmlLang)
+ .map((config) => (
+ <meta
+ key={`meta-og-${config.htmlLang}`}
+ property="og:locale:alternate"
+ content={bcp47ToOpenGraphLocale(config.htmlLang)}
+ />
+ ))}
+ </Head>
+ );
+}
+
+// Default canonical url inferred from current page location pathname
+function useDefaultCanonicalUrl() {
+ const {
+ siteConfig: {url: siteUrl, baseUrl, trailingSlash},
+ } = useDocusaurusContext();
+
+ // TODO using useLocation().pathname is not a super idea
+ // See https://github.com/facebook/docusaurus/issues/9170
+ const {pathname} = useLocation();
+
+ const canonicalPathname = applyTrailingSlash(useBaseUrl(pathname), {
+ trailingSlash,
+ baseUrl,
+ });
+
+ return siteUrl + canonicalPathname;
+}
+
+// TODO move to SiteMetadataDefaults or theme-common ?
+function CanonicalUrlHeaders({permalink}: {permalink?: string}) {
+ const {
+ siteConfig: {url: siteUrl},
+ } = useDocusaurusContext();
+ const defaultCanonicalUrl = useDefaultCanonicalUrl();
+
+ const canonicalUrl = permalink
+ ? `${siteUrl}${permalink}`
+ : defaultCanonicalUrl;
+ return (
+ <Head>
+ <meta property="og:url" content={canonicalUrl} />
+ <link rel="canonical" href={canonicalUrl} />
+ </Head>
+ );
+}
+
+export default function SiteMetadata(): JSX.Element {
+ const {
+ i18n: {currentLocale},
+ } = useDocusaurusContext();
+
+ // TODO maybe move these 2 themeConfig to siteConfig?
+ // These seems useful for other themes as well
+ const {metadata, image: defaultImage} = useThemeConfig();
+
+ return (
+ <>
+ <Head>
+ <meta name="twitter:card" content="summary_large_image" />
+ {/* The keyboard focus class name need to be applied when SSR so links
+ are outlined when JS is disabled */}
+ <body className={keyboardFocusedClassName} />
+ </Head>
+
+ {defaultImage && <PageMetadata image={defaultImage} />}
+
+ <CanonicalUrlHeaders />
+
+ <AlternateLangHeaders />
+
+ <SearchMetadata tag={DEFAULT_SEARCH_TAG} locale={currentLocale} />
+
+ {/*
+ It's important to have an additional <Head> element here, as it allows
+ react-helmet to override default metadata values set in previous <Head>
+ like "twitter:card". In same Head, the same meta would appear twice
+ instead of overriding.
+ */}
+ <Head>
+ {/* Yes, "metadatum" is the grammatically correct term */}
+ {metadata.map((metadatum, i) => (
+ <meta key={i} {...metadatum} />
+ ))}
+ </Head>
+ </>
+ );
+}
diff --git a/src/utils/translation.ts b/src/utils/translation.ts
new file mode 100644
index 00000000000..8e66fe292a6
--- /dev/null
+++ b/src/utils/translation.ts
@@ -0,0 +1,29 @@
+import useGlobalData from '@docusaurus/useGlobalData';
+import {useLocation} from '@docusaurus/router';
+
+export type TranslationState = {
+ status: string;
+ contentLocale: string;
+ indexable: boolean;
+ useTarget: boolean;
+ reviewStatus: 'machine' | 'human' | 'unverified';
+ translatedAt: string | null;
+ englishPermalink: string;
+ sourcePermalink: string;
+ englishPath: string;
+ sourcePath: string;
+ targetPath: string;
+};
+
+export function docTranslation(metadata: unknown): TranslationState |
undefined {
+ return (metadata as {translation?: TranslationState}).translation;
+}
+
+type StatusData = {alternates: Record<string, Record<string, string>>;
prefixes: string[]};
+export function useTranslationRoutes() {
+ const data = useGlobalData()['doris-translation-status']?.default as
StatusData | undefined;
+ const {pathname} = useLocation();
+ const route = pathname.replace(/\/$/, '') + '/';
+ return {group: data?.alternates[route],
+ managed: !!data?.prefixes.some(prefix => route === prefix ||
route.startsWith(prefix.replace(/\/$/, '') + '/'))};
+}
diff --git a/translations/README.md b/translations/README.md
new file mode 100644
index 00000000000..6f1bc243a26
--- /dev/null
+++ b/translations/README.md
@@ -0,0 +1,125 @@
+# Documentation translation state
+
+Japanese remains opt-in (`DORIS_ENABLE_JA=true`). Builds are static and never
+contact a translation service. English and Chinese retain their existing native
+content selection when Japanese is disabled.
+
+`config.json` declares the source policy. English is the default. A
+`sourceOverrides` entry identifies `pluginId`, `version`, `docId` and
+`sourceLocale` (`en` or `zh-CN`). Overrides apply only to nodes registered by
the
+English docs plugin. A Chinese-only file is reported as
+`unsupported_source_route`; setting an override does not create its route.
+
+## State schema, version 1
+
+`state/ja/docs.json` stores a `schemaVersion` and a `pages` array. Each record
+contains these fields:
+
+| Field | Meaning |
+| --- | --- |
+| `pluginId`, `version`, `docId`, `locale` | Identity from the native docs
plugin. `current` is a version identity; its route is `/docs/dev/`. Identical
IDs in other plugins or versions are separate pages. |
+| `sourceLocale`, `sourcePath` | The selected fact source and its
repository-relative filename. |
+| `sourceCommit` | Full Git commit at which the translation source was
recorded. Freshness uses hashes, not commit ancestry. |
+| `sourceHash` | SHA-256 of the entire source file, normalizing only CRLF to
LF. Includes code, MDX, front matter, headings and whitespace. No timestamp
field is silently discarded. |
+| `sourceTechnicalHash` | SHA-256 of the conservative structural projection
described below. Used only as an additional guard for an explicit prose
exception. |
+| `targetPath`, `targetHash` | Repository-relative translation filename and
the same full-file SHA-256 rule. These refer to committed content, before
temporary heading alignment. |
+| `reviewStatus` | `machine`, `human` or `unverified`. Identifies the review
of the recorded target, independently of freshness. `human` must be set only
after actual human proofreading. |
+| `ownership` | `machine` or `human`. Records who maintains the target; a
changed target hash always prevents automatic selection, regardless of
ownership. |
+| `translatedAt` | ISO-8601 translation time. Displayed separately from the
source's content-update date. |
+| `glossaryVersion` | Terminology revision, initially `"0"`. |
+| `safeSourceChange` | Optional human-reviewed prose exception, bound to the
exact revisions below. |
+
+The initial PoC records keep `reviewStatus: machine`. Reviewing a PR is not
+assumed to mean that a human has proofread its Japanese translation.
+
+### Freshness and content selection
+
+Missing targets use the framework's per-file English fallback. Targets without
a
+record, with conflicting identity, or with a changed target hash also display
+English with an explicit unverified notice. Valid records with matching source
+and target hashes display Japanese. Unverified records remain excluded from
+indexing even when their hashes match.
+
+A changed source displays its latest original unless a reviewer adds:
+
+```json
+{
+ "safeSourceChange": {
+ "kind": "prose",
+ "fromSourceHash": "<recorded sourceHash>",
+ "toSourceHash": "<current sourceHash>",
+ "targetHash": "<unchanged targetHash>",
+ "reviewedBy": "<reviewer>",
+ "reviewedAt": "<ISO-8601 time>"
+ }
+}
+```
+
+This exception retains Japanese with a stale notice and excludes it from
+indexing. A changed technical projection overrides the exception. The
projection
+includes code blocks and their options, inline code, numbers, link
destinations,
+heading IDs/order/depth, raw HTML, MDX imports, expressions and component
+attributes, and all front matter except `title`, `description` and
+`sidebar_label`. AST positions are excluded. Equal projections alone do **not**
+prove a change is prose: a parameter name in plain text could have changed, so
+an explicit human decision is still required. Unsupported parsing cannot
approve
+an exception and conservatively selects English.
+
+For an explicit Chinese source, the registered English identity and slug must
+remain valid. Its Chinese original must exist and match that identity/slug;
+otherwise the build reports a source problem and displays English. Japanese
+heading IDs still follow the current English page.
+
+Untranslated and unsafe pages always use the native English fallback, including
+pages configured with a Chinese translation source. That setting controls the
+translation's freshness and source-update time; its Chinese original remains
+available through a separate link. Fallback body, assets and relative imports
+come from the registered English node without mixing localization directories.
+
+### Build adapter and SEO
+
+`scripts/i18n/docs-plugin.cjs` wraps Docusaurus 3.6.3's native docs plugin,
retaining
+its name, option validation, version parsing, sidebar generation and routes.
For
+Japanese only, it copies localized inputs into the ignored
+`.i18n-build/ja/docusaurus-plugin-content-docs[-<pluginId>]/` tree. Its depth
+matches the committed localization tree, preserving project-relative MDX
+imports. It aligns usable translations' heading IDs
+with the native English heading transform. It omits unsafe targets so the
+framework selects English. Committed translations are neither overwritten nor
+deleted. Source deletion removes the route through native enumeration; orphan
+targets and stale records appear in the report.
+
+Relative Markdown links resolve through aliases for both the source and the
+temporary view, including links between Japanese and English fallback pages.
+Content reloads recompute hashes. Simultaneous builds of the same locale in one
+checkout are unsupported; use separate worktrees.
+
+The adapter emits actual framework permalinks and state into each doc's
metadata.
+The UI, body `lang`, `noindex`, canonical URL and alternate eligibility use
that
+state. Fallback pages canonicalize to their registered English node. Stale
Japanese keeps
+its Japanese canonical URL but is `noindex`. Native sitemap and local-search
+parsers honor `noindex`; the single `SiteMetadata` alternate emitter excludes
+Japanese for those pages. Current translations receive reciprocal en/zh-CN/ja
+links, using actual localized slugs.
+
+The source parser's explicit date, history map or Git history supplies content
+update metadata. The temporary file's creation time and translation commit do
+not replace it. Generated route/state reports are written to
+`.i18n-build/manifests/<build-locale>.json`; they are not published as site
pages.
+
+## Validation
+
+```bash
+node --test scripts/i18n/__tests__/*.test.cjs scripts/i18n/poc/*.test.cjs
+node scripts/i18n/verify-fixture.cjs
+DORIS_ENABLE_JA=true DOCS_VERSIONS=current DORIS_ASSET_ORIGIN='' \
+ yarn build:locales
+node scripts/i18n/verify-static.cjs --site build \
+ --manifest .i18n-build/manifests/ja.json
+```
+
+The fixture builds a small real three-language site, covering stale SQL,
unknown
+changes, approved prose, manual target edits, source deletion, unsupported
+Chinese-only nodes, independent versions/plugins, links and source timestamps.
+Full active-version and memory validation runs in the read-only Japanese CI
+workflow. Local website builds use only `DOCS_VERSIONS=current`.
diff --git a/translations/config.json b/translations/config.json
new file mode 100644
index 00000000000..5a8d7a28037
--- /dev/null
+++ b/translations/config.json
@@ -0,0 +1,9 @@
+{
+ "schemaVersion": 1,
+ "locales": {
+ "ja": {
+ "sourceLocale": "en",
+ "sourceOverrides": []
+ }
+ }
+}
diff --git a/translations/state/ja/docs.json b/translations/state/ja/docs.json
new file mode 100644
index 00000000000..97da48d32a4
--- /dev/null
+++ b/translations/state/ja/docs.json
@@ -0,0 +1,73 @@
+{
+ "schemaVersion": 1,
+ "pages": [
+ {
+ "pluginId": "default",
+ "version": "current",
+ "docId": "getting-started/what-is-apache-doris",
+ "locale": "ja",
+ "sourceLocale": "en",
+ "sourcePath": "docs/getting-started/what-is-apache-doris.md",
+ "targetPath":
"i18n/ja/docusaurus-plugin-content-docs/current/getting-started/what-is-apache-doris.md",
+ "sourceCommit": "ec0bc51d17ff1c32df0241d61371f5f63db29b13",
+ "sourceHash":
"c1b950ba12598be8495190f93fd341dd646d98c837bf09c72ccc199c2ebb941e",
+ "sourceTechnicalHash":
"2c3ee90a60bae82e00653c8a088e174bf2ecd6eb8ce1bdda96d08cc50a160318",
+ "targetHash":
"565ac481aaafe738b0fd193d6f4617cff6698351cfc71c8d0b575de97634e4c4",
+ "reviewStatus": "machine",
+ "ownership": "machine",
+ "translatedAt": "2026-10-11T19:57:10.000Z",
+ "glossaryVersion": "0"
+ },
+ {
+ "pluginId": "default",
+ "version": "current",
+ "docId": "getting-started/intro",
+ "locale": "ja",
+ "sourceLocale": "en",
+ "sourcePath": "docs/getting-started/intro.mdx",
+ "targetPath":
"i18n/ja/docusaurus-plugin-content-docs/current/getting-started/intro.mdx",
+ "sourceCommit": "ec0bc51d17ff1c32df0241d61371f5f63db29b13",
+ "sourceHash":
"5c870a2583db46c203dcd610b575d3888bf2ba442e41a58a8acb7177d9db4abe",
+ "sourceTechnicalHash":
"691b2315e50a90059f94a53fe7ffa14602ebfe5d0069d69053bf44580d12aa42",
+ "targetHash":
"7b759880571be86dfa613ef71ea4b018e2c9b7f323eb9873c9596c1dafb0b33b",
+ "reviewStatus": "machine",
+ "ownership": "machine",
+ "translatedAt": "2026-10-11T19:57:10.000Z",
+ "glossaryVersion": "0"
+ },
+ {
+ "pluginId": "default",
+ "version": "4.x",
+ "docId": "getting-started/what-is-apache-doris",
+ "locale": "ja",
+ "sourceLocale": "en",
+ "sourcePath":
"versioned_docs/version-4.x/getting-started/what-is-apache-doris.md",
+ "targetPath":
"i18n/ja/docusaurus-plugin-content-docs/version-4.x/getting-started/what-is-apache-doris.md",
+ "sourceCommit": "ec0bc51d17ff1c32df0241d61371f5f63db29b13",
+ "sourceHash":
"c1b950ba12598be8495190f93fd341dd646d98c837bf09c72ccc199c2ebb941e",
+ "sourceTechnicalHash":
"2c3ee90a60bae82e00653c8a088e174bf2ecd6eb8ce1bdda96d08cc50a160318",
+ "targetHash":
"565ac481aaafe738b0fd193d6f4617cff6698351cfc71c8d0b575de97634e4c4",
+ "reviewStatus": "machine",
+ "ownership": "machine",
+ "translatedAt": "2026-10-11T19:57:10.000Z",
+ "glossaryVersion": "0"
+ },
+ {
+ "pluginId": "community",
+ "version": "current",
+ "docId": "join-community",
+ "locale": "ja",
+ "sourceLocale": "en",
+ "sourcePath": "community/join-community.md",
+ "targetPath":
"i18n/ja/docusaurus-plugin-content-docs-community/current/join-community.md",
+ "sourceCommit": "ec0bc51d17ff1c32df0241d61371f5f63db29b13",
+ "sourceHash":
"c0fb07e339e759ca4bf071efabafb60ecb13e1978173dc88e3de79e7a32949b9",
+ "sourceTechnicalHash":
"7ba43a16e4a34f5071e5c95fdef1a9db9511fee84628708975f6e450281845db",
+ "targetHash":
"69442d5fd55657623ba9b98cd6d2875fda3e031d30314877e7b6d2758ba45c8c",
+ "reviewStatus": "machine",
+ "ownership": "machine",
+ "translatedAt": "2026-10-11T19:57:10.000Z",
+ "glossaryVersion": "0"
+ }
+ ]
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
