This is an automated email from the ASF dual-hosted git repository.
smiletan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new d444caddbb8 [k8s][recovery mode]probe doc,recovery mode doc (#2704)
d444caddbb8 is described below
commit d444caddbb838509bf17f5debf99ad62d9062f06
Author: smiletan <[email protected]>
AuthorDate: Mon Aug 11 12:55:45 2025 +0800
[k8s][recovery mode]probe doc,recovery mode doc (#2704)
## Versions
- [x] dev
- [x] 3.0
- [x] 2.1
- [ ] 2.0
## Languages
- [x] Chinese
- [x] English
## Docs Checklist
- [ ] Checked by AI
- [ ] Test Cases Built
---
.../cluster-operation.md | 28 ++++++++++++++++
.../install-config-cluster.md | 38 ++++++++++++++++++++--
.../separating-storage-compute/config-cg.md | 11 ++++++-
.../cluster-operation.md | 27 +++++++++++++++
.../install-config-cluster.md | 35 +++++++++++++++++++-
.../separating-storage-compute/config-cg.md | 11 ++++++-
.../deploy-on-kubernetes/cluster-operation.md | 27 +++++++++++++++
.../deploy-on-kubernetes/install-config-cluster.md | 35 +++++++++++++++++++-
.../cluster-operation.md | 27 +++++++++++++++
.../install-config-cluster.md | 37 +++++++++++++++++++--
.../separating-storage-compute/config-cg.md | 11 ++++++-
.../deploy-on-kubernetes/cluster-operation.md | 28 ++++++++++++++++
.../deploy-on-kubernetes/install-config-cluster.md | 35 +++++++++++++++++++-
.../cluster-operation.md | 28 ++++++++++++++++
.../install-config-cluster.md | 36 +++++++++++++++++++-
.../separating-storage-compute/config-cg.md | 11 ++++++-
16 files changed, 413 insertions(+), 12 deletions(-)
diff --git
a/docs/install/deploy-on-kubernetes/integrated-storage-compute/cluster-operation.md
b/docs/install/deploy-on-kubernetes/integrated-storage-compute/cluster-operation.md
index 852de64a173..ce068599b66 100644
---
a/docs/install/deploy-on-kubernetes/integrated-storage-compute/cluster-operation.md
+++
b/docs/install/deploy-on-kubernetes/integrated-storage-compute/cluster-operation.md
@@ -310,3 +310,31 @@ admin set frontend config("disable_balance" = "false");
admin set frontend config("disable_colocate_balance" = "false");
admin set frontend config("disable_tablet_scheduler" = "false");
```
+
+## Starting FE with `metadata_failure_recovery` Mode
+When the Frontend (FE) service is unable to elect a leader and becomes
unavailable, you can recover the cluster by selecting the node with the highest
`VLSN` and force-starting it as the master using the recovery mechanism.
+
+### Starting in Recovery Mode in a Containerized Environment
+1. Identify the node with the highest `VLSN`
+ In Kubernetes, each time an FE Pod starts, it outputs the last 10 `VLSN`
records of the node. An example is shown below:
+ ```
+ the annotations value:
+ the value not equal! debug
+ /opt/apache-doris/fe/doris-meta/bdb/je.info.0:19:2025-08-05 03:42:47.650
UTC INFO [fe_f35530c4_3ff1_48fe_80d1_cc8e32dbc942] Replica-feeder
fe_d8763579_92da_4d72_8c58_4e62b88bdff0 start stream at VLSN: 30
+ /opt/apache-doris/fe/doris-meta/bdb/je.info.0:21:2025-08-05 03:42:47.659
UTC INFO [fe_f35530c4_3ff1_48fe_80d1_cc8e32dbc942] Replica initialization
completed. Replica VLSN: -1 Heartbeat master commit VLSN: 49 DTVLSN:0 Replica
VLSN delta: 50
+ [Tue Aug 5 06:14:05 UTC 2025] start with meta run start_fe.sh with
additional options: '--console'
+ ```
+ In this example, the highest `VLSN` on the current node is 30, as indicated
by the log prefix `start stream at VLSN:`.
+2. Designate the Pod with the highest `VLSN` for recovery
+ After identifying the Pod corresponding to the node with the highest
`VLSN`, annotate it to enable the recovery mechanism:
+ ```
+ kubectl annotate pod {podName} "selectdb.com.doris/recovery=true"
+ ```
+ Upon restarting, the Pod will automatically append the
`--metadata_failure_recovery` flag to its startup command and start in recovery
mode.
+3. Remove the annotation after recovery
+ Once the FE service is running normally, make sure to remove the annotation
added in Step 2 to avoid unexpected behavior during future restarts.
+
+:::tip Note
+1. After adding the annotation, do not restart the Pod using kubectl delete
pod, as this will remove the annotation. Instead, allow kubelet to restart it
automatically or manually kill the process inside the container.
+2. Starting FE in `metadata_failure_recovery` mode can take a long time due to
extensive log replay. Before proceeding, increase the FE service's [startup
probe timeout](./install-config-cluster.md#startup-probe-timeout), and delete
all FE Pods before initiating the recovery startup.
+:::
\ No newline at end of file
diff --git
a/docs/install/deploy-on-kubernetes/integrated-storage-compute/install-config-cluster.md
b/docs/install/deploy-on-kubernetes/integrated-storage-compute/install-config-cluster.md
index cc1a1165e43..7eb387b38a0 100644
---
a/docs/install/deploy-on-kubernetes/integrated-storage-compute/install-config-cluster.md
+++
b/docs/install/deploy-on-kubernetes/integrated-storage-compute/install-config-cluster.md
@@ -575,7 +575,7 @@ mysql -h
ac4828493dgrftb884g67wg4tb68gyut-1137856348.us-east-1.elb.amazonaws.com
```
## Configuring the username and password for the management cluster
-Managing Doris nodes requires connecting to the live FE nodes via the MySQL
protocol using a username and password for administrative operations. Doris
implements [a permission management mechanism similar to
RBAC](../../../admin-manual/auth/authentication-and-authorization?_highlight=rbac),
where the user must have the
[Node_priv](../../../admin-manual/auth/authentication-and-authorization.md#types-of-permissions)
permission to perform node management. By default, the Doris Operator dep [...]
+Managing Doris nodes requires connecting to the live FE nodes via the MySQL
protocol using a username and password for administrative operations. Doris
implements [a permission management mechanism similar to
RBAC](../../../admin-manual/auth/authentication-and-authorization.md#authentication-and-authorization-framework),
where the user must have the
[Node_priv](../../../admin-manual/auth/authentication-and-authorization.md#types-of-permissions)
permission to perform node management. By d [...]
The process of configuring the username and password can be divided into three
scenarios:
- initializing the root user password during cluster deployment;
@@ -587,7 +587,7 @@ To secure access, you must configure a username and
password with Node_Priv perm
- Using a Kubernetes Secret
### Configuring the root user password during cluster deployment
-To set the root user's password securely, Doris supports encrypting it in
[`fe.conf`](../../../admin-manual/config/fe-config?_highlight=initial_#initial_root_password)
using a two-stage SHA-1 encryption process. Here's how to set up the password.
+To set the root user's password securely, Doris supports encrypting it in
[`fe.conf`](../../../admin-manual/config/fe-config.md#initial_root_password)
using a two-stage SHA-1 encryption process. Here's how to set up the password.
#### Step 1: Generate the root encrypted password
@@ -850,3 +850,37 @@ spec:
:::tip Tip
The `mountPath` parameter can use `${DORIS_HOME}` as a prefix. When
`${DORIS_HOME}` is used, it resolves to `/opt/apache-doris/fe` within FE
containers and `/opt/apache-doris/be` within BE containers.
:::
+
+## Configuring Probe Timeouts
+DorisCluster provides two types of probe timeout configurations for each
service: `startup probe timeout` and `liveness probe timeout`. If a service
fails to start within the specified startup timeout period, it is considered to
have failed and will be restarted.
+If a service becomes unresponsive for longer than the specified liveness
timeout, the corresponding Pod will be automatically restarted.
+
+### Startup Probe Timeout
+- FE Service Startup Timeout Configuration
+ ```
+ spec:
+ feSpec:
+ startTimeout: 3600
+ ```
+ The above configuration sets the FE service startup timeout to 3600 seconds.
+- BE Service Startup Timeout Configuration
+ ```
+ spec:
+ beSpec:
+ startTimeout: 3600
+ ```
+### Liveness Probe Timeout
+- FE Service Liveness Timeout Configuration
+ ```
+ spec:
+ feSpec:
+ liveTimeout: 60
+ ```
+ The above configuration sets the FE service liveness timeout to 60 seconds.
+- BE Service Liveness Timeout Configuration
+ ```
+ spec:
+ beSpec:
+ liveTimeout: 60
+ ```
+ The above configuration sets the BE service liveness timeout to 60 seconds.
diff --git
a/docs/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
b/docs/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
index d814274d142..13c234004d4 100644
--- a/docs/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
+++ b/docs/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
@@ -192,6 +192,15 @@ spec:
resources:
requests:
storage: 300Gi
+ - mountPaths:
+ - /opt/apache-doris/be/storage
+ persistentVolumeClaimSpec:
+ # storageClassName: ${storageclass_name}
+ accessModes:
+ - ReadWriteOnce
+ resources:
+ requests:
+ storage: 300Gi
- persistentVolumeClaimSpec:
# storageClassName: ${storageclass_name}
accessModes:
@@ -200,7 +209,7 @@ spec:
requests:
storage: 500Gi
```
-In the above configuration, the log directory is mounted using a custom
storage configuration with 300Gi, while the cache directory is mounted using
the storage template with 500Gi.
+In the above configuration, the log directory is mounted using a custom
storage configuration with 300Gi, A 300Gi storage disk is mounted to the
directory `/opt/apache-doris/be/storage` used for WAL and StreamLoad imports,
etc. while the cache directory is mounted using the storage template with 500Gi.
:::tip Note
If the `mountPaths` array is empty, it indicates that the current storage
configuration is using the template configuration.
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/integrated-storage-compute/cluster-operation.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/integrated-storage-compute/cluster-operation.md
index 77ff18d1ba3..d291e5b93dd 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/integrated-storage-compute/cluster-operation.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/integrated-storage-compute/cluster-operation.md
@@ -408,3 +408,30 @@ admin set frontend config("disable_colocate_balance" =
"false");
admin set frontend config("disable_tablet_scheduler" = "false");
```
+## FE 使用 metadata_failure_recovery 模式启动
+当 FE 无法选主时，服务处于不可用状态时，可以通过选定一个拥有 `VLSN` 最大值的节点使用 `metadata_failure_recovery`
机制强制启动作为 master 节点，依此来恢复集群。
+
+### 容器环境下使用 recovery 模式启动
+
+1. 找到 `VLSN` 最大值所在的节点
+ k8s 下，FE 的 Pod 每次启动时会输出本节点的上最近 10 条 `VLSN` 记录，如下图所示：
+ ```
+ the annotations value:
+ the value not equal! debug
+ /opt/apache-doris/fe/doris-meta/bdb/je.info.0:19:2025-08-05 03:42:47.650
UTC INFO [fe_f35530c4_3ff1_48fe_80d1_cc8e32dbc942] Replica-feeder
fe_d8763579_92da_4d72_8c58_4e62b88bdff0 start stream at VLSN: 30
+ /opt/apache-doris/fe/doris-meta/bdb/je.info.0:21:2025-08-05 03:42:47.659
UTC INFO [fe_f35530c4_3ff1_48fe_80d1_cc8e32dbc942] Replica initialization
completed. Replica VLSN: -1 Heartbeat master commit VLSN: 49 DTVLSN:0 Replica
VLSN delta: 50
+ [Tue Aug 5 06:14:05 UTC 2025] start with meta run start_fe.sh with
additional options: '--console'
+ ```
+ 以上是一个实例集群 FE 启动时输出的 `VLSN` 记录，当前节点最大 `VLSN` 为 30 （日志输出前缀为 `start stream at
VLSN:`）。
+2. 选定最大值节点的 pod 作为使用 recovery 机制的节点。
+ 找到 `VLSN` 最大值所在节点的 pod 后，通过如下命令给 pod 添加需要使用 recovery 机制启动的注解。
+ ```
+ kubectl annotate pod {podName} "selectdb.com.doris/recovery=true"
+ ```
+ 当 Pod 再次重新启动后，当前节点会自动在启动命令中添加 ` --metadata_failure_recovery`， 服务以 recovery
模式启动。
+3. 服务正常后，必须删除第二步添加的 annotation，否则后面节点重启后会出现不可预期的行为。
+
+:::tip 提示
+1. 添加注解后，不可以通过 delete pod 的模式重启，这样会导致注解丢失。等待 kubelet 自动重启拉起，或者进入容器手动 kill 进程。
+2. 使用 `metadata_failure_recovery` 模式启动，FE 回放日志耗时会很长，在使用该模式启动之前请先修改 FE
服务的[启动超时时间](install-config-cluster.md#启动探测超时配置)，然后删除所有的 FE Pod 在进行
`metadata_failure_recovery` 启动。
+:::
\ No newline at end of file
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/integrated-storage-compute/install-config-cluster.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/integrated-storage-compute/install-config-cluster.md
index 375aee65a2e..cfc44ef06fe 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/integrated-storage-compute/install-config-cluster.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/integrated-storage-compute/install-config-cluster.md
@@ -993,4 +993,37 @@ spec:
:::tip 提示
`mountPath` 支持使用 `${DORIS_HOME}` 作为路径前缀。当 `mountPath` 使用 `${DORIS_HOME}`
作为前缀使用时，在 FE 容器中 `${DORIS_HOME}` 指代 `/opt/apache-doris/fe`; 在 BE 容器中
`${DORIS_HOME}` 指代 `/opt/apache-doris/be`。
-:::
\ No newline at end of file
+:::
+
+## 配置探测超时
+`DorisCluster`
为每种服务提供两种探测超时配置：启动探测超时配置，存活探测超时配置。当服务启动时间超过配置的启动探测超时时间时，则认定服务启动失败并重新启动服务。当服务超过存活探测时间没有响应时，Pod
会被自动重启。
+### 启动探测超时配置
+- FE 服务启动探测超时配置
+ ```
+ spec:
+ feSpec:
+ startTimeout: 3600
+ ```
+ 以上配置将 FE 的启动超时设置为 3600 秒。
+- BE 服务启动探测超时配置
+ ```
+ spec:
+ beSpec:
+ startTimeout: 3600
+ ```
+ 以上配置将 BE 的启动超时设置为 3600 秒。
+### 存活探测超时配置
+- FE 服务存活探测超时配置
+ ```
+ spec:
+ feSpec:
+ liveTimeout: 60
+ ```
+ 以上配置将 FE 的存活超时设置为 60 秒。
+- BE 服务存活探测超时配置
+ ```
+ spec:
+ beSpec:
+ liveTimeout: 60
+ ```
+ 以上配置将 BE 的存活超时设置为 60 秒。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
index 978a030616f..e3d270c9153 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
@@ -187,6 +187,15 @@ spec:
resources:
requests:
storage: 300Gi
+ - mountPaths:
+ - /opt/apache-doris/be/storage
+ persistentVolumeClaimSpec:
+ # storageClassName: ${storageclass_name}
+ accessModes:
+ - ReadWriteOnce
+ resources:
+ requests:
+ storage: 300Gi
- persistentVolumeClaimSpec:
# storageClassName: ${storageclass_name}
accessModes:
@@ -195,7 +204,7 @@ spec:
requests:
storage: 500Gi
```
-上述配置中，日志目录使用自定义的存储配置挂载 300Gi 的存储磁盘，而缓存目录则使用存储模板挂载 500Gi 的存储磁盘。
+上述配置中，日志目录使用自定义的存储配置挂载 300Gi 的存储磁盘，WAL 以及 StreamLoad 导入时使用的目录配置挂载 300Gi
的存储磁盘，而缓存目录则使用存储模板挂载 500Gi 的存储磁盘。
:::tip 提示
若 `mountPaths` 数组为空，则表示当前存储配置为模板配置。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/install/deploy-on-kubernetes/cluster-operation.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/install/deploy-on-kubernetes/cluster-operation.md
index d0e61cf9e30..98923aaad08 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/install/deploy-on-kubernetes/cluster-operation.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/install/deploy-on-kubernetes/cluster-operation.md
@@ -408,3 +408,30 @@ admin set frontend config("disable_colocate_balance" =
"false");
admin set frontend config("disable_tablet_scheduler" = "false");
```
+## FE 使用 metadata_failure_recovery 模式启动
+当 FE 无法选主时，服务处于不可用状态时，可以通过选定一个拥有 `VLSN` 最大值的节点使用 `metadata_failure_recovery`
机制强制启动作为 master 节点，依此来恢复集群。
+
+### 容器环境下使用 recovery 模式启动
+
+1. 找到 `VLSN` 最大值所在的节点
+ k8s 下，FE 的 Pod 每次启动时会输出本节点的上最近 10 条 `VLSN` 记录，如下图所示：
+ ```
+ the annotations value:
+ the value not equal! debug
+ /opt/apache-doris/fe/doris-meta/bdb/je.info.0:19:2025-08-05 03:42:47.650
UTC INFO [fe_f35530c4_3ff1_48fe_80d1_cc8e32dbc942] Replica-feeder
fe_d8763579_92da_4d72_8c58_4e62b88bdff0 start stream at VLSN: 30
+ /opt/apache-doris/fe/doris-meta/bdb/je.info.0:21:2025-08-05 03:42:47.659
UTC INFO [fe_f35530c4_3ff1_48fe_80d1_cc8e32dbc942] Replica initialization
completed. Replica VLSN: -1 Heartbeat master commit VLSN: 49 DTVLSN:0 Replica
VLSN delta: 50
+ [Tue Aug 5 06:14:05 UTC 2025] start with meta run start_fe.sh with
additional options: '--console'
+ ```
+ 以上是一个实例集群 FE 启动时输出的 `VLSN` 记录，当前节点最大 `VLSN` 为 30 （日志输出前缀为 `start stream at
VLSN:`）。
+2. 选定最大值节点的 pod 作为使用 recovery 机制的节点。
+ 找到 `VLSN` 最大值所在节点的 pod 后，通过如下命令给 pod 添加需要使用 recovery 机制启动的注解。
+ ```
+ kubectl annotate pod {podName} "selectdb.com.doris/recovery=true"
+ ```
+ 当 Pod 再次重新启动后，当前节点会自动在启动命令中添加 ` --metadata_failure_recovery`， 服务以 recovery
模式启动。
+3. 服务正常后，必须删除第二步添加的 annotation，否则后面节点重启后会出现不可预期的行为。
+
+:::tip 提示
+1. 添加注解后，不可以通过 delete pod 的模式重启，这样会导致注解丢失。等待 kubelet 自动重启拉起，或者进入容器手动 kill 进程。
+2. 使用 `metadata_failure_recovery` 模式启动，FE 回放日志耗时会很长，在使用该模式启动之前请先修改 FE
服务的[启动超时时间](install-config-cluster.md#启动探测超时配置)，然后删除所有的 FE Pod 在进行
`metadata_failure_recovery` 启动。
+:::
\ No newline at end of file
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/install/deploy-on-kubernetes/install-config-cluster.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/install/deploy-on-kubernetes/install-config-cluster.md
index 2058db7c623..2e658c8156d 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/install/deploy-on-kubernetes/install-config-cluster.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/install/deploy-on-kubernetes/install-config-cluster.md
@@ -906,7 +906,7 @@ Doris Operator 使用 `ConfigMap` 资源挂载 krb5.conf 文件，使用 `Secret
keytabPath: ${keytabPath}
```
${krb5ConfigMapName} 为包含要使用的 `krb5.conf` 文件的 ConfigMap
名称。${keytabSecretName} 为包含 keytab 文件的 Secret 名称。${keytabPath} 为 Secret
希望挂载到容器中的路径，这个路径是创建 catalog 时，通过 `hadoop.kerberos.keytab` 指定 keytab 的文件所在目录。创建
- catalog 请参考配置 [Hive
Catalog](../../lakehouse/datalake-analytics/hive.md#catalog-配置) 文档。
+ catalog 请参考配置 [Hive
Catalog](../../lakehouse/catalogs/hive-catalog.md#配置-catalog) 文档。
## 配置共享存储
Doris Operator 从 25.4.0 版本开始支持为多个组件的所有 Pod 挂载一个 `ReadWriteMany`
的共享存储。使用前请提前创建好共享存储 `PersistentVolume` 和 `PersistentVolumeClaim` 资源，在部署 Doris
集群之前按照如下配置 `DorisCluster` 资源：
@@ -926,3 +926,36 @@ spec:
:::tip 提示
`mountPath` 支持使用 `${DORIS_HOME}` 作为路径前缀。当 `mountPath` 使用 `${DORIS_HOME}`
作为前缀使用时，在 FE 容器中 `${DORIS_HOME}` 指代 `/opt/apache-doris/fe`; 在 BE 容器中
`${DORIS_HOME}` 指代 `/opt/apache-doris/be`。
:::
+
+## 配置探测超时
+`DorisCluster`
为每种服务提供两种探测超时配置：启动探测超时配置，存活探测超时配置。当服务启动时间超过配置的启动探测超时时间时，则认定服务启动失败并重新启动服务。当服务超过存活探测时间没有响应时，Pod
会被自动重启。
+### 启动探测超时配置
+- FE 服务启动探测超时配置
+ ```
+ spec:
+ feSpec:
+ startTimeout: 3600
+ ```
+ 以上配置将 FE 的启动超时设置为 3600 秒。
+- BE 服务启动探测超时配置
+ ```
+ spec:
+ beSpec:
+ startTimeout: 3600
+ ```
+ 以上配置将 BE 的启动超时设置为 3600 秒。
+### 存活探测超时配置
+- FE 服务存活探测超时配置
+ ```
+ spec:
+ feSpec:
+ liveTimeout: 60
+ ```
+ 以上配置将 FE 的存活超时设置为 60 秒。
+- BE 服务存活探测超时配置
+ ```
+ spec:
+ beSpec:
+ liveTimeout: 60
+ ```
+ 以上配置将 BE 的存活超时设置为 60 秒。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/install/deploy-on-kubernetes/integrated-storage-compute/cluster-operation.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/install/deploy-on-kubernetes/integrated-storage-compute/cluster-operation.md
index 77ff18d1ba3..d291e5b93dd 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/install/deploy-on-kubernetes/integrated-storage-compute/cluster-operation.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/install/deploy-on-kubernetes/integrated-storage-compute/cluster-operation.md
@@ -408,3 +408,30 @@ admin set frontend config("disable_colocate_balance" =
"false");
admin set frontend config("disable_tablet_scheduler" = "false");
```
+## FE 使用 metadata_failure_recovery 模式启动
+当 FE 无法选主时，服务处于不可用状态时，可以通过选定一个拥有 `VLSN` 最大值的节点使用 `metadata_failure_recovery`
机制强制启动作为 master 节点，依此来恢复集群。
+
+### 容器环境下使用 recovery 模式启动
+
+1. 找到 `VLSN` 最大值所在的节点
+ k8s 下，FE 的 Pod 每次启动时会输出本节点的上最近 10 条 `VLSN` 记录，如下图所示：
+ ```
+ the annotations value:
+ the value not equal! debug
+ /opt/apache-doris/fe/doris-meta/bdb/je.info.0:19:2025-08-05 03:42:47.650
UTC INFO [fe_f35530c4_3ff1_48fe_80d1_cc8e32dbc942] Replica-feeder
fe_d8763579_92da_4d72_8c58_4e62b88bdff0 start stream at VLSN: 30
+ /opt/apache-doris/fe/doris-meta/bdb/je.info.0:21:2025-08-05 03:42:47.659
UTC INFO [fe_f35530c4_3ff1_48fe_80d1_cc8e32dbc942] Replica initialization
completed. Replica VLSN: -1 Heartbeat master commit VLSN: 49 DTVLSN:0 Replica
VLSN delta: 50
+ [Tue Aug 5 06:14:05 UTC 2025] start with meta run start_fe.sh with
additional options: '--console'
+ ```
+ 以上是一个实例集群 FE 启动时输出的 `VLSN` 记录，当前节点最大 `VLSN` 为 30 （日志输出前缀为 `start stream at
VLSN:`）。
+2. 选定最大值节点的 pod 作为使用 recovery 机制的节点。
+ 找到 `VLSN` 最大值所在节点的 pod 后，通过如下命令给 pod 添加需要使用 recovery 机制启动的注解。
+ ```
+ kubectl annotate pod {podName} "selectdb.com.doris/recovery=true"
+ ```
+ 当 Pod 再次重新启动后，当前节点会自动在启动命令中添加 ` --metadata_failure_recovery`， 服务以 recovery
模式启动。
+3. 服务正常后，必须删除第二步添加的 annotation，否则后面节点重启后会出现不可预期的行为。
+
+:::tip 提示
+1. 添加注解后，不可以通过 delete pod 的模式重启，这样会导致注解丢失。等待 kubelet 自动重启拉起，或者进入容器手动 kill 进程。
+2. 使用 `metadata_failure_recovery` 模式启动，FE 回放日志耗时会很长，在使用该模式启动之前请先修改 FE
服务的[启动超时时间](install-config-cluster.md#启动探测超时配置)，然后删除所有的 FE Pod 在进行
`metadata_failure_recovery` 启动。
+:::
\ No newline at end of file
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/install/deploy-on-kubernetes/integrated-storage-compute/install-config-cluster.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/install/deploy-on-kubernetes/integrated-storage-compute/install-config-cluster.md
index 9e364ab0fe7..07af69e1c66 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/install/deploy-on-kubernetes/integrated-storage-compute/install-config-cluster.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/install/deploy-on-kubernetes/integrated-storage-compute/install-config-cluster.md
@@ -997,7 +997,7 @@ Doris Operator 使用 `ConfigMap` 资源挂载 krb5.conf 文件，使用 `Secret
keytabPath: ${keytabPath}
```
${krb5ConfigMapName} 为包含要使用的 `krb5.conf` 文件的 ConfigMap
名称。${keytabSecretName} 为包含 keytab 文件的 Secret 名称。${keytabPath} 为 Secret
希望挂载到容器中的路径，这个路径是创建 catalog 时，通过 `hadoop.kerberos.keytab` 指定 keytab 的文件所在目录。创建
- catalog 请参考配置 [Hive
Catalog](../../../lakehouse/datalake-analytics/hive.md#catalog-配置) 文档。
+ catalog 请参考配置 [Hive
Catalog](../../../lakehouse/catalogs/hive-catalog.md#配置-catalog) 文档。
## 配置共享存储
Doris Operator 从 25.4.0 版本开始支持为多个组件的所有 Pod 挂载一个 `ReadWriteMany`
的共享存储。使用前请提前创建好共享存储 `PersistentVolume` 和 `PersistentVolumeClaim` 资源，在部署 Doris
集群之前按照如下配置 `DorisCluster` 资源：
@@ -1016,4 +1016,37 @@ spec:
:::tip 提示
`mountPath` 支持使用 `${DORIS_HOME}` 作为路径前缀。当 `mountPath` 使用 `${DORIS_HOME}`
作为前缀使用时，在 FE 容器中 `${DORIS_HOME}` 指代 `/opt/apache-doris/fe`; 在 BE 容器中
`${DORIS_HOME}` 指代 `/opt/apache-doris/be`。
-:::
\ No newline at end of file
+:::
+
+## 配置探测超时
+`DorisCluster`
为每种服务提供两种探测超时配置：启动探测超时配置，存活探测超时配置。当服务启动时间超过配置的启动探测超时时间时，则认定服务启动失败并重新启动服务。当服务超过存活探测时间没有响应时，Pod
会被自动重启。
+### 启动探测超时配置
+- FE 服务启动探测超时配置
+ ```
+ spec:
+ feSpec:
+ startTimeout: 3600
+ ```
+ 以上配置将 FE 的启动超时设置为 3600 秒。
+- BE 服务启动探测超时配置
+ ```
+ spec:
+ beSpec:
+ startTimeout: 3600
+ ```
+ 以上配置将 BE 的启动超时设置为 3600 秒。
+### 存活探测超时配置
+- FE 服务存活探测超时配置
+ ```
+ spec:
+ feSpec:
+ liveTimeout: 60
+ ```
+ 以上配置将 FE 的存活超时设置为 60 秒。
+- BE 服务存活探测超时配置
+ ```
+ spec:
+ beSpec:
+ liveTimeout: 60
+ ```
+ 以上配置将 BE 的存活超时设置为 60 秒。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
index 919ddc5cc4b..41a83036650 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
@@ -187,6 +187,15 @@ spec:
resources:
requests:
storage: 300Gi
+ - mountPaths:
+ - /opt/apache-doris/be/storage
+ persistentVolumeClaimSpec:
+ # storageClassName: ${storageclass_name}
+ accessModes:
+ - ReadWriteOnce
+ resources:
+ requests:
+ storage: 300Gi
- persistentVolumeClaimSpec:
# storageClassName: ${storageclass_name}
accessModes:
@@ -195,7 +204,7 @@ spec:
requests:
storage: 500Gi
```
-上述配置中，日志目录使用自定义的存储配置挂载 300Gi 的存储磁盘，而缓存目录则使用存储模板挂载 500Gi 的存储磁盘。
+上述配置中，日志目录使用自定义的存储配置挂载 300Gi 的存储磁盘，WAL 以及 StreamLoad 导入时使用的目录配置挂载 300Gi
的存储磁盘，而缓存目录则使用存储模板挂载 500Gi 的存储磁盘。
:::tip 提示
若 `mountPaths` 数组为空，则表示当前存储配置为模板配置。
diff --git
a/versioned_docs/version-2.1/install/deploy-on-kubernetes/cluster-operation.md
b/versioned_docs/version-2.1/install/deploy-on-kubernetes/cluster-operation.md
index a23cdf68923..8d27dd84425 100644
---
a/versioned_docs/version-2.1/install/deploy-on-kubernetes/cluster-operation.md
+++
b/versioned_docs/version-2.1/install/deploy-on-kubernetes/cluster-operation.md
@@ -308,3 +308,31 @@ admin set frontend config("disable_balance" = "false");
admin set frontend config("disable_colocate_balance" = "false");
admin set frontend config("disable_tablet_scheduler" = "false");
```
+
+## Starting FE with `metadata_failure_recovery` Mode
+When the Frontend (FE) service is unable to elect a leader and becomes
unavailable, you can recover the cluster by selecting the node with the highest
`VLSN` and force-starting it as the master using the recovery mechanism.
+
+### Starting in Recovery Mode in a Containerized Environment
+1. Identify the node with the highest `VLSN`
+ In Kubernetes, each time an FE Pod starts, it outputs the last 10 `VLSN`
records of the node. An example is shown below:
+ ```
+ the annotations value:
+ the value not equal! debug
+ /opt/apache-doris/fe/doris-meta/bdb/je.info.0:19:2025-08-05 03:42:47.650
UTC INFO [fe_f35530c4_3ff1_48fe_80d1_cc8e32dbc942] Replica-feeder
fe_d8763579_92da_4d72_8c58_4e62b88bdff0 start stream at VLSN: 30
+ /opt/apache-doris/fe/doris-meta/bdb/je.info.0:21:2025-08-05 03:42:47.659
UTC INFO [fe_f35530c4_3ff1_48fe_80d1_cc8e32dbc942] Replica initialization
completed. Replica VLSN: -1 Heartbeat master commit VLSN: 49 DTVLSN:0 Replica
VLSN delta: 50
+ [Tue Aug 5 06:14:05 UTC 2025] start with meta run start_fe.sh with
additional options: '--console'
+ ```
+ In this example, the highest `VLSN` on the current node is 30, as
indicated by the log prefix `start stream at VLSN:`.
+2. Designate the Pod with the highest `VLSN` for recovery
+ After identifying the Pod corresponding to the node with the highest
`VLSN`, annotate it to enable the recovery mechanism:
+ ```
+ kubectl annotate pod {podName} "selectdb.com.doris/recovery=true"
+ ```
+ Upon restarting, the Pod will automatically append the
`--metadata_failure_recovery` flag to its startup command and start in recovery
mode.
+3. Remove the annotation after recovery
+ Once the FE service is running normally, make sure to remove the
annotation added in Step 2 to avoid unexpected behavior during future restarts.
+
+:::tip Note
+1. After adding the annotation, do not restart the Pod using kubectl delete
pod, as this will remove the annotation. Instead, allow kubelet to restart it
automatically or manually kill the process inside the container.
+2. Starting FE in `metadata_failure_recovery` mode can take a long time due to
extensive log replay. Before proceeding, increase the FE service's [startup
probe timeout](./install-config-cluster.md#startup-probe-timeout), and delete
all FE Pods before initiating the recovery startup.
+:::
\ No newline at end of file
diff --git
a/versioned_docs/version-2.1/install/deploy-on-kubernetes/install-config-cluster.md
b/versioned_docs/version-2.1/install/deploy-on-kubernetes/install-config-cluster.md
index c3ca98fc993..55a3e20ae36 100644
---
a/versioned_docs/version-2.1/install/deploy-on-kubernetes/install-config-cluster.md
+++
b/versioned_docs/version-2.1/install/deploy-on-kubernetes/install-config-cluster.md
@@ -753,7 +753,7 @@ The Doris Operator mounts the krb5.conf file using a
ConfigMap resource and moun
keytabSecretName: ${keytabSecretName}
keytabPath: ${keytabPath}
```
- ${krb5ConfigMapName}: Name of the ConfigMap containing the krb5.conf file.
${keytabSecretName}: Name of the Secret containing the keytab files.
${keytabPath}: The directory path in the container where the Secret mounts the
keytab files. This path should match the directory specified by
hadoop.kerberos.keytab when creating a catalog. For catalog configuration
details, refer to the [Hive Catalog
configuration](../../lakehouse/datalake-analytics/hive.md#catalog-configuration)
documentation.
+ ${krb5ConfigMapName}: Name of the ConfigMap containing the krb5.conf file.
${keytabSecretName}: Name of the Secret containing the keytab files.
${keytabPath}: The directory path in the container where the Secret mounts the
keytab files. This path should match the directory specified by
hadoop.kerberos.keytab when creating a catalog. For catalog configuration
details, refer to the [Hive Catalog
configuration](../../lakehouse/catalogs/hive-catalog.md#configuring-catalog)
documentation.
## Configure Shared Storage
As of version 25.4.0, the Doris Operator supports mounting shared storage with
the ReadWriteMany access mode to all pods across multiple components. Before
using this feature, ensure that the shared storage PersistentVolume and
PersistentVolumeClaim resources have been created. Configure the DorisCluster
resource as shown below before deploying the Doris cluster:
@@ -774,3 +774,36 @@ spec:
The `mountPath` parameter can use `${DORIS_HOME}` as a prefix. When
`${DORIS_HOME}` is used, it resolves to `/opt/apache-doris/fe` within FE
containers and `/opt/apache-doris/be` within BE containers.
:::
+## Configuring Probe Timeouts
+DorisCluster provides two types of probe timeout configurations for each
service: `startup probe timeout` and `liveness probe timeout`. If a service
fails to start within the specified startup timeout period, it is considered to
have failed and will be restarted.
+If a service becomes unresponsive for longer than the specified liveness
timeout, the corresponding Pod will be automatically restarted.
+
+### Startup Probe Timeout
+- FE Service Startup Timeout Configuration
+ ```
+ spec:
+ feSpec:
+ startTimeout: 3600
+ ```
+ The above configuration sets the FE service startup timeout to 3600
seconds.
+- BE Service Startup Timeout Configuration
+ ```
+ spec:
+ beSpec:
+ startTimeout: 3600
+ ```
+### Liveness Probe Timeout
+- FE Service Liveness Timeout Configuration
+ ```
+ spec:
+ feSpec:
+ liveTimeout: 60
+ ```
+ The above configuration sets the FE service liveness timeout to 60 seconds.
+- BE Service Liveness Timeout Configuration
+ ```
+ spec:
+ beSpec:
+ liveTimeout: 60
+ ```
+ The above configuration sets the BE service liveness timeout to 60 seconds.
diff --git
a/versioned_docs/version-3.0/install/deploy-on-kubernetes/integrated-storage-compute/cluster-operation.md
b/versioned_docs/version-3.0/install/deploy-on-kubernetes/integrated-storage-compute/cluster-operation.md
index 852de64a173..58cce65579c 100644
---
a/versioned_docs/version-3.0/install/deploy-on-kubernetes/integrated-storage-compute/cluster-operation.md
+++
b/versioned_docs/version-3.0/install/deploy-on-kubernetes/integrated-storage-compute/cluster-operation.md
@@ -310,3 +310,31 @@ admin set frontend config("disable_balance" = "false");
admin set frontend config("disable_colocate_balance" = "false");
admin set frontend config("disable_tablet_scheduler" = "false");
```
+
+## Starting FE with `metadata_failure_recovery` Mode
+When the Frontend (FE) service is unable to elect a leader and becomes
unavailable, you can recover the cluster by selecting the node with the highest
`VLSN` and force-starting it as the master using the recovery mechanism.
+
+### Starting in Recovery Mode in a Containerized Environment
+1. Identify the node with the highest `VLSN`
+ In Kubernetes, each time an FE Pod starts, it outputs the last 10 `VLSN`
records of the node. An example is shown below:
+ ```
+ the annotations value:
+ the value not equal! debug
+ /opt/apache-doris/fe/doris-meta/bdb/je.info.0:19:2025-08-05 03:42:47.650
UTC INFO [fe_f35530c4_3ff1_48fe_80d1_cc8e32dbc942] Replica-feeder
fe_d8763579_92da_4d72_8c58_4e62b88bdff0 start stream at VLSN: 30
+ /opt/apache-doris/fe/doris-meta/bdb/je.info.0:21:2025-08-05 03:42:47.659
UTC INFO [fe_f35530c4_3ff1_48fe_80d1_cc8e32dbc942] Replica initialization
completed. Replica VLSN: -1 Heartbeat master commit VLSN: 49 DTVLSN:0 Replica
VLSN delta: 50
+ [Tue Aug 5 06:14:05 UTC 2025] start with meta run start_fe.sh with
additional options: '--console'
+ ```
+ In this example, the highest `VLSN` on the current node is 30, as indicated
by the log prefix `start stream at VLSN:`.
+2. Designate the Pod with the highest `VLSN` for recovery
+ After identifying the Pod corresponding to the node with the highest
`VLSN`, annotate it to enable the recovery mechanism:
+ ```
+ kubectl annotate pod {podName} "selectdb.com.doris/recovery=true"
+ ```
+ Upon restarting, the Pod will automatically append the
`--metadata_failure_recovery` flag to its startup command and start in recovery
mode.
+3. Remove the annotation after recovery
+ Once the FE service is running normally, make sure to remove the annotation
added in Step 2 to avoid unexpected behavior during future restarts.
+
+:::tip Note
+1. After adding the annotation, do not restart the Pod using kubectl delete
pod, as this will remove the annotation. Instead, allow kubelet to restart it
automatically or manually kill the process inside the container.
+2. Starting FE in `metadata_failure_recovery` mode can take a long time due to
extensive log replay. Before proceeding, increase the FE service's [startup
probe timeout](./install-config-cluster.md#startup-probe-timeout), and delete
all FE Pods before initiating the recovery startup.
+:::
diff --git
a/versioned_docs/version-3.0/install/deploy-on-kubernetes/integrated-storage-compute/install-config-cluster.md
b/versioned_docs/version-3.0/install/deploy-on-kubernetes/integrated-storage-compute/install-config-cluster.md
index 3e4cd9a6451..b1d52b5ebf5 100644
---
a/versioned_docs/version-3.0/install/deploy-on-kubernetes/integrated-storage-compute/install-config-cluster.md
+++
b/versioned_docs/version-3.0/install/deploy-on-kubernetes/integrated-storage-compute/install-config-cluster.md
@@ -830,7 +830,7 @@ The Doris Operator mounts the krb5.conf file using a
ConfigMap resource and moun
keytabSecretName: ${keytabSecretName}
keytabPath: ${keytabPath}
```
- ${krb5ConfigMapName}: Name of the ConfigMap containing the krb5.conf file.
${keytabSecretName}: Name of the Secret containing the keytab files.
${keytabPath}: The directory path in the container where the Secret mounts the
keytab files. This path should match the directory specified by
hadoop.kerberos.keytab when creating a catalog. For catalog configuration
details, refer to the [Hive Catalog
configuration](../../../lakehouse/datalake-analytics/hive.md#catalog-configuration)
documentation.
+ ${krb5ConfigMapName}: Name of the ConfigMap containing the krb5.conf file.
${keytabSecretName}: Name of the Secret containing the keytab files.
${keytabPath}: The directory path in the container where the Secret mounts the
keytab files. This path should match the directory specified by
hadoop.kerberos.keytab when creating a catalog. For catalog configuration
details, refer to the [Hive Catalog
configuration](../../../lakehouse/catalogs/hive-catalog.md#configuring-catalog)
documentation.
## Configure Shared Storage
As of version 25.4.0, the Doris Operator supports mounting shared storage with
the ReadWriteMany access mode to all pods across multiple components. Before
using this feature, ensure that the shared storage PersistentVolume and
PersistentVolumeClaim resources have been created. Configure the DorisCluster
resource as shown below before deploying the Doris cluster:
@@ -850,3 +850,37 @@ spec:
:::tip Tip
The `mountPath` parameter can use `${DORIS_HOME}` as a prefix. When
`${DORIS_HOME}` is used, it resolves to `/opt/apache-doris/fe` within FE
containers and `/opt/apache-doris/be` within BE containers.
:::
+
+## Configuring Probe Timeouts
+DorisCluster provides two types of probe timeout configurations for each
service: `startup probe timeout` and `liveness probe timeout`. If a service
fails to start within the specified startup timeout period, it is considered to
have failed and will be restarted.
+If a service becomes unresponsive for longer than the specified liveness
timeout, the corresponding Pod will be automatically restarted.
+
+### Startup Probe Timeout
+- FE Service Startup Timeout Configuration
+ ```
+ spec:
+ feSpec:
+ startTimeout: 3600
+ ```
+ The above configuration sets the FE service startup timeout to 3600 seconds.
+- BE Service Startup Timeout Configuration
+ ```
+ spec:
+ beSpec:
+ startTimeout: 3600
+ ```
+### Liveness Probe Timeout
+- FE Service Liveness Timeout Configuration
+ ```
+ spec:
+ feSpec:
+ liveTimeout: 60
+ ```
+ The above configuration sets the FE service liveness timeout to 60 seconds.
+- BE Service Liveness Timeout Configuration
+ ```
+ spec:
+ beSpec:
+ liveTimeout: 60
+ ```
+ The above configuration sets the BE service liveness timeout to 60 seconds.
diff --git
a/versioned_docs/version-3.0/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
b/versioned_docs/version-3.0/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
index ceb0e9f9faa..d79a6ce8020 100644
---
a/versioned_docs/version-3.0/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
+++
b/versioned_docs/version-3.0/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
@@ -192,6 +192,15 @@ spec:
resources:
requests:
storage: 300Gi
+ - mountPaths:
+ - /opt/apache-doris/be/storage
+ persistentVolumeClaimSpec:
+ # storageClassName: ${storageclass_name}
+ accessModes:
+ - ReadWriteOnce
+ resources:
+ requests:
+ storage: 300Gi
- persistentVolumeClaimSpec:
# storageClassName: ${storageclass_name}
accessModes:
@@ -200,7 +209,7 @@ spec:
requests:
storage: 500Gi
```
-In the above configuration, the log directory is mounted using a custom
storage configuration with 300Gi, while the cache directory is mounted using
the storage template with 500Gi.
+In the above configuration, the log directory is mounted using a custom
storage configuration with 300Gi, A 300Gi storage disk is mounted to the
directory `/opt/apache-doris/be/storage` used for WAL and StreamLoad imports,
etc. while the cache directory is mounted using the storage template with 500Gi.
:::tip Note
If the `mountPaths` array is empty, it indicates that the current storage
configuration is using the template configuration.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
