This is an automated email from the ASF dual-hosted git repository.
weichiu pushed a commit to branch HDDS-9225-website-v2
in repository https://gitbox.apache.org/repos/asf/ozone-site.git
The following commit(s) were added to refs/heads/HDDS-9225-website-v2 by this
push:
new 756703fb6 HDDS-14502. [Website v2] [Docs] [Administrator Guide]
Replacing Ozone Manager Disks (#286)
756703fb6 is described below
commit 756703fb65e56ab6a70e1a586d9f45def5f7e4d7
Author: Gargi Jaiswal <[email protected]>
AuthorDate: Tue Jan 27 11:42:18 2026 +0530
HDDS-14502. [Website v2] [Docs] [Administrator Guide] Replacing Ozone
Manager Disks (#286)
---
.../04-disk-replacement/01-ozone-manager.md | 106 ++++++++++++++++++++-
1 file changed, 105 insertions(+), 1 deletion(-)
diff --git
a/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md
b/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md
index 0ee974540..507dfd760 100644
---
a/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md
+++
b/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md
@@ -4,4 +4,108 @@ sidebar_label: Ozone Manager
# Replacing Ozone Manager Disks
-**TODO:** File a subtask under
[HDDS-9859](https://issues.apache.org/jira/browse/HDDS-9859) and complete this
page or section.
+**Audience:** Cluster Administrators
+
+**Prerequisites:** Familiarity with Ozone cluster administration and Linux
system administration.
+
+---
+
+## 1. Overview
+
+When a disk containing the Ozone Manager (OM) metadata directory fails, proper
recovery procedures are critical to maintain cluster availability and prevent
data loss. This document provides comprehensive, step-by-step guidance for
safely replacing failed disks on OM nodes, with distinct procedures for
standalone and High-Availability (HA) configurations. Following these
procedures correctly ensures minimal downtime and maintains the integrity of
your Ozone cluster's metadata.
+
+- **Purpose**
+This guide provides the steps required to safely replace a failed disk on an
Ozone Manager (OM) node.
+
+- **Impact of OM Disk Failure**
+The OM disk is critical as it stores the RocksDB database containing the
entire object store namespace (volumes, buckets, keys) and block locations. A
failure of this disk can lead to metadata loss if not handled correctly.
+
+- **Crucial Distinction: HA vs. Non-HA**
+The recovery procedure depends entirely on whether your OM is a single,
standalone instance or part of a High-Availability (HA) Ratis-based quorum. The
HA procedure is significantly safer and results in no cluster downtime. Running
a standalone OM is not recommended for production environments.
+
+---
+
+## 2. Pre-flight Checks
+
+> **Important:** The following steps assume Ozone configuration files are
intact. If the configuration files are corrupt, they need to be restored from a
backup (if applicable) before proceeding with the disk replacement procedure.
+
+Before starting, the administrator should:
+
+1. **Identify the Failed Disk:** Use system tools (`dmesg`, `smartctl`, etc.)
to confirm which disk has failed and its mount point.
+
+2. **Identify OM Directories:** Check your `ozone-site.xml` to confirm which
Ozone directories are on the failed disk. The most important ones are:
+ - `ozone.om.db.dirs`: The primary OM metadata database (RocksDB). This
directory stores the entire object store namespace.
+ - `ozone.om.ratis.storage.dir`: The Ratis storage directory (if configured
on a separate disk). This directory stores Ratis metadata like logs. If not
explicitly configured, it falls back to `ozone.metadata.dirs`. For production
environments, it is recommended to configure this on a separate, fast disk
(preferably SSD) for better performance.
+
+3. **Prepare the Replacement Disk:** Have a new, healthy disk physically
installed, formatted, and mounted on the system at the same path as the failed
disk. Ensure it has the correct ownership and permissions for the user that
runs the OM process. The default permissions for OM metadata directories are
**750** (configurable via `ozone.om.db.dirs.permissions`).
+
+---
+
+## 3. Procedure for a Standalone (Non-HA) Ozone Manager
+
+This is a high-risk, manual disaster recovery process that will require
cluster downtime.
+
+1. **STOP THE ENTIRE CLUSTER:** Shut down all clients, Datanodes, SCM, and the
Ozone Manager to prevent any further state changes.
+
+2. **Attempt Data Recovery:** If the failed disk is still partially readable,
make a best-effort attempt to copy the contents of the `ozone.om.db.dirs`
directory to a safe, temporary location.
+
+3. **If Recovery Fails, Restore from Backup:** If the OM database files are
unrecoverable, you must restore from your most recent backup. This document
does not cover the backup process itself, but it is the only path to recovery
in this scenario.
+
+4. **Replace and Configure Disk:** Physically replace the hardware and ensure
the new, empty disk is mounted at the correct path defined in
`ozone.om.db.dirs`.
+
+5. **Restore Metadata:** Copy the recovered data **(from Step 2)** or the
restored backup data **(from Step 3)** to the `ozone.om.db.dirs` path on the
new disk.
+
+6. **Restart and Verify:**
+ - Start the SCM and Ozone Manager services.
+ - Once the OM is running, start the Datanodes.
+ - Run `ozone sh volume list` and other basic commands to verify that the
namespace is intact and the cluster is operational.
+
+---
+
+## 4. Procedure for an HA (Ratis-based) Ozone Manager
+
+This procedure is much safer, leverages the built-in redundancy of the OM HA
cluster, and does not require full cluster downtime.
+
+### Bootstrap Procedure
+
+1. **STOP THE FAILED OM INSTANCE:** On the node with the failed disk, stop
only the Ozone Manager process. The other OMs will continue operating, and one
of them will remain the leader, serving client requests.
+
+2. **Replace and Configure Disk:** Physically replace the hardware. Mount the
new, empty disk at the path defined in `ozone.om.db.dirs` and ensure it has the
correct ownership and permissions. If `ozone.om.ratis.storage.dir` was also on
the failed disk, ensure it is properly configured on the new disk as well.
+
+3. **Verify Configuration:** Before proceeding, ensure that all existing OMs
have their `ozone-site.xml` configuration files updated with the configuration
details of the OM being recovered (nodeId, address, port, etc.). The bootstrap
process will verify this by checking all OMs' on-disk configurations. If an
existing OM does not have updated configs, it can crash when bootstrap is
initiated.
+
+4. **RE-INITIALIZE THE OM:**
+ - This is the key step. Since the local database is gone, the OM needs to
be "reborn" by getting a complete copy of the latest state from the current OM
leader.
+ - Run the OM with the `--bootstrap` parameter to trigger this process. For
example:
+
+ ```bash
+ ozone om --bootstrap
+ ```
+
+ This command will detect that the OM belongs to an existing Ratis ring
but has no local state.
+
+5. **START THE OM AND MONITOR:**
+ - Start the Ozone Manager service on the repaired node (if not already
started by the bootstrap command).
+ - Tail the OM's log file (`.log` and `.out`). You should see messages
indicating that it is connecting to the OM HA ring and that a "snapshot" is
being installed. This means the current OM leader is streaming the entire
metadata database to this new follower.
+ - This process can take some time, depending on the size of your metadata.
+
+6. **VERIFY:** Once the snapshot installation is complete, the OM will finish
starting, join the Ratis ring as a follower, and begin receiving live updates.
+ - You can check the OM Web UI on any of the OM nodes. The list of peers
should now show all OMs as healthy.
+ - Alternatively, use the command `ozone admin om roles -id <OM_SERVICE_ID>`
to verify that all OMs are showing as LEADER or FOLLOWER.
+ - The cluster is now back at full redundancy and the procedure is complete.
+
+---
+
+## 5. Additional Considerations
+
+1. **Disk Space Requirements for Snapshots**
+ - **Critical:** When an Ozone Manager (OM) acts as a follower in an HA
setup, it downloads snapshot tarballs from the leader to its local metadata
directory.
+ - Always ensure your OM disks have **at least 2x the current OM database
size** to accommodate the existing data and incoming snapshots.
+ - This prevents disk space issues and maintains cluster stability during
recovery operations.
+
+2. **Separating Ratis Logs**
+ - If you have configured `ozone.om.ratis.storage.dir` on a separate,
dedicated disk (recommended for performance), a failure of that disk would
follow the same HA recovery procedure.
+ - The OM would automatically rebuild its Ratis logs from the other members
of the ring when it starts.
+
+3. **Disk Monitoring**
+ - This procedure highlights the importance of actively monitoring disk
health (`smartd`, etc.) to replace disks proactively before a catastrophic
failure.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
