This is an automated email from the ASF dual-hosted git repository.
zitadombi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ozone.git
The following commit(s) were added to refs/heads/master by this push:
new a59b573fdf3 HDDS-12197. Update documentation for all ozone debug tools
(#8868)
a59b573fdf3 is described below
commit a59b573fdf30f6ae3e061f84b607d28d3a9c3601
Author: Tejaskriya <[email protected]>
AuthorDate: Tue Aug 12 20:20:41 2025 +0530
HDDS-12197. Update documentation for all ozone debug tools (#8868)
---
.../docs/content/tools/{ => debug}/AuditParser.md | 4 +-
.../content/tools/{ => debug}/AuditParser.zh.md | 0
.../tools/{ => debug}/ContainerLogParser.md | 7 +-
.../tools/{ => debug}/ContainerLogParser.zh.md | 0
.../docs/content/tools/debug/DebugDatanode.md | 39 +++++++
hadoop-hdds/docs/content/tools/debug/DebugOm.md | 57 ++++++++++
.../docs/content/tools/debug/DebugReplicas.md | 119 +++++++++++++++++++++
.../docs/content/tools/{Debug.md => debug/Ldb.md} | 54 +++-------
.../content/tools/{Debug.zh.md => debug/Ldb.zh.md} | 0
.../docs/content/tools/debug/RatisLogParser.md | 39 +++++++
hadoop-hdds/docs/content/tools/debug/_index.md | 39 +++++++
11 files changed, 315 insertions(+), 43 deletions(-)
diff --git a/hadoop-hdds/docs/content/tools/AuditParser.md
b/hadoop-hdds/docs/content/tools/debug/AuditParser.md
similarity index 97%
rename from hadoop-hdds/docs/content/tools/AuditParser.md
rename to hadoop-hdds/docs/content/tools/debug/AuditParser.md
index ebc5c8ae6a8..316d5effa7e 100644
--- a/hadoop-hdds/docs/content/tools/AuditParser.md
+++ b/hadoop-hdds/docs/content/tools/debug/AuditParser.md
@@ -1,7 +1,9 @@
---
title: "Audit Parser"
date: 2018-12-17
-summary: Audit Parser tool can be used for querying the ozone audit logs.
+summary: Tool for querying the ozone audit logs.
+menu: debug
+weight: 6
---
<!---
Licensed to the Apache Software Foundation (ASF) under one or more
diff --git a/hadoop-hdds/docs/content/tools/AuditParser.zh.md
b/hadoop-hdds/docs/content/tools/debug/AuditParser.zh.md
similarity index 100%
rename from hadoop-hdds/docs/content/tools/AuditParser.zh.md
rename to hadoop-hdds/docs/content/tools/debug/AuditParser.zh.md
diff --git a/hadoop-hdds/docs/content/tools/ContainerLogParser.md
b/hadoop-hdds/docs/content/tools/debug/ContainerLogParser.md
similarity index 97%
rename from hadoop-hdds/docs/content/tools/ContainerLogParser.md
rename to hadoop-hdds/docs/content/tools/debug/ContainerLogParser.md
index dbc349ca6ed..bb61a39f547 100644
--- a/hadoop-hdds/docs/content/tools/ContainerLogParser.md
+++ b/hadoop-hdds/docs/content/tools/debug/ContainerLogParser.md
@@ -2,7 +2,9 @@
title: "Container Replica Debugger Tool"
date: 2025-05-19
jira: HDDS-12579
-summary: The tool processes container log files from Ozone datanodes to track
state transitions. It provides CLI commands for querying containers based on
different attributes.
+summary: Processing and querying container log files from Ozone datanodes to
track state transitions.
+menu: debug
+weight: 7
---
<!--
Licensed under the Apache License, Version 2.0 (the "License");
@@ -15,7 +17,8 @@ summary: The tool processes container log files from Ozone
datanodes to track st
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
-# Container Replica Debugger Tool
+
+The tool processes container log files from Ozone datanodes to track state
transitions. It provides CLI commands for querying containers based on
different attributes.
## Background
Containers are the most important part of ozone. Most of the ozone operations
happen on them. Containers in ozone go through different states in their
lifecycle.
diff --git a/hadoop-hdds/docs/content/tools/ContainerLogParser.zh.md
b/hadoop-hdds/docs/content/tools/debug/ContainerLogParser.zh.md
similarity index 100%
rename from hadoop-hdds/docs/content/tools/ContainerLogParser.zh.md
rename to hadoop-hdds/docs/content/tools/debug/ContainerLogParser.zh.md
diff --git a/hadoop-hdds/docs/content/tools/debug/DebugDatanode.md
b/hadoop-hdds/docs/content/tools/debug/DebugDatanode.md
new file mode 100644
index 00000000000..c870dd7fad6
--- /dev/null
+++ b/hadoop-hdds/docs/content/tools/debug/DebugDatanode.md
@@ -0,0 +1,39 @@
+---
+title: "Debug Datanode"
+date: 2025-07-28
+summary: Debug commands related to datanode.
+menu: debug
+weight: 3
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Debug commands related to datanode. Currently, only container replica related
commands are available.
+Following is the usage and the subcommands available under the `ozone debug
datanode container` command:
+
+```bash
+Usage: ozone debug datanode container [-hV] [--verbose] [COMMAND]
+Container replica specific operations to be executed on datanodes only
+ -h, --help Show this help message and exit.
+ -V, --version Print version information and exit.
+ --verbose More verbose output. Show the stack trace of the errors.
+Commands:
+ list Show container info of all container replicas on datanode
+ info Show container info of a container replica on datanode
+ export Export one container to a tarball
+ inspect Check the metadata of all container replicas on this datanode.
+```
diff --git a/hadoop-hdds/docs/content/tools/debug/DebugOm.md
b/hadoop-hdds/docs/content/tools/debug/DebugOm.md
new file mode 100644
index 00000000000..28456a9f6b8
--- /dev/null
+++ b/hadoop-hdds/docs/content/tools/debug/DebugOm.md
@@ -0,0 +1,57 @@
+---
+title: "Debug OM"
+date: 2025-07-28
+summary: Debug commands related to OM.
+menu: debug
+weight: 2
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Debug commands related to OM.
+It has the following subcommands:
+## generate-compaction-dag
+Creates a DAG image of the current compaction log of an om.db instance. It is
downloaded to the specified location.
+```bash
+Usage: ozone debug om generate-compaction-dag [-hV] [--verbose] --db=<dbPath>
+ -o=<imageLocation>
+Create an image of the current compaction log DAG. This command is an offline
+command. i.e., it can run on any instance of om.db and does not require OM to
+be up.
+ --db=<dbPath> Path to OM RocksDB
+ -h, --help Show this help message and exit.
+ -o, --output-file=<imageLocation>
+ Path to location at which image will be downloaded.
+ Should include the image file name with ".png"
+ extension.
+ -V, --version Print version information and exit.
+ --verbose More verbose output. Show the stack trace of the errors.
+```
+
+## prefix
+Parses the contents of a prefix.
+```bash
+Usage: ozone debug om prefix [--verbose] --bucket=<bucket> --db=<dbPath>
+ --path=<filePath> --volume=<volume>
+Parse prefix contents
+ --bucket=<bucket> bucket name
+ --db=<dbPath> Path to OM RocksDB
+ --path=<filePath> prefixFile Path
+ --verbose More verbose output. Show the stack trace of the
+ errors.
+ --volume=<volume> volume name
+```
diff --git a/hadoop-hdds/docs/content/tools/debug/DebugReplicas.md
b/hadoop-hdds/docs/content/tools/debug/DebugReplicas.md
new file mode 100644
index 00000000000..0072638cb37
--- /dev/null
+++ b/hadoop-hdds/docs/content/tools/debug/DebugReplicas.md
@@ -0,0 +1,119 @@
+---
+title: "Debug Replicas"
+date: 2025-07-28
+summary: Debug commands for replica-related issues.
+menu: debug
+weight: 4
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Debug commands for retrieving information and performing various checks on the
key replicas in datanodes.
+
+```bash
+Usage: ozone debug replicas [--verbose] [COMMAND]
+Debug commands for replica-related issues, retrieving replica information from
+the OM and performing checks over the network against a running cluster.
+ --verbose More verbose output. Show the stack trace of the errors.
+Commands:
+ chunk-info Returns chunk location information about an existing key
+ verify Run checks to verify data across replicas. By default prints only
+ the keys with failed checks.
+```
+
+## chunk-info
+For a given URI of a key, the command returns all the chunks' location
information.
+
+```bash
+Usage: ozone debug replicas chunk-info [-hV] [--verbose] <value>
+Returns chunk location information about an existing key
+ <value> URI of the key (format: volume/bucket/key).
+ Ozone URI could either be a full URI or short URI.
+ Full URI should start with o3://, in case of non-HA
+ clusters it should be followed by the host name and
+ optionally the port number. In case of HA clusters
+ the service id should be used. Service id provides a
+ logical name for multiple hosts and it is defined
+ in the property ozone.om.service.ids.
+ Example of a full URI with host name and port number
+ for a key:
+ o3://omhostname:9862/vol1/bucket1/key1
+ With a service id for a volume:
+ o3://omserviceid/vol1/
+ Short URI should start from the volume.
+ Example of a short URI for a bucket:
+ vol1/bucket1
+ Any unspecified information will be identified from
+ the config files.
+
+ -h, --help Show this help message and exit.
+ -V, --version Print version information and exit.
+ --verbose More verbose output. Show the stack trace of the errors.
+```
+
+## verify
+Verify data across replicas. There are multiple checks available, which can be
selected using the command line options:
+
+```bash
+Usage: ozone debug replicas verify [-hV] [--all-results] [--verbose]
+
[--container-cache-size=<containerCacheSize>]
+ [-id=<scmServiceId>] [--scm=<scm>]
+ ([--checksums] [--block-existence]
+ [--container-state]) <uri>
+Run checks to verify data across replicas. By default prints only the keys with
+failed checks.
+ <uri> Ozone URI could either be a full URI or short URI.
+ Full URI should start with o3://, in case of non-HA
+ clusters it should be followed by the host name and
+ optionally the port number. In case of HA clusters
+ the service id should be used. Service id provides a
+ logical name for multiple hosts and it is defined
+ in the property ozone.om.service.ids.
+ Example of a full URI with host name and port number
+ for a key:
+ o3://omhostname:9862/vol1/bucket1/key1
+ With a service id for a volume:
+ o3://omserviceid/vol1/
+ Short URI should start from the volume.
+ Example of a short URI for a bucket:
+ vol1/bucket1
+ Any unspecified information will be identified from
+ the config files.
+
+ --all-results Print results for all passing and failing keys
+ --block-existence Check for block existence on datanodes.
+ --checksums Do client side data checksum validation of all
+ replicas.
+ --container-cache-size=<containerCacheSize>
+ Size (in number of containers) of the in-memory cache
+ for container state verification
+ '--container-state'. Default is 1 million
+ containers (which takes around 43MB). Value must be
+ greater than zero, otherwise the default of 1
+ million is considered. Note: This option is ignored
+ if '--container-state' option is not used.
+ --container-state Check the container and replica states. Containers in
+ [DELETING, DELETED] states, or it's replicas in
+ [DELETED, UNHEALTHY, INVALID] states fail the
check.
+ -h, --help Show this help message and exit.
+ -id, --service-id=<scmServiceId>
+ ServiceId of SCM HA Cluster
+ --scm=<scm> The destination scm (host:port)
+ -V, --version Print version information and exit.
+ --verbose More verbose output. Show the stack trace of the
+ errors.
+```
diff --git a/hadoop-hdds/docs/content/tools/Debug.md
b/hadoop-hdds/docs/content/tools/debug/Ldb.md
similarity index 86%
rename from hadoop-hdds/docs/content/tools/Debug.md
rename to hadoop-hdds/docs/content/tools/debug/Ldb.md
index 79c11f777ef..a6190b6c6d7 100644
--- a/hadoop-hdds/docs/content/tools/Debug.md
+++ b/hadoop-hdds/docs/content/tools/debug/Ldb.md
@@ -1,7 +1,9 @@
---
-title: "Ozone Debug"
+title: "LDB Tool"
date: 2024-10-14
-summary: Ozone Debug command can be used for all the debugging related tasks.
+summary: Parsing rocksDB and debugging related issues.
+menu: debug
+weight: 1
---
<!---
Licensed to the Apache Software Foundation (ASF) under one or more
@@ -20,43 +22,6 @@ summary: Ozone Debug command can be used for all the
debugging related tasks.
limitations under the License.
-->
-Ozone Debug command (`ozone debug`) is a collection of developer tools
intended to help in debugging and get more information of various components of
ozone.
-
-```bash
-Usage: ozone debug [-hV] [--verbose] [-conf=<configurationPath>]
- [-D=<String=String>]... [COMMAND]
-Developer tools for Ozone Debug operations
- -conf=<configurationPath>
- path to the ozone configuration file
- -D, --set=<String=String>
- Map of (configuration_key,configuration_value) for any
- configuration overrides
- -h, --help Show this help message and exit.
- -V, --version Print version information and exit.
- --verbose More verbose output. Show the stack trace of the errors.
-Commands:
- chunkinfo returns chunk location information about an
- existing key
- print-log-dag, pld Create an image of the current compaction log DAG
- in OM.
- find-missing-padding, fmp List all keys with any missing padding, optionally
- limited to a volume/bucket/key URI.
- recover recover the lease of a specified file. Make sure
- to specify file system scheme if ofs:// is not
- the default.
- prefix Parse prefix contents
- ldb Parse rocksdb file content
- read-replicas Reads every replica for all the blocks associated
- with a given key.
- container Container replica specific operations to be
- executed on datanodes only
- ratislogparser Shell of printing Ratis Log in understandable text
-```
-For more detailed usage see the output of `--help` for each of the subcommands.
-
-
-## ozone debug ldb
-
Ozone heavily uses RocksDB for storing metadata. This tool helps parse the
contents of RocksDB belonging to Ozone Roles.
Supported DB's : Ozone Manager (om.db) , StorageContainerManager (scm.db),
Datanode/Container (container.db)
Below is the usage:
@@ -69,6 +34,7 @@ Commands:
scan Parse specified metadataTable
list_column_families, ls list all column families in db.
value-schema Schema of value in metadataTable
+ checkpoint Create checkpoint for specified db
```
### list_column_families command
@@ -470,4 +436,12 @@ $ ozone debug ldb --db=/data/metadata/om.db value-schema
--cf=keyTable
"objectID" : "long"
}
}
-```
\ No newline at end of file
+```
+
+### checkpoint command
+
+`checkpoint` command takes a checkpoint of a rocksdb, at a provided path.
+
+```bash
+$ ozone debug ldb --db=/data/metadata/om.db checkpoint
--output=/tmp/om-checkpoint
+```
diff --git a/hadoop-hdds/docs/content/tools/Debug.zh.md
b/hadoop-hdds/docs/content/tools/debug/Ldb.zh.md
similarity index 100%
rename from hadoop-hdds/docs/content/tools/Debug.zh.md
rename to hadoop-hdds/docs/content/tools/debug/Ldb.zh.md
diff --git a/hadoop-hdds/docs/content/tools/debug/RatisLogParser.md
b/hadoop-hdds/docs/content/tools/debug/RatisLogParser.md
new file mode 100644
index 00000000000..88437130406
--- /dev/null
+++ b/hadoop-hdds/docs/content/tools/debug/RatisLogParser.md
@@ -0,0 +1,39 @@
+---
+title: "Ratis Log Parser"
+date: 2025-07-28
+summary: Parse ratis logs to be in human understandable form.
+menu: debug
+weight: 5
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+The Ratis log parser tool takes a segment file as input, and give the output
in a human-readable format.
+It has the following subcommands, which can be used to parse the Ratis logs of
different components:
+
+```bash
+Usage: ozone debug ratislogparser [-hV] [--verbose] [COMMAND]
+Shell of printing Ratis Log in understandable text
+ -h, --help Show this help message and exit.
+ -V, --version Print version information and exit.
+ --verbose More verbose output. Show the stack trace of the errors.
+Commands:
+ datanode dump datanode segment file
+ generic dump generic ratis segment file
+ om dump om ratis segment file
+ scm dump scm ratis segment file
+```
diff --git a/hadoop-hdds/docs/content/tools/debug/_index.md
b/hadoop-hdds/docs/content/tools/debug/_index.md
new file mode 100644
index 00000000000..01c6dd2e9e1
--- /dev/null
+++ b/hadoop-hdds/docs/content/tools/debug/_index.md
@@ -0,0 +1,39 @@
+---
+title: "Debug Tools"
+date: "2025-07-28"
+summary: Ozone Debug command can be used for all the debugging related tasks.
+menu: tools
+---
+
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Ozone Debug command (`ozone debug`) is a collection of developer tools
intended to help in debugging and get more information of various components of
ozone.
+
+It includes the following tools:
+
+ * **ldb** - Tools to debug RocksDB related issues.
+ * **om** - Debug commands related to OM.
+ * **datanode** - Debug commands related to Datanode.
+ * **replicas** - Debug commands for key replica related issues.
+ * **ratislogparser** - Shell of printing Ratis Log in understandable text
+ * **auditparser** - A tool to parse and query Ozone audit logs.
+ * **log** - A tool to parse and provide insights on logs, currently
supports only the datanode's container logs.
+ * **checknative** - Checks if native libraries are loaded
+ * **version** - Show internal version of Ozone components
+
+For more information see the following subpages:
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
