This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/ozone-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 823c8c6e [auto] Generated docs from Apache Ozone master
278bcdda947d28b15126a248989f8df530283812
823c8c6e is described below
commit 823c8c6e2f9726f374d7e86b3e44fe261b9b4656
Author: Github Actions <[email protected]>
AuthorDate: Wed Jun 25 12:46:51 2025 +0000
[auto] Generated docs from Apache Ozone master
278bcdda947d28b15126a248989f8df530283812
---
docs/edge/concept/datanodes.html | 2 +-
docs/edge/design.html | 10 +
docs/edge/design/container-reconciliation.html | 925 +++++++++++++++++++++++++
docs/edge/design/index.xml | 7 +
docs/edge/en/sitemap.xml | 5 +-
docs/edge/index.xml | 7 +
docs/edge/interface/ofs.html | 2 +-
docs/edge/zh/concept/datanodes.html | 2 +-
docs/edge/zh/interface/ofs.html | 2 +-
docs/edge/zh/sitemap.xml | 2 +-
10 files changed, 958 insertions(+), 6 deletions(-)
diff --git a/docs/edge/concept/datanodes.html b/docs/edge/concept/datanodes.html
index 15426f74..cab4ec44 100644
--- a/docs/edge/concept/datanodes.html
+++ b/docs/edge/concept/datanodes.html
@@ -630,7 +630,7 @@ different service are critical to scaling Ozone.</p>
<footer class="footer">
<div class="container">
<span class="small text-muted">
- Version: 2.1.0-SNAPSHOT, Last Modified: September 23, 2024 <a
class="hide-child link primary-color"
href="https://github.com/apache/ozone/commit/0fcb64564d88defc8d2c0ab585bff5717c26632b";>0fcb64564d</a>
+ Version: 2.1.0-SNAPSHOT, Last Modified: December 18, 2024 <a
class="hide-child link primary-color"
href="https://github.com/apache/ozone/commit/7bcc943077097125e16d705ba1366139ba4c5871";>7bcc943077</a>
</span>
</div>
</footer>
diff --git a/docs/edge/design.html b/docs/edge/design.html
index 15dfdcc4..25dc0b62 100644
--- a/docs/edge/design.html
+++ b/docs/edge/design.html
@@ -527,6 +527,16 @@ s=d.getElementsByTagName('script')[0];
<td>accepted</td>
</tr>
+ <tr>
+ <td>2024-01-29</td>
+ <td>
+ <a
href="https://issues.apache.org/jira/browse/HDDS-10239";>HDDS-10239</a>
+ </td>
+ <td><a
href="./design/container-reconciliation.html">Container Reconciliation</a></td>
+ <td>Allow Datanodes to reconcile mismatched
container contents regardless of their state.</td>
+ <td>draft</td>
+ </tr>
+
<tr>
<td>2023-05-17</td>
<td>
diff --git a/docs/edge/design/container-reconciliation.html
b/docs/edge/design/container-reconciliation.html
new file mode 100644
index 00000000..09756405
--- /dev/null
+++ b/docs/edge/design/container-reconciliation.html
@@ -0,0 +1,925 @@
+
+
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <meta name="description" content="Apache Ozone Documentation">
+
+ <title>Documentation for Apache Ozone</title>
+
+
+ <link href="../css/bootstrap.min.css" rel="stylesheet">
+
+
+ <link href="../css/ozonedoc.css" rel="stylesheet">
+
+
+
+ <link href="../swagger-resources/swagger-ui.css" rel="stylesheet">
+
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
+ </head>
+
+
+ <body>
+
+<nav class="navbar navbar-inverse navbar-fixed-top">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed"
data-toggle="collapse" data-target="#sidebar" aria-expanded="false"
aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="../index.html" class="navbar-left ozone-logo">
+ <img src="../ozone-logo-small.png"/>
+ </a>
+ <a class="navbar-brand hidden-xs" href="../index.html">
+ Apache Ozone/HDDS Documentation
+ </a>
+ <a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="https://github.com/apache/ozone";>Source</a></li>
+ <li><a href="https://ozone.apache.org";>Apache Ozone</a></li>
+ <li><a href="https://apache.org";>ASF</a></li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+ <div class="container-fluid">
+ <div class="row">
+
+<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
+ <ul class="nav nav-sidebar">
+
+
+
+ <li class="">
+
+ <a href="../index.html">
+
+
+
+ <span>Overview</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../start.html">
+
+
+
+ <span>Getting Started</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+ <a href="../concept.html">
+
+ <span>Architecture</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../concept/overview.html">Overview</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/ozonemanager.html">Ozone
Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a
href="../concept/storagecontainermanager.html">Storage Container Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/containers.html">Containers</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/datanodes.html">Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/recon.html">Recon</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/networkports.html">Network
Ports</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../feature.html">
+
+ <span>Features</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a
href="../feature/decommission.html">Decommissioning</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/om-ha.html">OM High
Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/erasurecoding.html">Ozone
Erasure Coding</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/snapshot.html">Ozone
Snapshot</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/scm-ha.html">SCM High
Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a
href="../feature/streaming-write-pipeline.html">Streaming Write Pipeline</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/dn-merge-rocksdb.html">Merge
Container RocksDB in DN</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/prefixfso.html">Prefix based
File System Optimization</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/topology.html">Topology
awareness</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/quota.html">Quota in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/recon.html">Recon Server</a>
+
+ </li>
+
+ <li class="">
+
+ <a
href="../feature/observability.html">Observability</a>
+
+ </li>
+
+ <li class="">
+
+ <a
href="../feature/nonrolling-upgrade.html">Non-Rolling Upgrades and
Downgrades</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/s3-multi-tenancy.html">
+
+ <span>S3 Multi-Tenancy</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+ <a
href="../feature/s3-multi-tenancy-setup.html">Setup</a>
+ </li>
+
+ <li class="">
+ <a
href="../feature/s3-tenant-commands.html">Tenant commands</a>
+ </li>
+
+ <li class="">
+ <a
href="../feature/s3-multi-tenancy-access-control.html">Access Control</a>
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="">
+
+ <a
href="../feature/reconfigurability.html">Reconfigurability</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../integration.html">
+
+ <span>Application Integrations</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../integration/distcp.html">Hadoop
DistCp</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../integration/hive.html">Hive</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../integration/impala.html">Impala</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../interface.html">
+
+ <span>Client Interfaces</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../interface/ofs.html">Ofs (Hadoop
compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/o3fs.html">O3fs (Hadoop
compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/s3.html">S3 Protocol</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/cli.html">Command Line
Interface</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/reconapi.html">Recon API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/javaapi.html">Java API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/python.html">Accessing Apache
Ozone from Python</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/csi.html">CSI Protocol</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/httpfs.html">HttpFS
Gateway</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/native-cpp.html">Native
C/C++ Client Access to Ozone</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../security.html">
+
+ <span>Security</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../security/secureozone.html">Securing
Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingtde.html">Transparent
Data Encryption</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/gdpr.html">GDPR in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a
href="../security/securingdatanodes.html">Securing Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a
href="../security/securingozonehttp.html">Securing HTTP</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securings3.html">Securing
S3</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securityacls.html">Ozone
ACLs</a>
+
+ </li>
+
+ <li class="">
+
+ <a
href="../security/securitywithranger.html">Apache Ranger</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../tools.html">
+
+
+
+ <span>Tools</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../recipe.html">
+
+
+
+ <span>Recipes</span>
+ </a>
+ </li>
+
+
+ <li><a href="../design.html"><span><b>Design docs</b></span></a></li>
+ <li class="visible-xs"><a href="#">References</a>
+ <ul class="nav">
+ <li><a href="https://github.com/apache/ozone";><span class="glyphicon
glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
+ <li><a href="https://ozone.apache.org";><span class="glyphicon
glyphicon-new-window" aria-hidden="true"></span> Apache Ozone</a></li>
+ <li><a href="https://apache.org";><span class="glyphicon
glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
+ </ul></li>
+ </ul>
+
+</div>
+
+ <div class="col-sm-9 col-sm-offset-3 col-md-10 col-md-offset-2
main-content">
+ <div class="col-md-9">
+ <h1><a
href="https://issues.apache.org/jira/browse/HDDS-10239";>[HDDS-10239]</a>
Container Reconciliation (draft) </h1>
+ <div><i>Authors: </i><div
class="pull-right">2024-01-29</div></div>
+ <p> </p>
+
+ <div class="panel panel-success">
+ <div class="panel-heading">Summary</div>
+ <div class="panel-body">
+ Allow Datanodes to reconcile mismatched container
contents regardless of their state.
+ </div>
+ </div>
+
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+<h1 id="container-reconciliation">Container Reconciliation</h1>
+<p>This document outlines the proposed recovery protocol for containers where
one or more replicas are not cleanly closed or have potential data
inconsistencies. It aims to provide an overview of the planned changes and
their implications, focusing on the overall flow and key design decisions.</p>
+<h2 id="nomenclature">Nomenclature</h2>
+<ol>
+<li>Container: A container is a logical unit of storage management in Ozone.
It is a collection of blocks that are used to store data.</li>
+<li>Container Replica/Instance: A container replica is a copy of a container
that is stored on a Datanode or a shard of an Erasure Coded Container.</li>
+<li>Block: A block is a collection of chunks that are used to store data. An
Ozone object consists of one or more blocks.</li>
+<li>Chunk: A chunk is a collection of bytes that are used to store data. A
chunk is the smallest unit of read and write in Ozone.</li>
+</ol>
+<h2 id="background">Background</h2>
+<p>This proposal is motivated by the need to reconcile mismatched container
replica states and contents among container replicas.
+This covers</p>
+<ol>
+<li>Containers replicas that are not cleanly closed.</li>
+<li>Containers replicas that have potential data inconsistencies due to bugs
or broad failure handling on the write path.</li>
+<li>Silent data corruption that may occur in the system.</li>
+<li>The need to verify the equality and integrity of all closed containers
replicas.</li>
+<li>Deleted blocks within a container that still exists in some container
replicas.</li>
+<li>The need to simplify the replication manager for how to handle cases where
only quasi-closed and unhealthy container replicas are available.</li>
+</ol>
+<p>Ideally, a healthy Ozone cluster would contain only open and closed
container replicas. However, container replicas commonly end up with a mix of
states including quasi-closed and unhealthy that the current system is not able
to resolve to cleanly closed replicas. The cause of these states is often bugs
or broad failure handling on the write path. While we should fix these causes,
they raise the problem that Ozone is not able to reconcile these mismatched
container replica states on i [...]
+<p>Even when all container replicas are closed, the system assumes that these
closed container replicas are equal with no way to verify this. During writes a
client provides a checksum for the chunk that is written.
+The scanner validates periodically that the checksums of the chunks on disk
match the checksums provided by the client. It is possible that the checksum of
a chunk on disk does not match the client provided checksum recorded at the
time of write. Additionally, during container replica copying, the consistency
of the data is not validated, opening the possibility of silent data corruption
propagating through the system.</p>
+<p>This document proposes a container reconciliation protocol to solve these
problems. After implementing the proposal:</p>
+<ol>
+<li>It should be possible for a cluster to progress to a state where all not
open containers are closed and meeting the desired replication factor.</li>
+<li>We can verify the equality and integrity of all closed containers.</li>
+</ol>
+<p>Note: This document does not cover the case where the checksums recorded at
the time of write match the chunks locally within a Datanode but differ across
replicas. We assume that the replication code path is correct and that the
checksums are correct. If this is not the case, the system is already in a
failed state and the reconciliation protocol will not be able to recover it.
Chunks once written are not updated, thus this scenario is not expected to
occur.</p>
+<h2 id="guiding-principles">Guiding Principles</h2>
+<ol>
+<li>
+<p><strong>User Focus</strong>: Users prioritize data durability and
availability above all else.</p>
+<ul>
+<li>From the user perspective, containers labelled quasi-closed and unhealthy
represent compromised durability and availability, regardless of the
container’s actual contents.</li>
+</ul>
+</li>
+<li>
+<p><strong>Focus on Recovery Paths</strong>: Focusing on the path to a failed
state is secondary to focusing on the path out of failed states.</p>
+<ul>
+<li>For example, we should not focus on whether it is possible for two
replicated closed containers to have differing content, only on whether the
system could detect and recover from this case if it were to happen.</li>
+</ul>
+</li>
+<li>
+<p><strong>System Safety</strong>: If a decision made by software will make
data more durable a single trigger is sufficient. If a decision can potentially
reduce durability of data or execute an unsafe operation (unlink, trim, delete)
then the confidence level has to be high, the clarity of the decision precise
and clear and preferably the decision is made within services that have a wider
view of the cluster (SCM/Recon).</p>
+</li>
+<li>
+<p><strong>Datanode Simplicity</strong>: Datanodes should only be responsible
for safe decisions and eager to make safe choices, avoiding unsafe autonomy.</p>
+</li>
+</ol>
+<h2 id="assumptions">Assumptions</h2>
+<ol>
+<li>A closed container will not accept new blocks from clients.</li>
+<li>Empty containers are excluded in this proposal. Empty containers create
unnecessary noise and overhead in the system but are not relevant to the
durability of existing data.</li>
+<li>If checksums of chunks match locally they should match across
replicas.</li>
+<li>The longest block is always the correct block to preserve at the Datanode
level based on the limited information a single Datanode has. Whether the data
within a block is ever accessed by the client depends on the consistency
between Datanode and Ozone Manager which is not transactional and can vary. The
safe decision is to preserve the longest block and let an external entity that
process cluster wide usage of blocks decide if the block can be trimmed or
deleted. The longest block m [...]
+</ol>
+<h2 id="solution-proposal">Solution Proposal</h2>
+<p>The proposed solution involves defining a container level checksum that can
be used to quickly tell if two containers replicas match or not based on their
data. This container checksum can be defined as a three level Merkle tree:</p>
+<ol>
+<li>Level 1 (leaves): The existing chunk level checksums (written by the
client and verified by the existing datanode container scanner).</li>
+<li>Level 2: A block level checksum created by hashing all the chunk checksums
within the block.</li>
+<li>Level 3 (root): A container level checksum created by hashing all the
block checksums within the container.
+<ol>
+<li>This top level container hash is what is reported to SCM to detect
diverged replicas. SCM does not need block or chunk level hashes.</li>
+</ol>
+</li>
+</ol>
+<p>When SCM sees that replicas of a non-open container have diverged container
checksums, it can trigger a reconciliation process on all datanode replicas.
SCM does not need to know which container hash is correct (if any of them are
correct), only that all containers match. Datanodes will use their merkle tree
and those of the other replicas to identify issues with their container. Next,
datanodes can read the missing data from existing replicas and use it to repair
their container repl [...]
+<p>Since the container hash is generated leveraging the checksum recorded at
the time of writing, the container hash represents consistency of the data from
a client perspective.</p>
+<h3 id="phase-i-outlined-in-this-document">Phase I (outlined in this
document)</h3>
+<ol>
+<li>
+<p>Add container level checksums that datanodes can compute and store.</p>
+</li>
+<li>
+<p>Add a mechanism for datanodes to reconcile their replica of a container
with another datanode’s replica so that both replicas can be verified to
be equal at the end of the process.</p>
+</li>
+<li>
+<p>Add a mechanism for SCM to trigger this reconciliation as part of the
existing heartbeat command protocol SCM uses to communicate with datanodes.</p>
+</li>
+<li>
+<p>An <code>ozone admin container reconcile <container-id></code> CLI
that can be used to manually resolve diverged container states among non-open
container replicas.</p>
+<ul>
+<li>When SCM gets this command, it would trigger one reconcile command for
each replica. The CLI would be asynchronous so progress could be checked using
container level checksum info added to <code>ozone admin container info</code>
output.</li>
+</ul>
+</li>
+<li>
+<p>Delete blocks that a Container Replica has not yet deleted.</p>
+</li>
+</ol>
+<h3 id="phase-ii-out-of-scope-for-this-document">Phase II (out of scope for
this document)</h3>
+<ul>
+<li>
+<p>Automate container reconciliation requests as part of SCM’s
replication manager.</p>
+</li>
+<li>
+<p>Simplify SCM replication manager decommission and recovery logic based on
mismatch of container checksums, instead of the combination of all possible
container states.</p>
+</li>
+</ul>
+<h3 id="phase-iii-out-of-scope-for-this-document">Phase III (out of scope for
this document)</h3>
+<ul>
+<li>Extend container level checksum verification to erasure coded containers.
+<ul>
+<li>EC container replicas do not have the same data and are not expected to
have matching container checksums.</li>
+<li>EC containers already use offline recovery as a reconciliation
mechanism.</li>
+</ul>
+</li>
+</ul>
+<h2 id="solution-implementation">Solution Implementation</h2>
+<h3 id="container-hash-tree--merkle-tree">Container Hash Tree / Merkle
Tree</h3>
+<p>The only extra information we will need to store is the container merkle
tree on each datanode container replica. The current proposal is store this
separately as a proto file on disk so that it can be copied over the network
exactly as stored. The structure would look something like this (not finalized,
for illustrative purposes only):</p>
+<div class="highlight"><pre tabindex="0"
style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code
class="language-diff" data-lang="diff"><span style="display:flex;"><span>diff
--git
a/hadoop-hdds/interface-client/src/main/proto/DatanodeClientProtocol.proto
b/hadoop-hdds/interface-client/src/main/proto/DatanodeClientProtocol.proto
+</span></span><span style="display:flex;"><span>index 718e2a108c7..d8d508af356
100644
+</span></span><span style="display:flex;"><span><span
style="color:#f92672">---
a/hadoop-hdds/interface-client/src/main/proto/DatanodeClientProtocol.proto
+</span></span></span><span style="display:flex;"><span><span
style="color:#f92672"></span><span style="color:#a6e22e">+++
b/hadoop-hdds/interface-client/src/main/proto/DatanodeClientProtocol.proto
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e"></span><span style="color:#75715e">@@ -382,6 +382,7 @@
message ChunkInfo {
+</span></span></span><span style="display:flex;"><span><span
style="color:#75715e"></span> repeated KeyValue metadata = 4;
+</span></span><span style="display:flex;"><span> required ChecksumData
checksumData =5;
+</span></span><span style="display:flex;"><span> optional bytes
stripeChecksum = 6;
+</span></span><span style="display:flex;"><span><span style="color:#a6e22e">+
optional bool healthy = 7; // If all the chunks on disk match the expected
checksums provided by the client during write
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e"></span> }
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span> message ChunkInfoList {
+</span></span><span style="display:flex;"><span><span style="color:#75715e">@@
-525,3 +526,38 @@ service IntraDatanodeProtocolService {
+</span></span></span><span style="display:flex;"><span><span
style="color:#75715e"></span> rpc download (CopyContainerRequestProto)
returns (stream CopyContainerResponseProto);
+</span></span><span style="display:flex;"><span> rpc upload (stream
SendContainerRequest) returns (SendContainerResponse);
+</span></span><span style="display:flex;"><span> }
+</span></span><span style="display:flex;"><span><span style="color:#a6e22e">+
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+/*
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+BlockMerkle tree stores the checksums of the chunks in a
block.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+The Block checksum is derived from the checksums of the
chunks in case of replicated blocks and derived from the
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+metadata of the chunks in case of erasure coding.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+Two Blocks across container instances on two nodes have
the same checksum if they have the same set of chunks.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+A Block upon deletion will be marked as deleted but will
preserve the rest of the metadata.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+*/
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+message BlockMerkleTree {
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+ optional BlockData blockData = 1; // The chunks in
this should be sorted by the order of chunks written.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+ optional ChecksumData checksumData = 2; // Checksum of
the checksums of the chunks.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+ optional bool deleted = 3; // If the block is deleted.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+ optional int64 length = 4; // Length of the block.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+ optional int64 chunkCount = 5; // Number of chunks in
the block.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+}
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+/*
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+ContainerMerkleTree stores the checksums of the blocks
in a container.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+The Container checksum is derived from the checksums of
the blocks.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+Two containers across container instances on two nodes
have the same checksum if they have the same set of blocks.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+If a block is deleted within the container, the checksum
of the container will remain unchanged.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+ */
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+message ContainerMerkleTree {
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+ enum FailureCause {
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+ NO_HEALTHY_CHUNK_FOUND_WITH_PEERS = 1; // No healthy
chunk found with peers.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+ NO_PEER_FOUND = 2; // No peer found.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+ }
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+ optional int64 containerID = 1; // The container ID.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+ repeated BlockMerkleTree blockMerkleTrees = 2; // The
blocks in this should be sorted by the order of blocks written.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+ optional ChecksumData checksumData = 3; // Checksum of
the checksums of the blocks.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+ optional int64 length = 5; // Length of the container.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+ optional int64 blockCount = 6; // Number of blocks in
the container.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+ optional FailureCause failureCause = 7; // The cause
of the failure.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+ optional int64 reconciliationCount = 8; // The
reconciliation count.
+</span></span></span><span style="display:flex;"><span><span
style="color:#a6e22e">+}
+</span></span></span></code></pre></div><p>This is written as a file to avoid
bloating the RocksDB instance which is in the IO path.</p>
+<h2 id="apis">APIs</h2>
+<p>The following APIs would be added to datanodes to support container
reconciliation. The actions performed when calling them is defined in <a
href="../design/container-reconciliation.html#events">Events</a>.</p>
+<ul>
+<li><code>reconcileContainer(containerID, List<Replica>)</code>
+<ul>
+<li>Instructs a datanode to reconcile its copy of the specified container with
the provided list of other container replicas.</li>
+<li>Datanodes will call <code>getContainerHashes</code> for each container
replica to identify repairs needed, and use existing chunk read/write APIs to
do repairs necessary.</li>
+<li>This would likely be a new command as part of the SCM heartbeat protocol,
not actually a new API.</li>
+</ul>
+</li>
+<li><code>getContainerHashes(containerID)</code>
+<ul>
+<li>A datanode API that returns the merkle tree for a given container. The
proto structure would be similar to that outlined in <a
href="../design/container-reconciliation.html#Container-Hash-Tree-/-Merkle-Tree">Merkle
Tree</a>.</li>
+</ul>
+</li>
+</ul>
+<h2 id="reconciliation-process">Reconciliation Process</h2>
+<p>SCM: Storage Container Manager</p>
+<p>DN: Datanode</p>
+<h3 id="scm-sets-up-the-reconciliation-process-as-follows">SCM sets up the
reconciliation process as follows:</h3>
+<ol>
+<li>SCM triggers a reconciliation on <code>DN 1</code> for container 12 with
replicas on <code>DN 2</code> and <code>DN 3</code>.
+<ol>
+<li><code>SCM -> reconcileContainer(Container #12, DN2, DN3) -> DN
1</code></li>
+</ol>
+</li>
+</ol>
+<h3 id="datanodes-set-up-the-reconciliation-process-as-follows">Datanodes set
up the reconciliation process as follows:</h3>
+<ol>
+<li><code>DN 1</code> schedules the Merkle Tree to be calculated if it is not
already present and report it to SCM.
+<ol>
+<li>SCM can then compare the container replica hashes and schedule
reconciliation if they are different.</li>
+</ol>
+</li>
+<li>If <code>DN 1</code> already has the Merkle Tree locally, it will compare
it with the Merkle Trees of the other container replicas and schedule
reconciliation if they are different. Example:
+<ol>
+<li><code>DN 1 -> getContainerHashes(Container #12) -> DN 2</code> //
Datanode 1 gets the merkle tree of container 12 from Datanode 2.</li>
+<li><code>DN 1 -> getContainerHashes(Container #12) -> DN 3</code> //
Datanode 1 gets the merkle tree of container 12 from Datanode 3.</li>
+<li>… // Continue for all replicas.</li>
+</ol>
+</li>
+</ol>
+<h3
id="reconcile-loop-once-the-merkle-trees-are-obtained-from-allmost-replicas">Reconcile
loop once the merkle trees are obtained from all/most replicas:</h3>
+<ol>
+<li><code>DN 1</code> checks if any blocks are missing. For each missing Block:
+<ol>
+<li>Make a <strong>union</strong> of all chunks and which DNs has the chunk.
It is possible that certain chunks are missing in other container replicas. The
<strong>union</strong> of chunks represents all the chunks that are present in
any of the replicas.</li>
+<li>Read the remote chunks and store them locally and create the local block.
Example:
+<ol>
+<li><code>DN 1 -> readChunk(Container #12, Block #3, Chunk #1) -> DN
2</code> // Read the chunk from DN 2</li>
+<li><code>DN 1 -> readChunk(Container #12, Block #3, Chunk #2) -> DN
3</code> // Read the chunk from DN 3</li>
+<li>… // Continue for all chunks that are in the union of chunks.</li>
+<li>Datanode will validate the checksum of the chunk read from the peer
Datanode before using it to reconcile. This is already performed by the
existing client code. Thus, no silent data corruption will be introduced.</li>
+</ol>
+</li>
+</ol>
+</li>
+<li><code>DN 1</code> checks if any blocks are corrupted. For each corrupted
Block:
+<ol>
+<li>Make a union of all chunks and which DN has the chunks. It is possible
that certain chunks are corrupted in one of the container replicas. The
<strong>union</strong> of chunks represents all the chunks that are present in
any of the replicas.</li>
+<li>Read the remote chunks and store them locally. Example:
+<ol>
+<li><code>DN 1 -> readChunk(Container #12, Block #20, Chunk #13) -> DN
2</code></li>
+<li><code>DN 1 -> readChunk(Container #12, Block #20, Chunk #21) -> DN
3</code></li>
+<li>… // Continue for all chunks that are in the union of chunks.</li>
+<li>Datanode will validate the checksum of the chunk read from the peer
Datanode before using it to reconcile. This is already performed by the
existing client code. Thus, no silent data corruption will be introduced.</li>
+</ol>
+</li>
+</ol>
+</li>
+<li><code>DN 1</code> deletes any blocks that are marked as deleted.
+<ol>
+<li>The block continues to be in the tree with the updated checksum to avoid
redundant Merkle tree updates.</li>
+</ol>
+</li>
+<li><code>DN 1</code> recomputes Merkle tree and sends it to SCM via ICR
(Incremental Container Report).</li>
+</ol>
+<p>Note: This document does not cover the case where the checksums recorded at
the time of write match the chunks locally within a Datanode but differ across
replicas. We assume that the replication code path is correct and that the
checksums are correct. If this is not the case, the system is already in a
failed state and the reconciliation protocol will not be able to recover it.
Chunks once written are not updated, thus this scenario is not expected to
occur.</p>
+<h4 id="getcontainerhashes"><code>getContainerHashes</code></h4>
+<p>When a datanode receives a request to get container hashes. The following
steps are performed:</p>
+<ol>
+<li>If the Merkle Tree is present, return it.</li>
+<li>If the Merkle Tree is not present, the call
<code>getContainerHashes</code> will return and error.
+<ol>
+<li>Datanode missing the Merkle Tree will be schedule the scanner to calculate
the Merkle Tree and then report to SCM.</li>
+</ol>
+</li>
+<li>If container not closed, return error.</li>
+<li>If container not found, return error.</li>
+<li>The merkle tree returned by this call will represent the status of the
data on disk (post scanner scrub).</li>
+</ol>
+<h2 id="sample-scenarios">Sample scenarios</h2>
+<ol>
+<li>
+<p><strong>Container is missing blocks</strong></p>
+<ol>
+<li><code>DN 1</code> has 10 blocks, <code>DN 2</code> has 11 blocks, <code>DN
3</code> has 12 blocks.</li>
+<li><code>DN 1</code> will read the missing block from <code>DN 2</code> and
<code>DN 3</code> and store it locally.</li>
+<li><code>DN 1</code> will recompute the merkle tree and send it to SCM.</li>
+</ol>
+</li>
+<li>
+<p><strong>Container has corrupted chunks</strong></p>
+<ol>
+<li><code>DN 1</code> has block 20: chunk 13, <code>DN 2</code> has block 12:
chunk 13</li>
+<li><code>DN 1</code> will read the corrupted block from <code>DN 2</code> and
store it locally.</li>
+<li><code>DN 1</code> will recompute the merkle tree and send it to SCM.</li>
+</ol>
+</li>
+<li>
+<p>Closed container has a chunk that is corrupted</p>
+<ol>
+<li>Scanner will detect the corruption and mark the container as unhealthy and
reported an updated Merkle tree to SCM.
+<ol>
+<li>The Merkle tree will be updated to reflect the current contents of the
disk and the hash will be updated to reflect the current contents of the
disk.</li>
+</ol>
+</li>
+<li>SCM will trigger a reconciliation process on the Datanode that sees the
corruption.</li>
+<li>The Datanode will read the corrupted chunk from another Datanode and store
it locally as part of reconciliation covered in #2 above.</li>
+<li>The Datanode will recompute the Merkle tree and send it to SCM.</li>
+<li>In the meantime if a client or a peer Datanode reads the corrupted chunk
the checksum match computed by the client code will fail and the client will
get a checksum mismatch error.</li>
+</ol>
+</li>
+</ol>
+<h2 id="events">Events</h2>
+<p>A Datanode is still accepting writes and reads while the reconciliation
process is ongoing. The reconciliation process is a background process that
does not affect the normal operation of the Datanode.
+This section defines how container reconciliation will function as events
occur in the system.</p>
+<h3 id="reconciliation-events">Reconciliation Events</h3>
+<p>These events occur as part of container reconciliation in the happy path
case.</p>
+<h4 id="on-data-write">On Data Write</h4>
+<ul>
+<li>No change to existing datanode operations. Existing chunk checksums passed
and validated from the client will be persisted to RocksDB.</li>
+<li>Reconciliation is only performed on containers that are not open.</li>
+<li>Merkle tree is updated on container close.</li>
+</ul>
+<h4 id="on-container-close">On Container Close</h4>
+<ul>
+<li>Container checksum is calculated using the checksums that were recorded in
RocksDB at the time of write. This calculation must finish before the close is
acked back to SCM.
+<ul>
+<li><strong>Invariant</strong>: All closed containers have a checksum even in
the case of restarts and failures because SCM will retry the close command if
it does not receive an ack.
+<ul>
+<li>Null handling should still be in place for containers created before this
feature, and to guard against bugs.</li>
+</ul>
+</li>
+</ul>
+</li>
+</ul>
+<h4 id="on-container-scan">On Container Scan</h4>
+<ol>
+<li>Scanner reads chunks and compares it to the checksums in RocksDB.</li>
+<li>Scanner updates Merkle tree to match the current contents of the disk. It
does not update RocksDB.
+<ul>
+<li>Update Merkle tree checksums in a rolling fashion per chunk and block as
scanning progresses.</li>
+<li><strong>Note</strong>: The checksums recorded in RocksDB are immutable and
cannot be updated at any point in time. Only new chunks and their checksums can
be added as part of reconciliation.</li>
+</ul>
+</li>
+<li>If scanner detects local corruption (RocksDB hash does not match hash
calculated from disk), marks container unhealthy and sends an incremental
container report to SCM with its calculated container hash.
+<ul>
+<li>The Merkle Tree is still updated to reflect the current contents of the
disk in this case.</li>
+</ul>
+</li>
+</ol>
+<h4 id="on-container-import">On Container import</h4>
+<p>In multiple scenarios it is safer to immediately create a container replica
and not wait for reconciliation.
+Example: The cluster has only one single closed container replica for the
container.</p>
+<ul>
+<li>Schedule on-demand scan after the container import completes.
+<ul>
+<li>There may be cases where we need to import a container without verifying
the contents.</li>
+<li>Instead, schedule the imported container for on-demand scanning to be sure
SCM knows of corruption if it does not already.</li>
+<li><strong>Optimization</strong>: Clear last scanned timestamp on container
import to make the background scanner get to it faster if datanode restarts
before the on-demand scan completes.</li>
+</ul>
+</li>
+</ul>
+<h3 id="concurrent-events">Concurrent Events</h3>
+<p>These may happen to the container during reconciliation and must remain
consistent during reconciliation. Note that we cannot depend on SCM
coordination to shelter datanodes from other commands after SCM has instructed
them to reconcile because datanode command queues may get backed up at various
points and process commands in a different order, or long after they were
initially issued.</p>
+<h4 id="on-client-read-of-container-under-reconciliation">On Client read of
container under reconciliation</h4>
+<ul>
+<li>No action required.</li>
+</ul>
+<p>This read could be an end client or another datanode asking for data to
repair its replica. The result will only be inconsistent if the chunk/block is
being repaired. In this case the data is already corrupt and the client will
get a read checksum failure either way. Even if all datanodes are reconciling
the same container at the same time, this should not cause an availability
problem because there has to be one correct replica that others are repairing
from that the client can also [...]
+<h4 id="replicate-command-for-a-container-being-reconciled">Replicate command
for a container being reconciled</h4>
+<ul>
+<li>Allow replication to proceed with best effort, possibly with a warning
log.</li>
+</ul>
+<p>There may be cases where a container is critically under-replicated and we
need a good copy of the data as soon as possible. Meanwhile, reconciliation
could remain blocked for an unrelated reason. We can provide a way to
pause/unpause reconciliation between chunk writes to do reasonably consistent
container exports.</p>
+<h4 id="on-block-delete">On block delete</h4>
+<ul>
+<li>
+<p>When SCM sends block delete commands to datanodes, update the merkle tree
when the datanode block deleting service runs and processes that block.</p>
+<ul>
+<li>
+<p>Merkle tree update should be done before RocksDB update to make sure it is
persisted in case of a failure during delete.</p>
+</li>
+<li>
+<p>Merkle tree update should not be done when datanodes first receives the
block delete command from SCM, because this command only adds the delete block
proto to the container. It does not iterate the blocks to be deleted so we
should not add that additional step here.</p>
+</li>
+</ul>
+</li>
+</ul>
+<h4 id="on-container-delete-for-container-being-reconciled">On Container
delete for container being reconciled</h4>
+<ul>
+<li>Provide a way to interrupt reconciliation and allow container delete to
proceed.</li>
+</ul>
+<h2 id="backwards-compatibility">Backwards Compatibility</h2>
+<p>Since the scanner can generate the container merkle trees in the
background, existing containers created before this feature will still be
eligible for reconciliation. These old containers may not have all of their
block deletes present in the merkle tree, however, which could cause some false
positives about missing blocks on upgrade if one node had already deleted
blocks from a container before the upgrade, and another datanode has not yet
processed the delete of those blocks.</p>
+<p>This can be mitigated by:</p>
+<ol>
+<li>Having datanodes delete blocks from their container replica on
reconciliation that another replica has marked as deleted.</li>
+<li>Having SCM periodically ask datanodes to reconcile otherwise matching
containers when there are no other mismatched containers in the cluster.</li>
+</ol>
+<p>The sequence of events would look like this. Assume software v1 does not
have container reconciliation, but v2 does.</p>
+<ol>
+<li>Datanode 1 in v1 deletes block 1 in container 1. This does not leave a
tombstone entry because v1 does not have container reconciliation.</li>
+<li>Cluster is upgraded to v2.</li>
+<li>Reconciliation is triggered between datanodes 1 and 2 for container 1.
Datanode 2 has not yet deleted block 1.</li>
+<li>Datanode 1 will add block 1 to its container when reconciling with
datanode 2.
+<ul>
+<li>If we stop here, then block 1 will remain orphaned on datanode 1.</li>
+</ul>
+</li>
+<li>Datanode 2 deletes block 1 and adds a tombstone entry for it since the
cluster is running software v2 which has container reconciliation.</li>
+<li>Since container checksums disregard deleted blocks, container 1 will be
seen as matching from SCM’s point of view. However, periodic
reconciliation requests for closed containers will still eventually ask these
two replicas to reconcile.</li>
+<li>Datanode 1 learns that block 1 was deleted from datanode 2, so it moves
the block metadata to the deleted table in RocksDB
+<ul>
+<li>A delete transaction entry with an ID key would need to be created to do
this. Currently these are only received from SCM and not created by the
datanode.</li>
+</ul>
+</li>
+<li>Datanode block deleting service deletes this block as usual.</li>
+</ol>
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+
+
+<footer class="footer">
+ <div class="container">
+ <span class="small text-muted">
+ Version: 2.1.0-SNAPSHOT, Last Modified: May 23, 2024 <a
class="hide-child link primary-color"
href="https://github.com/apache/ozone/commit/a4bb94ee6390741f3b454f62dfd5a80a0d0b69d8";>a4bb94ee63</a>
+ </span>
+ </div>
+</footer>
+
+
+
+<script src="../js/jquery-3.5.1.min.js"></script>
+<script src="../js/ozonedoc.js"></script>
+<script src="../js/bootstrap.min.js"></script>
+
+
+ </body>
+</html>
diff --git a/docs/edge/design/index.xml b/docs/edge/design/index.xml
index 2c3827f4..14a9e94c 100644
--- a/docs/edge/design/index.xml
+++ b/docs/edge/design/index.xml
@@ -29,6 +29,13 @@
<guid>/design/overwrite-key-only-if-unchanged.html</guid>
<description>A minimal design illustrating how to replace a key in Ozone
only if it has not changes since it was read.</description>
</item>
+ <item>
+ <title>Container Reconciliation</title>
+ <link>/design/container-reconciliation.html</link>
+ <pubDate>Mon, 29 Jan 2024 00:00:00 +0000</pubDate>
+ <guid>/design/container-reconciliation.html</guid>
+ <description>Allow Datanodes to reconcile mismatched container contents
regardless of their state.</description>
+ </item>
<item>
<title>Ozone Token Signatures using Symmetric Keys</title>
<link>/design/symmetric-token-signatures.html</link>
diff --git a/docs/edge/en/sitemap.xml b/docs/edge/en/sitemap.xml
index 618ac7c9..cddefd42 100644
--- a/docs/edge/en/sitemap.xml
+++ b/docs/edge/en/sitemap.xml
@@ -437,7 +437,7 @@
/>
</url><url>
<loc>/concept/datanodes.html</loc>
- <lastmod>2024-09-23T17:47:30+08:00</lastmod>
+ <lastmod>2024-12-18T11:22:00-08:00</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -652,6 +652,9 @@
</url><url>
<loc>/design/overwrite-key-only-if-unchanged.html</loc>
<lastmod>2024-04-29T10:37:20+01:00</lastmod>
+ </url><url>
+ <loc>/design/container-reconciliation.html</loc>
+ <lastmod>2024-05-23T09:40:47-07:00</lastmod>
</url><url>
<loc>/design/symmetric-token-signatures.html</loc>
<lastmod>2023-07-31T14:07:54-07:00</lastmod>
diff --git a/docs/edge/index.xml b/docs/edge/index.xml
index 72ecc8b0..f0da0f63 100644
--- a/docs/edge/index.xml
+++ b/docs/edge/index.xml
@@ -427,6 +427,13 @@
<guid>/design/overwrite-key-only-if-unchanged.html</guid>
<description>A minimal design illustrating how to replace a key in Ozone
only if it has not changes since it was read.</description>
</item>
+ <item>
+ <title>Container Reconciliation</title>
+ <link>/design/container-reconciliation.html</link>
+ <pubDate>Mon, 29 Jan 2024 00:00:00 +0000</pubDate>
+ <guid>/design/container-reconciliation.html</guid>
+ <description>Allow Datanodes to reconcile mismatched container contents
regardless of their state.</description>
+ </item>
<item>
<title>Ozone Token Signatures using Symmetric Keys</title>
<link>/design/symmetric-token-signatures.html</link>
diff --git a/docs/edge/interface/ofs.html b/docs/edge/interface/ofs.html
index 7ab94b8f..90956e57 100644
--- a/docs/edge/interface/ofs.html
+++ b/docs/edge/interface/ofs.html
@@ -574,7 +574,7 @@ For example:</p>
<p>Or use the put command to write a file to the bucket.</p>
<div class="highlight"><pre tabindex="0"
style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code
class="language-bash" data-lang="bash"><span style="display:flex;"><span>hdfs
dfs -put /etc/hosts /volume1/bucket1/test</span></span></code></pre></div>
<p>For more usage, see: <a
href="https://issues.apache.org/jira/secure/attachment/12987636/Design%20ofs%20v1.pdf";>https://issues.apache.org/jira/secure/attachment/12987636/Design%20ofs%20v1.pdf</a></p>
-<h2 id="differences-from-o3fshahahugoshortcode100s5hbhb">Differences from <a
href="../interface/o3fs.html">o3fs</a></h2>
+<h2 id="differences-from-o3fshahahugoshortcode101s5hbhb">Differences from <a
href="../interface/o3fs.html">o3fs</a></h2>
<h3 id="creating-files">Creating files</h3>
<p>OFS doesn’t allow creating keys(files) directly under root or volumes.
Users will receive an error message when they try to do that:</p>
diff --git a/docs/edge/zh/concept/datanodes.html
b/docs/edge/zh/concept/datanodes.html
index cef07bed..3747dd32 100644
--- a/docs/edge/zh/concept/datanodes.html
+++ b/docs/edge/zh/concept/datanodes.html
@@ -514,7 +514,7 @@ s=d.getElementsByTagName('script')[0];
<footer class="footer">
<div class="container">
<span class="small text-muted">
- Version: 2.1.0-SNAPSHOT, Last Modified: September 23, 2024 <a
class="hide-child link primary-color"
href="https://github.com/apache/ozone/commit/0fcb64564d88defc8d2c0ab585bff5717c26632b";>0fcb64564d</a>
+ Version: 2.1.0-SNAPSHOT, Last Modified: December 18, 2024 <a
class="hide-child link primary-color"
href="https://github.com/apache/ozone/commit/7bcc943077097125e16d705ba1366139ba4c5871";>7bcc943077</a>
</span>
</div>
</footer>
diff --git a/docs/edge/zh/interface/ofs.html b/docs/edge/zh/interface/ofs.html
index beb5f0d2..530b6b0c 100644
--- a/docs/edge/zh/interface/ofs.html
+++ b/docs/edge/zh/interface/ofs.html
@@ -478,7 +478,7 @@ ofs://omservice/tmp/key1
<p>或者使用 put 命令向桶中写入一个文件</p>
<div class="highlight"><pre tabindex="0"
style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code
class="language-bash" data-lang="bash"><span style="display:flex;"><span>hdfs
dfs -put /etc/hosts /volume1/bucket1/test</span></span></code></pre></div>
<p>有关更多用法，请参见： <a
href="https://issues.apache.org/jira/secure/attachment/12987636/Design%20ofs%20v1.pdf";>https://issues.apache.org/jira/secure/attachment/12987636/Design%20ofs%20v1.pdf</a></p>
-<h2 id="与-o3fshahahugoshortcode99s5hbhb-的区别">与 <a
href="../../zh/interface/o3fs.html">o3fs</a> 的区别</h2>
+<h2 id="与-o3fshahahugoshortcode100s5hbhb-的区别">与 <a
href="../../zh/interface/o3fs.html">o3fs</a> 的区别</h2>
<h3 id="创建文件">创建文件</h3>
<p>OFS 不允许直接在根目录或卷下创建键（文件）。
当用户尝试这样做时，他们将收到一个错误消息：</p>
diff --git a/docs/edge/zh/sitemap.xml b/docs/edge/zh/sitemap.xml
index 6a2c97c8..869577c8 100644
--- a/docs/edge/zh/sitemap.xml
+++ b/docs/edge/zh/sitemap.xml
@@ -315,7 +315,7 @@
/>
</url><url>
<loc>/zh/concept/datanodes.html</loc>
- <lastmod>2024-09-23T17:47:30+08:00</lastmod>
+ <lastmod>2024-12-18T11:22:00-08:00</lastmod>
<xhtml:link
rel="alternate"
hreflang="en"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
