[
https://issues.apache.org/jira/browse/HDFS-17960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhenyuLi updated HDFS-17960:
----------------------------
Description:
When edit logs generated with ordered snapshot deletion enabled are replayed
with dfs.namenode.snapshot.deletion.ordered=false, SnapshotManager.numSnapshots
may be decremented even though no snapshot is removed.
HDFS-15590 added a replay-tolerance path in
DirectorySnapshottableFeature.removeSnapshot(). If the requested snapshot does
not exist, ordered deletion is disabled, and the namespace image is still being
loaded, removeSnapshot() returns null:
{code:java}
if (!snapshotManager.isSnapshotDeletionOrdered()
&& !snapshotManager.isImageLoaded()) {
return null;
}
{code}
However, SnapshotManager.deleteSnapshot() ignores the return value and
unconditionally decrements numSnapshots:
{code:java}
srcRoot.removeSnapshot(reclaimContext, snapshotName, now, this);
numSnapshots.getAndDecrement();
{code}
Therefore, a tolerated no-op edit changes numSnapshots without changing the
authoritative snapshotsByNames lists.
h3. Proposed fix
Use the return value of removeSnapshot() and decrement numSnapshots only when
a Snapshot was actually removed:
{code:java}
final Snapshot removed = srcRoot.removeSnapshot(
reclaimContext, snapshotName, now, this);
if (removed != null) {
numSnapshots.decrementAndGet();
}
{code}
The resulting snapshot count inconsistency causes subsequent checkpoint or
saveNamespace operations to fail at Preconditions.checkState(i ==
sm.getNumSnapshots()) in
FSImageFormatPBSnapshot.Saver.serializeSnapshotSection().
was:
When edit logs generated with ordered snapshot deletion enabled are replayed
with dfs.namenode.snapshot.deletion.ordered=false, SnapshotManager.numSnapshots
may be decremented even though no snapshot is removed.
HDFS-15590 added a replay-tolerance path in
DirectorySnapshottableFeature.removeSnapshot(). If the requested snapshot does
not exist, ordered deletion is disabled, and the namespace image is still being
loaded, removeSnapshot() returns null:
{code:java}
if (!snapshotManager.isSnapshotDeletionOrdered()
&& !snapshotManager.isImageLoaded()) {
return null;
}
{code}
However, SnapshotManager.deleteSnapshot() ignores the return value and
unconditionally decrements numSnapshots:
{code:java}
srcRoot.removeSnapshot(reclaimContext, snapshotName, now, this);
numSnapshots.getAndDecrement();
{code}
Therefore, a tolerated no-op edit changes numSnapshots without changing the
authoritative snapshotsByNames lists.
h3. Proposed fix
Use the return value of removeSnapshot() and decrement numSnapshots only when
a Snapshot was actually removed:
{code:java}
final Snapshot removed = srcRoot.removeSnapshot(
reclaimContext, snapshotName, now, this);
if (removed != null) {
numSnapshots.decrementAndGet();
}
{code}
The replay-tolerance behavior added by HDFS-15590 should remain unchanged.
h3. Test plan
Extend TestOrderedSnapshotDeletion.testOrderedDeletionWithRestart() to:
# Assert that the actual snapshot list and SnapshotManager.numSnapshots are
both 1 after restart.
# Enter safe mode and run saveNamespace().
# Verify that the checkpoint succeeds.
A second regression test may extend
TestOrderedSnapshotDeletionGc.testSingleDir() by restarting with ordered
snapshot deletion disabled. The existing workload already creates the client
and GC deletion records required to reproduce the issue.
> Snapshot count becomes inconsistent when replaying ordered snapshot deletion
> edits
> -----------------------------------------------------------------------------------
>
> Key: HDFS-17960
> URL: https://issues.apache.org/jira/browse/HDFS-17960
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
> Affects Versions: 3.5.0
> Reporter: ZhenyuLi
> Priority: Major
>
> When edit logs generated with ordered snapshot deletion enabled are replayed
> with dfs.namenode.snapshot.deletion.ordered=false,
> SnapshotManager.numSnapshots may be decremented even though no snapshot is
> removed.
> HDFS-15590 added a replay-tolerance path in
> DirectorySnapshottableFeature.removeSnapshot(). If the requested snapshot
> does not exist, ordered deletion is disabled, and the namespace image is
> still being loaded, removeSnapshot() returns null:
> {code:java}
> if (!snapshotManager.isSnapshotDeletionOrdered()
> && !snapshotManager.isImageLoaded()) {
> return null;
> }
> {code}
> However, SnapshotManager.deleteSnapshot() ignores the return value and
> unconditionally decrements numSnapshots:
> {code:java}
> srcRoot.removeSnapshot(reclaimContext, snapshotName, now, this);
> numSnapshots.getAndDecrement();
> {code}
> Therefore, a tolerated no-op edit changes numSnapshots without changing the
> authoritative snapshotsByNames lists.
> h3. Proposed fix
> Use the return value of removeSnapshot() and decrement numSnapshots only
> when
> a Snapshot was actually removed:
> {code:java}
> final Snapshot removed = srcRoot.removeSnapshot(
> reclaimContext, snapshotName, now, this);
> if (removed != null) {
> numSnapshots.decrementAndGet();
> }
> {code}
> The resulting snapshot count inconsistency causes subsequent checkpoint or
> saveNamespace operations to fail at Preconditions.checkState(i ==
> sm.getNumSnapshots()) in
> FSImageFormatPBSnapshot.Saver.serializeSnapshotSection().
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
