[
https://issues.apache.org/jira/browse/HDFS-17897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhenyuLi updated HDFS-17897:
----------------------------
Description:
HDFS-12931 added handling for InvalidEncryptionKeyException in
ReplicatedFileChecksumComputer.checksumBlock() to support getFileChecksum()
with encrypted data transfer. However, the parallel striped file path
StripedFileNonStripedChecksumComputer.checksumBlockGroup() was not updated.
Both paths call DFSClient.connectToDN() which performs a SASL handshake using a
cached DataEncryptionKey (DEK). When the DEK references a BlockKey that has
been removed from the DataNode (e.g., after
NameNode restart or key rotation), the handshake fails with
InvalidEncryptionKeyException.
In the replicated path, this exception is caught, clearDataEncryptionKey() is
called to invalidate the cached DEK, and the block is retried. In the striped
path, the exception falls through to the
generic catch (IOException) block, which only logs a warning. The stale DEK
is never cleared, so every DataNode in the block group fails with the same
error. The operation fails permanently — even
user-level retries will reuse the same stale cached DEK.
Fix: Add catch (InvalidEncryptionKeyException) in checksumBlockGroup(),
mirroring the existing handling in checksumBlock().
was:
HDFS-12931 added handling for InvalidEncryptionKeyException in
ReplicatedFileChecksumComputer.checksumBlock() to support getFileChecksum()
with encrypted data transfer. However, the parallel striped file
path StripedFileNonStripedChecksumComputer.checksumBlockGroup() was not
updated.
Both paths call DFSClient.connectToDN() which performs a SASL handshake using
a cached DataEncryptionKey (DEK). When the DEK references a BlockKey that has
been removed from the DataNode (e.g., after
NameNode restart or key rotation), the handshake fails with
InvalidEncryptionKeyException.
In the replicated path, this exception is caught, clearDataEncryptionKey() is
called to invalidate the cached DEK, and the block is retried. In the striped
path, the exception falls through to the
generic catch (IOException) block, which only logs a warning. The stale DEK
is never cleared, so every DataNode in the block group fails with the same
error. The operation fails permanently — even
user-level retries will reuse the same stale cached DEK.
Fix: Add catch (InvalidEncryptionKeyException) in checksumBlockGroup(),
mirroring the existing handling in checksumBlock().
> Handle InvalidEncryptionKeyException during striped file checksum
> ------------------------------------------------------------------
>
> Key: HDFS-17897
> URL: https://issues.apache.org/jira/browse/HDFS-17897
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
> Affects Versions: 3.1.2
> Reporter: ZhenyuLi
> Priority: Major
>
> HDFS-12931 added handling for InvalidEncryptionKeyException in
> ReplicatedFileChecksumComputer.checksumBlock() to support getFileChecksum()
> with encrypted data transfer. However, the parallel striped file path
> StripedFileNonStripedChecksumComputer.checksumBlockGroup() was not updated.
> Both paths call DFSClient.connectToDN() which performs a SASL handshake using
> a cached DataEncryptionKey (DEK). When the DEK references a BlockKey that has
> been removed from the DataNode (e.g., after
> NameNode restart or key rotation), the handshake fails with
> InvalidEncryptionKeyException.
> In the replicated path, this exception is caught, clearDataEncryptionKey()
> is called to invalidate the cached DEK, and the block is retried. In the
> striped path, the exception falls through to the
> generic catch (IOException) block, which only logs a warning. The stale DEK
> is never cleared, so every DataNode in the block group fails with the same
> error. The operation fails permanently — even
> user-level retries will reuse the same stale cached DEK.
> Fix: Add catch (InvalidEncryptionKeyException) in checksumBlockGroup(),
> mirroring the existing handling in checksumBlock().
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
