[
https://issues.apache.org/jira/browse/HDFS-17897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhenyuLi updated HDFS-17897:
----------------------------
Description:
*Description:*
HDFS-12931 added handling for {{InvalidEncryptionKeyException}} in
{{ReplicatedFileChecksumComputer.checksumBlock()}} to support
{{getFileChecksum()}} with encrypted data transfer. However, the parallel
striped file path
{{StripedFileNonStripedChecksumComputer.checksumBlockGroup()}} was not updated.
Both paths call {{{}DFSClient.connectToDN(){}}}, which performs a SASL
handshake using a cached {{DataEncryptionKey}} (DEK). After NameNode key
rotation, the client may still hold a cached DEK derived from an old
{{BlockKey}} that has already expired and been removed from DataNodes (as
described in HDFS-12931), causing the handshake to fail with
{{{}InvalidEncryptionKeyException{}}}.
In the replicated path, this exception is caught by the handling added in
HDFS-12931: {{clearDataEncryptionKey()}} is called to invalidate the cached
DEK, and the block operation is retried with a freshly negotiated key. In the
striped path, the exception falls through to the generic {{catch
(IOException)}} block, which only logs a warning and moves on to the next
DataNode in the block group. Because the stale DEK is never cleared, every
subsequent {{connectToDN()}} call within the same block group fails with the
same error. The operation fails permanently — even client-side retries reuse
the same stale cached DEK.
*Proposed Fix:* Add {{catch (InvalidEncryptionKeyException)}} in
{{{}checksumBlockGroup(){}}}, mirroring the existing handling in
{{{}checksumBlock(){}}}: call {{clearDataEncryptionKey()}} to invalidate the
cached DEK, then retry the block group.
*Testing:* Added unit test {{testStripedFileChecksumWithInvalidEncryptionKey}}
to reproduce the failure and verify the fix.
was:
HDFS-12931 added handling for InvalidEncryptionKeyException in
ReplicatedFileChecksumComputer.checksumBlock() to support getFileChecksum()
with encrypted data transfer. However, the parallel striped file path
StripedFileNonStripedChecksumComputer.checksumBlockGroup() was not updated.
Both paths call {{{}DFSClient.connectToDN(){}}}, which performs a SASL
handshake using a cached {{DataEncryptionKey}} (DEK). After NameNode key
rotation, the client may still hold a cached DEK derived from an old
{{BlockKey}} that has already expired and been removed from DataNodes (as
described in HDFS-12931), causing the handshake to fail with
{{{}InvalidEncryptionKeyException{}}}.
In the replicated path, this exception is caught in HDFS-12931's fix,
clearDataEncryptionKey() is called to invalidate the cached DEK, and the block
is retried. In the striped path, the exception falls through to the generic
catch (IOException) block, which only logs a warning. The stale DEK is never
cleared, so every DataNode in the block group fails with the same error. The
operation fails permanently — even client side retries will reuse the same
stale cached DEK.
Proposed Fix: Add catch (InvalidEncryptionKeyException) in
checksumBlockGroup(), mirroring the existing handling in checksumBlock().
I wrote a unit test testStripedFileChecksumWithInvalidEncryptionKey to
reproduce it.
> Handle InvalidEncryptionKeyException during striped file checksum
> ------------------------------------------------------------------
>
> Key: HDFS-17897
> URL: https://issues.apache.org/jira/browse/HDFS-17897
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
> Affects Versions: 3.1.2
> Reporter: ZhenyuLi
> Priority: Major
> Labels: pull-request-available
>
> *Description:*
> HDFS-12931 added handling for {{InvalidEncryptionKeyException}} in
> {{ReplicatedFileChecksumComputer.checksumBlock()}} to support
> {{getFileChecksum()}} with encrypted data transfer. However, the parallel
> striped file path
> {{StripedFileNonStripedChecksumComputer.checksumBlockGroup()}} was not
> updated.
> Both paths call {{{}DFSClient.connectToDN(){}}}, which performs a SASL
> handshake using a cached {{DataEncryptionKey}} (DEK). After NameNode key
> rotation, the client may still hold a cached DEK derived from an old
> {{BlockKey}} that has already expired and been removed from DataNodes (as
> described in HDFS-12931), causing the handshake to fail with
> {{{}InvalidEncryptionKeyException{}}}.
> In the replicated path, this exception is caught by the handling added in
> HDFS-12931: {{clearDataEncryptionKey()}} is called to invalidate the cached
> DEK, and the block operation is retried with a freshly negotiated key. In the
> striped path, the exception falls through to the generic {{catch
> (IOException)}} block, which only logs a warning and moves on to the next
> DataNode in the block group. Because the stale DEK is never cleared, every
> subsequent {{connectToDN()}} call within the same block group fails with the
> same error. The operation fails permanently — even client-side retries reuse
> the same stale cached DEK.
> *Proposed Fix:* Add {{catch (InvalidEncryptionKeyException)}} in
> {{{}checksumBlockGroup(){}}}, mirroring the existing handling in
> {{{}checksumBlock(){}}}: call {{clearDataEncryptionKey()}} to invalidate the
> cached DEK, then retry the block group.
> *Testing:* Added unit test
> {{testStripedFileChecksumWithInvalidEncryptionKey}} to reproduce the failure
> and verify the fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
