[
https://issues.apache.org/jira/browse/HDFS-17899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18072860#comment-18072860
]
ASF GitHub Bot commented on HDFS-17899:
---------------------------------------
JHSUYU commented on code in PR #8383:
URL: https://github.com/apache/hadoop/pull/8383#discussion_r3068087822
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java:
##########
@@ -372,29 +374,54 @@ private void dispatch() {
LOG.info("Start moving " + this);
assert !(reportedBlock instanceof DBlockStriped);
- sock.connect(
- NetUtils.createSocketAddr(target.getDatanodeInfo().
- getXferAddr(Dispatcher.this.connectToDnViaHostname)),
- HdfsConstants.READ_TIMEOUT);
-
- // Set read timeout so that it doesn't hang forever against
- // unresponsive nodes. Datanode normally sends IN_PROGRESS response
- // twice within the client read timeout period (every 30 seconds by
- // default). Here, we make it give up after 5 minutes of no response.
- sock.setSoTimeout(HdfsConstants.READ_TIMEOUT * 5);
- sock.setKeepAlive(true);
-
- OutputStream unbufOut = sock.getOutputStream();
- InputStream unbufIn = sock.getInputStream();
ExtendedBlock eb = new ExtendedBlock(nnc.getBlockpoolID(),
reportedBlock.getBlock());
- final KeyManager km = nnc.getKeyManager();
+ final KeyManager km = nnc.getKeyManager();
Token<BlockTokenIdentifier> accessToken = km.getAccessToken(eb,
new StorageType[]{target.storageType}, new String[0]);
- IOStreamPair saslStreams = saslClient.socketSend(sock, unbufOut,
- unbufIn, km, accessToken, target.getDatanodeInfo());
- unbufOut = saslStreams.out;
- unbufIn = saslStreams.in;
+ OutputStream unbufOut;
+ InputStream unbufIn;
+ int encryptionKeyRetryCount = 0;
+ while (true) {
+ try {
+ sock.connect(
+ NetUtils.createSocketAddr(target.getDatanodeInfo().
+ getXferAddr(Dispatcher.this.connectToDnViaHostname)),
+ HdfsConstants.READ_TIMEOUT);
+
+ // Set read timeout so that it doesn't hang forever against
+ // unresponsive nodes. Datanode normally sends IN_PROGRESS
+ // response twice within the client read timeout period (every
+ // 30 seconds by default). Here, we make it give up after 5
+ // minutes of no response.
+ sock.setSoTimeout(HdfsConstants.READ_TIMEOUT * 5);
+ sock.setKeepAlive(true);
+
+ unbufOut = sock.getOutputStream();
+ unbufIn = sock.getInputStream();
+ IOStreamPair saslStreams = saslClient.socketSend(sock, unbufOut,
+ unbufIn, km, accessToken, target.getDatanodeInfo());
+ unbufOut = saslStreams.out;
+ unbufIn = saslStreams.in;
+ break;
+ } catch (InvalidEncryptionKeyException e) {
+ IOUtils.closeSocket(sock);
+ if (++encryptionKeyRetryCount > 1) {
+ throw e;
+ }
+ LOG.info("Retrying connection to {} for block {} after "
+ + "InvalidEncryptionKeyException, will wait before retry",
+ target.getDatanodeInfo(), reportedBlock.getBlock(), e);
+ km.clearDataEncryptionKey();
+ try {
+ Thread.sleep(HdfsConstants.READ_TIMEOUT);
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ throw e;
+ }
Review Comment:
Thanks for the review. I removed the sleep.
> Handle InvalidEncryptionKeyException in Balancer Dispatcher, SPS
> BlockDispatcher and DataNode DataTransfer
> -----------------------------------------------------------------------------------------------------------
>
> Key: HDFS-17899
> URL: https://issues.apache.org/jira/browse/HDFS-17899
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, datanode, encryption
> Affects Versions: 3.4.1
> Reporter: ZhenyuLi
> Assignee: ZhenyuLi
> Priority: Major
> Labels: pull-request-available
>
> This is a follow-up of HDFS-17897.
> HDFS-17897 fixed {{InvalidEncryptionKeyException}} handling in DFSClient
> read/write and striped file checksum paths. However, three other
> code paths that establish SASL-encrypted connections still lack this
> handling:
> {{Dispatcher.PendingMove.dispatch()}} — Balancer block moves
> {{BlockDispatcher.moveBlock()}} — SPS block moves
> {{DataNode.DataTransfer.run()}} — DataNode block replication
> When {{dfs.encrypt.data.transfer=true}} and block keys rotate, these paths
> are possible to fail with {{InvalidEncryptionKeyException}} and the stale key
> stays cached, causing all subsequent transfers to fail until process restart.
> Fix: Add the same retry pattern to all three paths — catch the exception,
> clear the cached encryption key via a new {{clearDataEncryptionKey()}}
> default
> method on {{DataEncryptionKeyFactory}}, and retry once with a fresh key.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
