[
https://issues.apache.org/jira/browse/HDFS-14186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated HDFS-14186:
---------------------------------
Description:
StyleIn the current implementation, the datanode sends blockreport immediately
after register to namenode successfully when restart, and the blockreport storm
will make namenode high load to process them. One result is some received RPC
have to skip because queue time is timeout. If some datanodes' heartbeat RPC
are continually skipped for long times (default is
heartbeatExpireInterval=630s) it will be set DEAD, then datanode has to
re-register and send blockreport again, aggravate blockreport storm and trap in
a vicious circle, and slow down (more than one hour and even more) namenode
startup seriously in a large (several thousands of datanodes) and busy cluster
especially. Although there are many work to optimize namenode startup, the
issue still exists.
I propose to postpone dead datanode check when namenode have finished startup.
Any comments and suggestions are welcome.
was:
In the current implementation, the datanode sends blockreport immediately after
register to namenode successfully when restart, and the blockreport storm will
make namenode high load to process them. One result is some received RPC have
to skip because queue time is timeout. If some datanodes' heartbeat RPC are
continually skipped for long times (default is heartbeatExpireInterval=630s) it
will be set DEAD, then datanode has to re-register and send blockreport again,
aggravate blockreport storm and trap in a vicious circle, and slow down (more
than one hour and even more) namenode startup seriously in a large (several
thousands of datanodes) and busy cluster especially. Although there are many
work to optimize namenode startup, the issue still exists.
I propose to postpone dead datanode check when namenode have finished startup.
Any comments and suggestions are welcome.
> blockreport storm slow down namenode restart seriously in large cluster
> -----------------------------------------------------------------------
>
> Key: HDFS-14186
> URL: https://issues.apache.org/jira/browse/HDFS-14186
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 2.7.1
> Reporter: Xiaoqiao He
> Assignee: Xiaoqiao He
> Priority: Major
> Attachments: HDFS-14186.001.patch
>
>
> StyleIn the current implementation, the datanode sends blockreport
> immediately after register to namenode successfully when restart, and the
> blockreport storm will make namenode high load to process them. One result is
> some received RPC have to skip because queue time is timeout. If some
> datanodes' heartbeat RPC are continually skipped for long times (default is
> heartbeatExpireInterval=630s) it will be set DEAD, then datanode has to
> re-register and send blockreport again, aggravate blockreport storm and trap
> in a vicious circle, and slow down (more than one hour and even more)
> namenode startup seriously in a large (several thousands of datanodes) and
> busy cluster especially. Although there are many work to optimize namenode
> startup, the issue still exists.
> I propose to postpone dead datanode check when namenode have finished startup.
> Any comments and suggestions are welcome.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
