Hello. I think I understood why the number of Readers threads is irrelevant in my case. The Readers threads are not responsible of the doAccept(), only the listener thread perform the Accept().
Which means that there is no way to increase the speed of acceptance of SYN requests for the namenode, unfortunately. The only way to mitigate this problem seems to increase again and again the listenQueue to not see the TCP errors. It is not perfect, but at least it mitigates the problem. On the other side, I'm going to monitore the SYN_RECV connections to namenode host and see if there is a specific client which could be too active. If anyone encounter this problem, hope this thread on the mailing list will help them a little bit ^_^. T@le Le jeu. 17 févr. 2022 à 17:24, Tale Hive <[email protected]> a écrit : > Hello. > > Increasing ipc.server.read.threadpool.size does not seem to help > unfortunately. > I can see in the logs more Readers launched : > 2022-02-17 15:19:35,942 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #2 for port 8040 > 2022-02-17 15:19:35,942 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #1 for port 8040 > 2022-02-17 15:19:35,943 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #4 for port 8040 > 2022-02-17 15:19:35,943 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #3 for port 8040 > 2022-02-17 15:19:35,943 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #5 for port 8040 > 2022-02-17 15:19:35,944 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #6 for port 8040 > 2022-02-17 15:19:35,944 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #7 for port 8040 > 2022-02-17 15:19:35,944 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #8 for port 8040 > 2022-02-17 15:19:35,945 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #9 for port 8040 > 2022-02-17 15:19:35,945 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #10 for port 8040 > 2022-02-17 15:19:35,961 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #1 for port 8020 > 2022-02-17 15:19:35,961 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #2 for port 8020 > 2022-02-17 15:19:35,962 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #3 for port 8020 > 2022-02-17 15:19:35,962 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #4 for port 8020 > 2022-02-17 15:19:35,962 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #5 for port 8020 > 2022-02-17 15:19:35,963 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #6 for port 8020 > 2022-02-17 15:19:35,963 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #7 for port 8020 > 2022-02-17 15:19:35,964 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #8 for port 8020 > 2022-02-17 15:19:35,964 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #9 for port 8020 > 2022-02-17 15:19:35,965 INFO ipc.Server (Server.java:run(1074)) - > Starting Socket Reader #10 for port 8020 > 2022-02-17 15:19:37,473 INFO ipc.Server (Server.java:run(1314)) - IPC > Server Responder: starting > 2022-02-17 15:19:37,473 INFO ipc.Server (Server.java:run(1153)) - IPC > Server listener on 8020: starting > 2022-02-17 15:19:37,943 INFO ipc.Server (Server.java:run(1314)) - IPC > Server Responder: starting > 2022-02-17 15:19:37,943 INFO ipc.Server (Server.java:run(1153)) - IPC > Server listener on 8040: starting > > I thought that starting more sockets reader could help to handle these TCP > errors and timeouts, but it seems not. > If someone has worked on this apache project, may you help me to better > understand this part, please ? > > Normally, I can see my ACK backlog for port 8020 is well set (I increased > it to 4096). > But why isn't the Accept() faster when I increased by 10 the number of > Readers ? > > Best regards. > > T@le > > > > > Le mer. 16 févr. 2022 à 15:00, Tale Hive <[email protected]> a écrit : > >> Hey guys. >> >> After some investigations, I can see I have sometimes the listen Queue >> overloaded for port 8020, thanks to a system metric provided by Grafana >> (TCP errors). >> >> I understand that it means the machine where the namenode is hosted >> receive too many SYN packets to handle them. >> Normally, these SYN packets should be ACKed but the listen queue for this >> port is full. >> So the excess of SYN packets are dropped and I got timeouts. >> >> An ss command showed me for this port 8020 : 129 in the RECV-Q and 128 in >> the SEND-Q at the same moment when I have the TCP errors. >> >> So I tried to increase the listen queue thanks to the property >> ipc.server.listen.queue.size. >> The default value is 128 so I bumped it to 256 and then 1024. But I still >> have the same behaviour. >> When I monitor the TCP errors metrics and with the ss command for port >> 8020, I still see the queue is frequently overloaded. >> >> So I was wondering if namenode was accepting fast enough the SYN packets. >> I checked the code and saw this class >> "hadoop-common-project\hadoop-common\src\main\java\org\apache\hadoop\ipc\Server.java". >> In this class, I can see a Listener method with readThreads variable >> which is related to the property ipc.server.read.threadpool.size. >> >> I cannot find this property in the documentation : >> >> https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-common/core-default.xml >> >> I can just find this in the code : >> >> /** Number of threads in RPC server reading from the socket */ >> public static final String IPC_SERVER_RPC_READ_THREADS_KEY = >> "ipc.server.read.threadpool.size"; >> /** Default value for IPC_SERVER_RPC_READ_THREADS_KEY */ >> public static final int IPC_SERVER_RPC_READ_THREADS_DEFAULT = 1; >> >> >> I have currently 200 threads for dfs.namenode.handler.countand 100 for >> dfs.namenode.service.handler.count. >> >> Is it safe to increase this property to 10 ? >> Will it impact other components like datanodes, journalnodes, zookeeper >> failover controller, resource managers, node managers, timeline server, >> history server or even hbase ? >> >> Thank you in advance for your answers ! >> >> T@le >> >> >> >> >> >> Le lun. 14 févr. 2022 à 16:56, Tale Hive <[email protected]> a écrit : >> >>> Hello. >>> >>> I encounter a strange problem with my namenode. I have the following >>> architecture : >>> - Two namenodes in HA >>> - 600 datanodes >>> - HDP 3.1.4 >>> - 150 millions of files and folders >>> >>> Sometimes, when I query the namenode with the hdfs client, I got a >>> timeout error like this : >>> hdfs dfs -ls -d /user/myuser >>> >>> 22/02/14 15:07:44 INFO retry.RetryInvocationHandler: >>> org.apache.hadoop.net.ConnectTimeoutException: Call From >>> <my-client-hostname>/<my-client-ip> to <active-namenode-hostname>:8020 >>> failed on socket timeout exception: >>> org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout >>> while waiting for channel to be ready for connect. ch : >>> java.nio.channels.SocketChannel[connection-pending >>> remote=<active-namenode-hostname>/<active-namenode-ip>:8020]; >>> For more details see: http://wiki.apache.org/hadoop/SocketTimeout, >>> while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over >>> <active-namenode-hostname>/<active-namenode-ip>:8020 after 2 failover >>> attempts. Trying to failover after sleeping for 2694ms. >>> >>> I checked the heap of the namenode and there is no problem (I have 75 GB >>> of max heap, I'm around 50 used GB). >>> I checked the threads of the clientRPC for the namenode and I'm at 200 >>> which respects the recommandations from hadoop operations book. >>> I have serviceRPC enabled to prevent any problem which could be coming >>> from datanodes or ZKFC. >>> General resources seems OK, CPU usage is pretty fine, same for memory, >>> network or IO. >>> No firewall is enabled on my namenodes nor my client. >>> >>> I was wondering what could cause this problem, please ? >>> >>> Thank you in advance for your help ! >>> >>> Best regards. >>> >>> T@le >>> >>
