Hi, After 3 days running, my zookeeper showing this error.
2019-07-29 15:10:41,906 [myid:1] - WARN [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1176] - Connection broken for id 4332550065071534382, my id = 1, error = java.io.IOException: Received packet with invalid packet: 824196618 at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163) 2019-07-29 15:10:41,906 [myid:1] - WARN [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1179] - Interrupting SendWorker 2019-07-29 15:10:41,907 [myid:1] - WARN [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1092] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080) 2019-07-29 15:10:41,907 [myid:1] - WARN [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1102] - Send worker leaving thread id 4332550065071534382 my id = 1 2019-07-29 15:10:41,917 [myid:1] - INFO [/177.55.55.152:3888 :QuorumCnxManager$Listener@888] - Received connection request / 177.55.55.63:53972 2019-07-29 15:10:41,920 [myid:1] - WARN [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1176] - Connection broken for id 4332550065071534382, my id = 1, error = java.io.IOException: Received packet with invalid packet: 840973834 at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163) 2019-07-29 15:10:41,921 [myid:1] - WARN [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1179] - Interrupting SendWorker 2019-07-29 15:10:41,922 [myid:1] - WARN [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1092] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080) 2019-07-29 15:10:41,922 [myid:1] - WARN [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1102] - Send worker leaving thread id 4332550065071534382 my id = 1 2019-07-29 15:10:41,932 [myid:1] - INFO [/177.55.55.152:3888 :QuorumCnxManager$Listener@888] - Received connection request / 177.55.55.63:38633 2019-07-29 15:10:41,933 [myid:1] - WARN [RecvWorker:4332550065071534638:QuorumCnxManager$RecvWorker@1176] - Connection broken for id 4332550065071534638, my id = 1, error = java.io.IOException: Received packet with invalid packet: 807419402 at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163) 2019-07-29 15:10:41,933 [myid:1] - WARN [RecvWorker:4332550065071534638:QuorumCnxManager$RecvWorker@1179] - Interrupting SendWorker 2019-07-29 15:10:41,934 [myid:1] - WARN [SendWorker:4332550065071534638:QuorumCnxManager$SendWorker@1092] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080) 2019-07-29 15:10:41,935 [myid:1] - WARN [SendWorker:4332550065071534638:QuorumCnxManager$SendWorker@1102] - Send worker leaving thread id 4332550065071534638 my id = 1 2019-07-29 15:10:41,981 [myid:1] - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: Len error 1008750420 2019-07-29 15:10:41,987 [myid:1] - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: Len error 1008750420 2019-07-29 15:10:41,992 [myid:1] - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: Len error 1008750420 2019-07-29 15:10:42,008 [myid:1] - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: Len error 1008750420 2019-07-29 15:10:45,375 [myid:1] - WARN [NIOWorkerThread-11:NIOServerCnxn@366] - Unable to read additional data from client sessionid 0x400044d531804b6, likely client has closed socket 2019-07-29 15:10:46,062 [myid:1] - WARN [NIOWorkerThread-4:NIOServerCnxn@366] - Unable to read additional data from client sessionid 0x300044d504c0471, likely client has closed socket 2019-07-29 15:11:12,965 [myid:1] - WARN [qtp1789550256-23:URIUtil@396] - /%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/etc/passwd org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! byte C0 in state 0 2019-07-29 15:11:17,588 [myid:1] - INFO [/177.153.44.152:3888 :QuorumCnxManager$Listener@888] - Received connection request / 177.55.55.63:33721 java.lang.OutOfMemoryError: Java heap space Dumping heap to java_pid97598.hprof ... 2019-07-29 15:11:17,590 [myid:1] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1176] - Connection broken for id 1, my id = 1, error = java.io.IOException: Received packet with invalid packet: 1966420818 at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163) 2019-07-29 15:11:19,452 [myid:1] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1179] - Interrupting SendWorker Heap dump file created [638922790 bytes in 3,132 secs] # # java.lang.OutOfMemoryError: Java heap space # -XX:OnOutOfMemoryError="kill -9 %p" # Executing /bin/sh -c "kill -9 97598"... That's right. Any tips ou tricks to workaround this error? How to increase the memory in Zookeeper? Look my zoo.conf. [15:30:09] root@eddison0001:/zoop/conf$ cat zoo.cfg # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/zoop/zookeeperdata # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to "0" to disable auto purge feature #autopurge.purgeInterval=1 oom_heap_dump_dir=/tmp zookeeper_server_java_heaps=4G server.1=177.55.55.152:2888:3888 server.2=177.55.55.153:2888:3888 server.3=177.55.55.154:2888:3888 server.4=177.55.55.155:2888:3888 server.5=177.55.55.156:2888:3888 Any workaround? Regards,
