https://bz.apache.org/bugzilla/show_bug.cgi?id=65408
--- Comment #15 from Thomas Sartissohn <t.sartiss...@seeburger.de> --- Some update on this topic. We use the tomcat embeddded version 8.5.x Since a while we faced the same issue. We see NullPointer exceptions when working on the response object and java.lang.AssertionErrors when a new request arrives. We then finally drilled the issue down to a misconfiguration in our ThreadPoolExecutor. There the Linkedblocking queue to store the processor requests in was too small. When the queue is too small and the executor is out of threads, any new request is rejected with an RejectedExecutionException. This execption is handled in the tomcat and cancelledKey is executed which recycles the assigned processor. (See stack below) And here seems there is an issue in the tomcat code somewhere. It happens that the call of cancelledKeys recycles a processor which is currently in use. This explains the NullPointer we get when sending the response. The processor was recycled while we are working on. That is why we are no longer able to send the response. But there is also another issue. When we call cancelledKey while the processor is still in work the processor ends finally twice in the recycledProcessor stack. We have seen that in the log that the same processor was pushed twice to the recycledStack without a pop in between. Also a heap dump showed that the recycledProcessor stack was filled with the same processor object. Having the same processor more then once on the recycled stack brings us then to the AssertionError. When we have two therads getting the same processor out of the stack we will get the AssertionError. We fixed the issue with increasing the queue size for the Executor. However maybe the processor handling should reviewed regarding the cancelledKey event and pushing the same object more the once to the stack. The relevant stacks are 2022-02-08T16:33:25.113+0000 WARN Executor rejected socket for processing java.util.concurrent.RejectedExecutionException: Task org.apache.tomcat.util.net.NioEndpoint$SocketProcessor@10842846 rejected from c.s.ListenerThreadPoolExecutor@4d77f359[Running, pool size = 250, active threads = 130, queued tasks = 247, completed tasks = 27887] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355) ~[?:?] at org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1105) ~[?:?] at org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:896) ~[?:?] at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:872) ~[?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2022-02-08T16:33:25.117+0000 INFO HTTP Listener Encountered a non-recycled request and recycled it forcedly. org.apache.catalina.connector.CoyoteAdapter$RecycleRequiredException: null at org.apache.catalina.connector.CoyoteAdapter.checkRecycled(CoyoteAdapter.java:546) ~[?:?] at org.apache.coyote.http11.Http11Processor.recycle(Http11Processor.java:1578) ~[?:?] at org.apache.coyote.AbstractProtocol$ConnectionHandler.release(AbstractProtocol.java:1021) ~[?:?] at org.apache.coyote.AbstractProtocol$ConnectionHandler.release(AbstractProtocol.java:1054) ~[?:?] at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:769) ~[?:?] at org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:906) ~[?:?] at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:872) ~[?:?] at java.lang.Thread.run(Thread.java:829) [?:?] -- You are receiving this mail because: You are the assignee for the bug. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
