https://bz.apache.org/bugzilla/show_bug.cgi?id=65408
--- Comment #15 from Thomas Sartissohn ---
Some update on this topic. We use the tomcat embeddded version 8.5.x
Since a while we faced the same issue. We see NullPointer exceptions when
working on the response object and java.lang.AssertionErrors when a new request
arrives.
We then finally drilled the issue down to a misconfiguration in our
ThreadPoolExecutor. There the Linkedblocking queue to store the processor
requests in was too small.
When the queue is too small and the executor is out of threads, any new request
is rejected with an RejectedExecutionException.
This execption is handled in the tomcat and cancelledKey is executed which
recycles the assigned processor. (See stack below)
And here seems there is an issue in the tomcat code somewhere. It happens that
the call of cancelledKeys recycles a processor which is currently in use. This
explains the NullPointer we get when sending the response. The processor was
recycled while we are working on. That is why we are no longer able to send the
response.
But there is also another issue. When we call cancelledKey while the processor
is still in work the processor ends finally twice in the recycledProcessor
stack. We have seen that in the log that the same processor was pushed twice to
the recycledStack without a pop in between.
Also a heap dump showed that the recycledProcessor stack was filled with the
same processor object.
Having the same processor more then once on the recycled stack brings us then
to the AssertionError. When we have two therads getting the same processor out
of the stack we will get the AssertionError.
We fixed the issue with increasing the queue size for the Executor. However
maybe the processor handling should reviewed regarding the cancelledKey event
and pushing the same object more the once to the stack.
The relevant stacks are
2022-02-08T16:33:25.113+ WARN Executor rejected socket for processing
java.util.concurrent.RejectedExecutionException: Task
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor@10842846 rejected from
c.s.ListenerThreadPoolExecutor@4d77f359[Running, pool size = 250, active
threads = 130, queued tasks = 247, completed tasks = 27887]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
~[?:?]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
~[?:?]
at
org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1105)
~[?:?]
at
org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:896)
~[?:?]
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:872)
~[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2022-02-08T16:33:25.117+ INFO HTTP Listener Encountered a non-recycled
request and recycled it forcedly.
org.apache.catalina.connector.CoyoteAdapter$RecycleRequiredException: null
at
org.apache.catalina.connector.CoyoteAdapter.checkRecycled(CoyoteAdapter.java:546)
~[?:?]
at org.apache.coyote.http11.Http11Processor.recycle(Http11Processor.java:1578)
~[?:?]
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.release(AbstractProtocol.java:1021)
~[?:?]
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.release(AbstractProtocol.java:1054)
~[?:?]
at
org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:769)
~[?:?]
at
org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:906)
~[?:?]
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:872)
~[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
--
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
