https://bz.apache.org/bugzilla/show_bug.cgi?id=66077
Bug ID: 66077
Summary: Deadlock when concurrent processing of incoming HTTP/2
frames
Product: Tomcat 10
Version: 10.0.16
Hardware: All
OS: other
Status: NEW
Severity: normal
Priority: P2
Component: Connectors
Assignee: dev@tomcat.apache.org
Reporter: r...@tier1app.com
Target Milestone: ------
Application is running on
Tomcat: 10.0.16.0
JDK: 1.8.0_144-b01 Open JDK
OS Version: Cent OS 7.3.1611
All of sudden application is becoming unresponsive because 600+ threads are
getting in to BLOCKED states because of the deadlock caused in tomcat. Below
are the threads which is causing deadlock:
https-openssl-apr-443-exec-486
PRIORITY : 5
THREAD ID : 0X000000000220D800
NATIVE ID : 0X471A
NATIVE ID (DECIMAL) : 18202
STATE : BLOCKED
stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.coyote.http2.StreamStateMachine.receivedReset(StreamStateMachine.java:107)
- waiting to lock <0x00000006d8af60f8> (a
org.apache.coyote.http2.StreamStateMachine)
at org.apache.coyote.http2.Stream.receiveReset(Stream.java:180)
at
org.apache.coyote.http2.Http2UpgradeHandler.close(Http2UpgradeHandler.java:1200)
at
org.apache.coyote.http2.Http2UpgradeHandler.handleAppInitiatedIOException(Http2UpgradeHandler.java:803)
at
org.apache.coyote.http2.Http2UpgradeHandler.writeBody(Http2UpgradeHandler.java:785)
- locked <0x00000006d741dbb8> (a
org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper)
at org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:945)
- locked <0x00000006d86fbf18> (a
org.apache.coyote.http2.Stream$StreamOutputBuffer)
at org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:882)
- locked <0x00000006d86fbf18> (a
org.apache.coyote.http2.Stream$StreamOutputBuffer)
at org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:1005)
at org.apache.coyote.http2.Http2OutputBuffer.flush(Http2OutputBuffer.java:77)
at org.apache.coyote.http2.StreamProcessor.flush(StreamProcessor.java:242)
at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:402)
at org.apache.coyote.Response.action(Response.java:210)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:306)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:273)
at org.apache.catalina.connector.CoyoteWriter.flush(CoyoteWriter.java:94)
at org.apache.jasper.runtime.JspWriterImpl.flush(JspWriterImpl.java:162)
at
org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:981)
at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:489)
at org.apache.struts.taglib.template.InsertTag.doEndTag(InsertTag.java:158)
at
org.apache.jsp.error_jsp._jspx_meth_template_005finsert_005f1(error_jsp.java:254)
at org.apache.jsp.error_jsp._jspx_meth_html_005fhtml_005f0(error_jsp.java:171)
at org.apache.jsp.error_jsp._jspService(error_jsp.java:128)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:777)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:380)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:328)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:777)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:223)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:158)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:691)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:443)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:367)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:295)
at
org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:387)
at
org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:233)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:353)
at org.apache.coyote.http2.StreamProcessor.service(StreamProcessor.java:413)
at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:74)
- locked <0x00000006d70414c8> (a org.apache.coyote.http2.StreamProcessor)
at org.apache.coyote.http2.StreamRunnable.run(StreamRunnable.java:35)
at
org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
https-openssl-apr-443-exec-477
PRIORITY : 5
THREAD ID : 0X00007F71381AC000
NATIVE ID : 0X4711
NATIVE ID (DECIMAL) : 18193
STATE : BLOCKED
stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.coyote.http2.Http2UpgradeHandler.sendStreamReset(Http2UpgradeHandler.java:557)
- waiting to lock <0x00000006d741dbb8> (a
org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper)
at org.apache.coyote.http2.Stream.close(Stream.java:664)
- locked <0x00000006d8af60f8> (a org.apache.coyote.http2.StreamStateMachine)
at org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:102)
- locked <0x00000006d7041418> (a org.apache.coyote.http2.StreamProcessor)
at org.apache.coyote.http2.StreamRunnable.run(StreamRunnable.java:35)
at
org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
You can see the complete thread dump analysis report here:
https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjIvMDUvMTgvLS10aHJlYWRfZHVtcF8yODEwLS0xLTI1LTEw&&s=t
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
