https://bz.apache.org/bugzilla/show_bug.cgi?id=64082
Bug ID: 64082
Summary: Nio2Endpoint for async request doesn't clear
OutputBuffer when socket has already been closed
(response mixup)
Product: Tomcat 8
Version: 8.5.50
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: Connectors
Assignee: dev@tomcat.apache.org
Reporter: william.crow...@roguewave.com
Target Milestone: ----
This issue is similar to what is described in resolved bug 63022:
https://bz.apache.org/bugzilla/show_bug.cgi?id=63022
We use OpenIG 6.5.1 as a reverse proxy which copies the contents received from
the backend server into HttpServlet's OutputStream. The following link is the
servlet we are using here:
https://github.com/WrenArchiver/forgerock-http-framework/blob/master/http-servlet/src/main/java/org/forgerock/http/servlet/HttpFrameworkServlet.java
We are running into an issue where the user agent (browser) is receiving a
response which was meant for another user. It appears Tomcat does not
clear/recycle the response properly when the socket has already been closed. We
downgraded the Tomcat version from 8.5.47 to 8.5.35 to see if bringing back the
change that was removed by bug 63022 will resolve their issue. So far, the
issue has not been recreated with 8.5.35 with a change to
org.apache.tomcat.util.net.Nio2Endpoint.isClosed():
https://github.com/apache/tomcat/blob/8.5.37/java/org/apache/tomcat/util/net/Nio2Endpoint.java
...
@Override
public boolean isClosed() {
return closed || !getSocket().isOpen();
}
...
This issue also occurs in Tomcat 9.0.30.
Tomcat added a boolean variable to check whether a request/socket is closed or
not (revision 1833907). Which is "closed" flag in addition to
!getSocket().isOpen() inside org.apache.tomcat.util.net.Nio2Endpoint.
!getSocket()isOpen() check was removed starting in Tomcat 8.5.38 in bug 63022
to let Tomcat complete async request gracefully to increment
connectionCount(LimitLatch) for JMX.
However, bug 61918 (referenced from 63022) states:
"I'm not convinced closing sockets like this is a really good idea. The patch
may reintroduce more serious problems, but adding a closed flag seems
possible."
We see a response mixup which should have been fixed with CVE-2018-8037 because
of the change done in bug 63022 as our issue is not recreated on 8.5.35 which
does not have fix for bug 63022.
We have confirmed with the OpenIG log that the response received from backend
application is in tact. We have provided data from production environment
which used several different backend applications but had a response mix-up. We
have never encountered the response mix-up issue until OpenIG was running on
tomcat 8.5.47. We cannot recreate the issue using OpenIG running on Jetty with
the same test application. We have not been able to recreate the issue using
OpenIG running on Tomcat v8.5.35. When the mixup happens, HttpFrameworkServlet
was sometimes passed HttpServletResponse object which contained headers from
other requests:
https://github.com/WrenArchiver/forgerock-http-framework/blob/master/http-servlet/src/main/java/org/forgerock/http/servlet/HttpFrameworkServlet.java
Attaching some errors we see around the time of issue.
----
[2020-01-04 15:45:32,273] ERROR [I/O dispatcher 5242] o.f.u.p.PromiseImpl
[green] - Ignored unexpected exception thrown by ResultHandler
java.lang.IllegalStateException: Calling [asyncComplete()] is not valid for a
request with Async state [MUST_ERROR]
at
org.apache.coyote.AsyncStateMachine.doComplete(AsyncStateMachine.java:331)
at
org.apache.coyote.AsyncStateMachine.asyncComplete(AsyncStateMachine.java:316)
at
org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:502)
at org.apache.coyote.Request.action(Request.java:431)
at
org.apache.catalina.core.AsyncContextImpl.complete(AsyncContextImpl.java:92)
at
org.forgerock.http.servlet.Servlet3Adapter$ServletAsynchronousSynchronizer.signalAndComplete(Servlet3Adapter.java:75)
----
----
[2020-01-02 20:16:56,354] ERROR [AsyncStreamingHttpClient-promises-3530]
o.f.u.p.PromiseImpl [green] - Ignored unexpected exception thrown by
ResultHandler
java.lang.IllegalStateException: The request associated with the AsyncContext
has already completed processing.
at
org.apache.catalina.core.AsyncContextImpl.check(AsyncContextImpl.java:508)
at
org.apache.catalina.core.AsyncContextImpl.complete(AsyncContextImpl.java:91)
at
org.forgerock.http.servlet.Servlet3Adapter$ServletAsynchronousSynchronizer.signalAndComplete(Servlet3Adapter.java:75)
at
org.forgerock.http.servlet.HttpFrameworkServlet.writeResponse(HttpFrameworkServlet.java:343)
at
org.forgerock.http.servlet.HttpFrameworkServlet.lambda$service$0(HttpFrameworkServlet.java:265)
at
org.forgerock.util.promise.PromiseImpl.lambda$thenOnResult$1(PromiseImpl.java:292)
at
org.forgerock.util.promise.PromiseImpl.lambda$then$6(PromiseImpl.java:374)
at
org.forgerock.util.promise.PromiseImpl.handleCompletion(PromiseImpl.java:536)
at
org.forgerock.util.promise.PromiseImpl.setState(PromiseImpl.java:577)
at
org.forgerock.util.promise.PromiseImpl.tryHandleResult(PromiseImpl.java:258)
at
org.forgerock.util.promise.PromiseImpl.handleResult(PromiseImpl.java:208)
----
or
----
[2020-01-02 00:42:13,595] INFO [I/O dispatcher 2350]
o.f.o.f.S.UnauthorizedLogger [blue] - [TRANSACTION_ID:
a9a42e42-adb3-4fc8-953e-58309f5aed54-5743700][URI:
https://xxx.com/xxx/test.css][METHOD: GET][UID: XXXXX]Unauthorized.
[2020-01-02 00:42:38,890] ERROR [I/O dispatcher 6597]
o.f.h.s.HttpFrameworkServlet [blue] - Failed to write response
org.apache.catalina.connector.ClientAbortException: java.io.IOException:
Connection reset by peer
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:372)
at
org.apache.catalina.connector.OutputBuffer.appendByteArray(OutputBuffer.java:811)
at
org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:740)
at
org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:407)
at
org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:385)
at
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96)
at org.forgerock.http.io.IO.stream(IO.java:290)
at
org.forgerock.http.servlet.HttpFrameworkServlet.writeResponse(HttpFrameworkServlet.java:372)
at
org.forgerock.http.servlet.HttpFrameworkServlet.writeResponse(HttpFrameworkServlet.java:340)
at
org.forgerock.http.servlet.HttpFrameworkServlet.lambda$service$0(HttpFrameworkServlet.java:265)
----
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
