Thanks for your response. I managed to snag a tcp dump of what's going on in this scenario. From what I can see the sequence of events is the following. Recall that our Jetty server is fronted by a Varnish cache.
1) Varnish sends the headers and initial part of the content for a large POST. 2) On the Jetty server, we use a streaming parser and begin validating the content. 3) We detect a problem with the content and throw an exception that results in a 400 Bad Request to the client (via JAX-RS exception mapper) 4) An ACK is sent for the segment containing the 400 error. 5) The Jetty server sends a FIN. 6) An ACK is sent for the FIN 7) Varnish sends another segment that continues the content from #1. 8) The Jetty server sends a RST. In the server logs, we see an Early EOF from our JAX-RS resource that is parsing the content. This all seems pretty ok from the Jetty side, and it certainly seems like Varnish is misbehaving here (I'm thinking it may be this bug https://github.com/varnishcache/varnish-cache/issues/2332). But I'm still unclear as to why this started after our upgrade from Jetty 9.2 -> 9.4. Any thoughts? On Thu, Sep 20, 2018 at 5:14 AM Simone Bordet <[email protected]> wrote: > Hi, > > On Wed, Sep 19, 2018 at 4:06 PM Tommy Becker <[email protected]> wrote: > > > > Hi folks, > > We recently upgraded our application from Jetty 9.2 to 9.4. After doing > so, we've noticed a lot of Early EOF stacktraces happening on POST > requests. Of note, our application is fronted by a Varnish cache, which > holds connections to the Jetty backend open indefinitely, even though we > have the Jetty idle timeout configured to 50s. We're aware this is sub > optimal, but it has been this way for some time, and worked fine under > Jetty 9.2. To be clear, we do not have any evidence, or even believe that > Jetty is doing anything wrong here per-se. But I understand that a fair bit > of work happened in Jetty in this area (timeout/socket close) between 9.2 > and 9.4 and I'm trying to get a high level description of what changed in > an attempt to explain the behavior we're seeing now. Any info you could > provide would be appreciated! > > > > There have been many changes between Jetty 9.2 and 9.4 in many places. > As an example, we have improved and made more compliant/strict HTTP > parsing, and that may be the cause of your POSTs not going through. > > The best thing you can do to figure out this issue is to grab a > network trace via Wireshark on the Jetty host, along with possibly a > stack trace (that you did not report and you don't even say if it's on > client or on server), and possibly server DEBUG logs. > > The network trace will be important to know who's closing the > connection. Stack traces and logs to know why it was closed. > > -- > Simone Bordet > ---- > http://cometd.org > http://webtide.com > Developer advice, training, services and support > from the Jetty & CometD experts. > _______________________________________________ > jetty-users mailing list > [email protected] > To change your delivery options, retrieve your password, or unsubscribe > from this list, visit > https://dev.eclipse.org/mailman/listinfo/jetty-users >
_______________________________________________ jetty-users mailing list [email protected] To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/jetty-users
