We definitely do not see one. But I'm still a bit confused as to how Jetty is determining that it wants to close the connection. Although our JAX-RS resource throws an exception, that exception should be handled by the JAX-RS runtime and not propagated to Jetty (We have ExceptionMappers defined for everything). So our application is generating the response and not Jetty itself.
On Wed, Sep 26, 2018 at 12:49 AM Greg Wilkins <[email protected]> wrote: > > Thomas, > > I see a connection:close header when the connection is closed after > sending a 400 or 500. > > However, nothing is sent if the connection is closed due to an abort while > giving up reading unconsumed content, which can happen before/during/after > a response hence we keep that simple. > > So are you sure you are seeing a 400/500 response without connection:close > ? > > > > > On Wed, 26 Sep 2018 at 14:33, Greg Wilkins <[email protected]> wrote: > >> >> It is more about how the response was generated and less about the >> response code itself. >> If the application throws and exception to Jetty during request handling, >> we now always make the connection non persistent before trying to send a >> response. If the request input is terminated early or is not fully consumed >> and would block, then we also abort the connection. >> >> Interesting that you say we don't set the Connection: close header. >> There is actually no requirement to do so as the server can close a >> connection at any time, but I thought we would do so as a courtesy.... >> checking.... >> >> cheers >> >> >> >> On Wed, 26 Sep 2018 at 10:25, Tommy Becker <[email protected]> wrote: >> >>> Thanks Greg. Just so I’m clear, what does Jetty key on to know whether >>> to close the connection? Just the 4xx/5xx response code? I’m trying to >>> understand the difference between this case and the “normal unconsumed >>> input” case you describe. Also, I did notice that Jetty does not set the >>> Connection: close header when it does this, is that intentional? >>> >>> >>> On Sep 25, 2018, at 6:37 PM, Greg Wilkins <[email protected]> wrote: >>> >>> >>> Thomas, >>> >>> There is no configuration to avoid this behaviour. If jetty sees and >>> exception in the application it will send the 400 and close the connection. >>> >>> However, as Simone says, your application can be setup to avoid this >>> situation by catching the exception and consuming any input. You can do >>> this in a filter that catches Throwable, it can then check the request >>> input stream (and/or reader) for unconsumed input and read & discard to end >>> of file. If the response is not committed, it can then send a 400 or any >>> other response that you like. >>> >>> Just remember that this may make your application somewhat vulnerable to >>> DOS attacks as it will be easy to hold a thread in that filter slowly >>> consuming data. I would suggest imposing a total time and total data limit >>> on the input consumption. >>> >>> Note that for normal unconsumed input, jetty 9.4 does make some attempt >>> to consume it... but if the reading of that data would block, it gives up >>> and closes the connection, as there is no point blocking for data that will >>> be discarded. >>> >>> regards >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Wed, 26 Sep 2018 at 07:35, Thomas Becker <[email protected]> wrote: >>> >>>> Thanks so much again for your response, this is great information. What >>>> you say makes sense, but I now see I failed to mention the most critical >>>> part of this problem. Which is that the client never actually sees the 400 >>>> response we are sending from Jetty. When varnish sees the RST, it considers >>>> the backend request failed and returns 503 Service Unavailable to the >>>> client, effectively swallowing our application’s response. We can pursue a >>>> solution to this on the Varnish side, but in the interim I’m guessing there >>>> is no way to configure this behavior in Jetty? >>>> >>>> >>>> >>>> On Sep 25, 2018, at 4:28 PM, Simone Bordet <[email protected]> wrote: >>>> >>>> Hi, >>>> >>>> On Tue, Sep 25, 2018 at 8:34 PM Tommy Becker <[email protected]> >>>> wrote: >>>> >>>> >>>> Update: we setup an environment with the old Jetty 9.2 code and this >>>> does not occur. 9.2 does not send the FIN in #5 above, and seems happy to >>>> receive the rest of the content, despite having sent a response already. >>>> >>>> On Tue, Sep 25, 2018 at 10:01 AM Tommy Becker <[email protected]> >>>> wrote: >>>> >>>> >>>> Thanks for your response. I managed to snag a tcp dump of what's going >>>> on in this scenario. From what I can see the sequence of events is the >>>> following. Recall that our Jetty server is fronted by a Varnish cache. >>>> >>>> 1) Varnish sends the headers and initial part of the content for a >>>> large POST. >>>> 2) On the Jetty server, we use a streaming parser and begin validating >>>> the content. >>>> 3) We detect a problem with the content and throw an exception that >>>> results in a 400 Bad Request to the client (via JAX-RS exception mapper) >>>> 4) An ACK is sent for the segment containing the 400 error. >>>> 5) The Jetty server sends a FIN. >>>> 6) An ACK is sent for the FIN >>>> 7) Varnish sends another segment that continues the content from #1. >>>> 8) The Jetty server sends a RST. >>>> >>>> In the server logs, we see an Early EOF from our JAX-RS resource that >>>> is parsing the content. This all seems pretty ok from the Jetty side, and >>>> it certainly seems like Varnish is misbehaving here (I'm thinking it may be >>>> this bug https://github.com/varnishcache/varnish-cache/issues/2332). >>>> But I'm still unclear as to why this started after our upgrade from Jetty >>>> 9.2 -> 9.4. Any thoughts? >>>> >>>> >>>> This is normal. >>>> In Jetty 9.4 we are more aggressive in closing the connection because >>>> we don't want to be at the mercy of a possible nasty client sending us >>>> GiB of data when we know the application does not want to handle them. >>>> Varnish behavior is correct too: it sees the FIN from Jetty but does >>>> not know that Jetty does not want to read until it tries to send more >>>> content and gets a RST. >>>> At that point, it should relay the RST (or FIN) back to the client. >>>> >>>> So you have 2 choices: you catch the exception during your validation, >>>> and finish to read (and discard) the content in the application; or >>>> you ignore the early EOFs in the logs. >>>> I don't think that those early EOFs are logged above DEBUG level, is >>>> that correct? >>>> >>>> -- >>>> Simone Bordet >>>> ---- >>>> http://cometd.org >>>> http://webtide.com >>>> Developer advice, training, services and support >>>> from the Jetty & CometD experts. >>>> _______________________________________________ >>>> jetty-users mailing list >>>> [email protected] >>>> To change your delivery options, retrieve your password, or unsubscribe >>>> from this list, visit >>>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>>> >>>> >>>> _______________________________________________ >>>> jetty-users mailing list >>>> [email protected] >>>> To change your delivery options, retrieve your password, or unsubscribe >>>> from this list, visit >>>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>> >>> >>> >>> -- >>> Greg Wilkins <[email protected]> CTO http://webtide.com >>> _______________________________________________ >>> jetty-users mailing list >>> [email protected] >>> To change your delivery options, retrieve your password, or unsubscribe >>> from this list, visit >>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>> >>> >>> _______________________________________________ >>> jetty-users mailing list >>> [email protected] >>> To change your delivery options, retrieve your password, or unsubscribe >>> from this list, visit >>> https://dev.eclipse.org/mailman/listinfo/jetty-users >> >> >> >> -- >> Greg Wilkins <[email protected]> CTO http://webtide.com >> > > > -- > Greg Wilkins <[email protected]> CTO http://webtide.com > _______________________________________________ > jetty-users mailing list > [email protected] > To change your delivery options, retrieve your password, or unsubscribe > from this list, visit > https://dev.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________ jetty-users mailing list [email protected] To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/jetty-users
