Makes sense, I wondered if that might be the case. I'm still curious how/where it decides to close the connection if it doesn't see the exception.
On Wed, Sep 26, 2018 at 7:39 AM Joakim Erdfelt <[email protected]> wrote: > Not sure if this is the case, but the "Connection: close" header cannot be > added by Jetty if the HttpServletResponse.isCommitted() == true before > using something like sendError(). > > Joakim Erdfelt / [email protected] > > > On Wed, Sep 26, 2018 at 6:29 AM Tommy Becker <[email protected]> wrote: > >> We definitely do not see one. But I'm still a bit confused as to how >> Jetty is determining that it wants to close the connection. Although our >> JAX-RS resource throws an exception, that exception should be handled by >> the JAX-RS runtime and not propagated to Jetty (We have ExceptionMappers >> defined for everything). So our application is generating the response and >> not Jetty itself. >> >> On Wed, Sep 26, 2018 at 12:49 AM Greg Wilkins <[email protected]> wrote: >> >>> >>> Thomas, >>> >>> I see a connection:close header when the connection is closed after >>> sending a 400 or 500. >>> >>> However, nothing is sent if the connection is closed due to an abort >>> while giving up reading unconsumed content, which can happen >>> before/during/after a response hence we keep that simple. >>> >>> So are you sure you are seeing a 400/500 response without >>> connection:close ? >>> >>> >>> >>> >>> On Wed, 26 Sep 2018 at 14:33, Greg Wilkins <[email protected]> wrote: >>> >>>> >>>> It is more about how the response was generated and less about the >>>> response code itself. >>>> If the application throws and exception to Jetty during request >>>> handling, we now always make the connection non persistent before trying to >>>> send a response. If the request input is terminated early or is not fully >>>> consumed and would block, then we also abort the connection. >>>> >>>> Interesting that you say we don't set the Connection: close header. >>>> There is actually no requirement to do so as the server can close a >>>> connection at any time, but I thought we would do so as a courtesy.... >>>> checking.... >>>> >>>> cheers >>>> >>>> >>>> >>>> On Wed, 26 Sep 2018 at 10:25, Tommy Becker <[email protected]> wrote: >>>> >>>>> Thanks Greg. Just so I’m clear, what does Jetty key on to know whether >>>>> to close the connection? Just the 4xx/5xx response code? I’m trying to >>>>> understand the difference between this case and the “normal unconsumed >>>>> input” case you describe. Also, I did notice that Jetty does not set the >>>>> Connection: close header when it does this, is that intentional? >>>>> >>>>> >>>>> On Sep 25, 2018, at 6:37 PM, Greg Wilkins <[email protected]> wrote: >>>>> >>>>> >>>>> Thomas, >>>>> >>>>> There is no configuration to avoid this behaviour. If jetty sees and >>>>> exception in the application it will send the 400 and close the >>>>> connection. >>>>> >>>>> However, as Simone says, your application can be setup to avoid this >>>>> situation by catching the exception and consuming any input. You can do >>>>> this in a filter that catches Throwable, it can then check the request >>>>> input stream (and/or reader) for unconsumed input and read & discard to >>>>> end >>>>> of file. If the response is not committed, it can then send a 400 or any >>>>> other response that you like. >>>>> >>>>> Just remember that this may make your application somewhat vulnerable >>>>> to DOS attacks as it will be easy to hold a thread in that filter slowly >>>>> consuming data. I would suggest imposing a total time and total data >>>>> limit >>>>> on the input consumption. >>>>> >>>>> Note that for normal unconsumed input, jetty 9.4 does make some >>>>> attempt to consume it... but if the reading of that data would block, it >>>>> gives up and closes the connection, as there is no point blocking for data >>>>> that will be discarded. >>>>> >>>>> regards >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, 26 Sep 2018 at 07:35, Thomas Becker <[email protected]> >>>>> wrote: >>>>> >>>>>> Thanks so much again for your response, this is great information. >>>>>> What you say makes sense, but I now see I failed to mention the most >>>>>> critical part of this problem. Which is that the client never actually >>>>>> sees >>>>>> the 400 response we are sending from Jetty. When varnish sees the RST, it >>>>>> considers the backend request failed and returns 503 Service Unavailable >>>>>> to >>>>>> the client, effectively swallowing our application’s response. We can >>>>>> pursue a solution to this on the Varnish side, but in the interim I’m >>>>>> guessing there is no way to configure this behavior in Jetty? >>>>>> >>>>>> >>>>>> >>>>>> On Sep 25, 2018, at 4:28 PM, Simone Bordet <[email protected]> >>>>>> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> On Tue, Sep 25, 2018 at 8:34 PM Tommy Becker <[email protected]> >>>>>> wrote: >>>>>> >>>>>> >>>>>> Update: we setup an environment with the old Jetty 9.2 code and this >>>>>> does not occur. 9.2 does not send the FIN in #5 above, and seems happy to >>>>>> receive the rest of the content, despite having sent a response already. >>>>>> >>>>>> On Tue, Sep 25, 2018 at 10:01 AM Tommy Becker <[email protected]> >>>>>> wrote: >>>>>> >>>>>> >>>>>> Thanks for your response. I managed to snag a tcp dump of what's >>>>>> going on in this scenario. From what I can see the sequence of events is >>>>>> the following. Recall that our Jetty server is fronted by a Varnish >>>>>> cache. >>>>>> >>>>>> 1) Varnish sends the headers and initial part of the content for a >>>>>> large POST. >>>>>> 2) On the Jetty server, we use a streaming parser and begin >>>>>> validating the content. >>>>>> 3) We detect a problem with the content and throw an exception that >>>>>> results in a 400 Bad Request to the client (via JAX-RS exception mapper) >>>>>> 4) An ACK is sent for the segment containing the 400 error. >>>>>> 5) The Jetty server sends a FIN. >>>>>> 6) An ACK is sent for the FIN >>>>>> 7) Varnish sends another segment that continues the content from #1. >>>>>> 8) The Jetty server sends a RST. >>>>>> >>>>>> In the server logs, we see an Early EOF from our JAX-RS resource that >>>>>> is parsing the content. This all seems pretty ok from the Jetty side, and >>>>>> it certainly seems like Varnish is misbehaving here (I'm thinking it may >>>>>> be >>>>>> this bug https://github.com/varnishcache/varnish-cache/issues/2332). >>>>>> But I'm still unclear as to why this started after our upgrade from Jetty >>>>>> 9.2 -> 9.4. Any thoughts? >>>>>> >>>>>> >>>>>> This is normal. >>>>>> In Jetty 9.4 we are more aggressive in closing the connection because >>>>>> we don't want to be at the mercy of a possible nasty client sending us >>>>>> GiB of data when we know the application does not want to handle them. >>>>>> Varnish behavior is correct too: it sees the FIN from Jetty but does >>>>>> not know that Jetty does not want to read until it tries to send more >>>>>> content and gets a RST. >>>>>> At that point, it should relay the RST (or FIN) back to the client. >>>>>> >>>>>> So you have 2 choices: you catch the exception during your validation, >>>>>> and finish to read (and discard) the content in the application; or >>>>>> you ignore the early EOFs in the logs. >>>>>> I don't think that those early EOFs are logged above DEBUG level, is >>>>>> that correct? >>>>>> >>>>>> -- >>>>>> Simone Bordet >>>>>> ---- >>>>>> http://cometd.org >>>>>> http://webtide.com >>>>>> Developer advice, training, services and support >>>>>> from the Jetty & CometD experts. >>>>>> _______________________________________________ >>>>>> jetty-users mailing list >>>>>> [email protected] >>>>>> To change your delivery options, retrieve your password, or >>>>>> unsubscribe from this list, visit >>>>>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> jetty-users mailing list >>>>>> [email protected] >>>>>> To change your delivery options, retrieve your password, or >>>>>> unsubscribe from this list, visit >>>>>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>>>> >>>>> >>>>> >>>>> -- >>>>> Greg Wilkins <[email protected]> CTO http://webtide.com >>>>> _______________________________________________ >>>>> jetty-users mailing list >>>>> [email protected] >>>>> To change your delivery options, retrieve your password, or >>>>> unsubscribe from this list, visit >>>>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>>>> >>>>> >>>>> _______________________________________________ >>>>> jetty-users mailing list >>>>> [email protected] >>>>> To change your delivery options, retrieve your password, or >>>>> unsubscribe from this list, visit >>>>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>>> >>>> >>>> >>>> -- >>>> Greg Wilkins <[email protected]> CTO http://webtide.com >>>> >>> >>> >>> -- >>> Greg Wilkins <[email protected]> CTO http://webtide.com >>> _______________________________________________ >>> jetty-users mailing list >>> [email protected] >>> To change your delivery options, retrieve your password, or unsubscribe >>> from this list, visit >>> https://dev.eclipse.org/mailman/listinfo/jetty-users >> >> _______________________________________________ >> jetty-users mailing list >> [email protected] >> To change your delivery options, retrieve your password, or unsubscribe >> from this list, visit >> https://dev.eclipse.org/mailman/listinfo/jetty-users > > _______________________________________________ > jetty-users mailing list > [email protected] > To change your delivery options, retrieve your password, or unsubscribe > from this list, visit > https://dev.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________ jetty-users mailing list [email protected] To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/jetty-users
