Thanks for your help. I have opened https://github.com/eclipse/jetty.project/issues/1705
Cheers > > ------------------------------ > > Message: 2 > Date: Wed, 2 Aug 2017 21:56:43 +1000 > From: Greg Wilkins <[email protected]> > To: JETTY user mailing list <[email protected]> > Subject: Re: [jetty-users] Leaked EndPoint objects in > AbstractConnector._endpoints due to ManagedSelector. > destroyEndPoint > task rejected by QueuedTheadPool > Message-ID: > <CAAPGdfFp-q=_9xHd4tXt34UWnyydC7V6Msi4emz8tXN9B3DPzg@mail. > gmail.com> > Content-Type: text/plain; charset="utf-8" > > Julien, > > Having the QueuedThreadPool configured so that it may reject jobs is not > recommended and is likely to result in problems. More over, having such a > small threadpool (4 threads!) and a small queue (also 4) is going to mean > that only moderately low load is required for jobs to be discarded. > > Having said that, we should not leak memory. However in this case, if an > task that would have destroyed an endpoint is lost, then the endpoint will > not be destroyed. We can't destroy it on rejection as that kind of defeats > the purpose of doing it as a task it in the first place. The EndPoint > will eventually timeout, but that may just queue another destroy event > which could be rejected, or maybe even the timeout task itself is being > rejected. > > In an asynchronous server, if you randomly reject tasks, then some async > lifecycles will never complete and a leak will result. I highly recommend > having an infinite job queue and using other means to restrict the size > (limite connections or requests rather than tasks). At the very least, > have a limited job queue that blocks when full rather than rejects. > > However, please raise an issue on github for this, as we should at least > look at a reproduction and see if there is anything more we can do with > that rejected execution to be more robust. > > regards > > > > > > > > On 2 August 2017 at 19:48, Julien Moumne <[email protected]> wrote: > > > Hello, > > > > We are experiencing a memory leak in Jetty 9.4.6. > > > > I am seeking for help in determining if we are not using Jetty properly > or > > if there is indeed a limitation in Jetty. > > > > This behavior is observed under high load in a simulated environment > using > > gatling and bench-rest, where QueuedTheadPool is saturated and rejects > > jobs > > <https://github.com/eclipse/jetty.project/blob/jetty-9.4. > 6.v20170531/jetty-util/src/main/java/org/eclipse/jetty/ > util/thread/QueuedThreadPool.java#L384> > > . > > > > We are getting the following WARN log from QueuedThreadPool : > > > > org.eclipse.jetty.util.thread.QueuedThreadPool: > > dw{STARTED,4<=4<=4,i=0,q=4} rejected org.eclipse.jetty.io. > > *ManagedSelector$$Lambda*$72/1495355211@413f1bdf > > > > The Lambda is defined in ManagedSelector.destroyEndPoint > > <https://github.com/eclipse/jetty.project/blob/jetty-9.4. > 6.v20170531/jetty-io/src/main/java/org/eclipse/jetty/io/ > ManagedSelector.java#L431> > > > > > > Since the Lambda is not called, AbstractConnector.onEndPointClosed > > <https://github.com/eclipse/jetty.project/blob/jetty-9.4. > 6.v20170531/jetty-server/src/main/java/org/eclipse/jetty/ > server/AbstractConnector.java#L676> > > is never called. > > > > Therefore, EndPoint objects remain in AbstractConnector._endpoints > > <https://github.com/eclipse/jetty.project/blob/jetty-9.4. > 6.v20170531/jetty-server/src/main/java/org/eclipse/jetty/ > server/AbstractConnector.java#L147> > > and lead to out of memory errors. > > > > A screenshot of a memory dump showing 624159 endpoint objects : > > https://imagebin.ca/v/3VTm2n97Ntki > > A screenshot of how we validated the behavior using a "logging > breakpoint" > > in Intellij : https://imagebin.ca/v/3VTnTSgJDnXx > > > > Please tell me if you need any additional information > > > > Julien > > > > _______________________________________________ > > jetty-users mailing list > > [email protected] > > To change your delivery options, retrieve your password, or unsubscribe > > from this list, visit > > https://dev.eclipse.org/mailman/listinfo/jetty-users > > > > > > -- > Greg Wilkins <[email protected]> CTO http://webtide.com > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: <https://dev.eclipse.org/mailman/private/jetty-users/ > attachments/20170802/3dbea6b5/attachment.html> > > ------------------------------ > > _______________________________________________ > jetty-users mailing list > [email protected] > To change your delivery options, retrieve your password, or unsubscribe > from this list, visit > https://dev.eclipse.org/mailman/listinfo/jetty-users > > End of jetty-users Digest, Vol 99, Issue 2 > ****************************************** > -- Julien MOUMNÉ Software Engineering - Data Science Mail: [email protected] 12 rue d'Athènes 75009 Paris - France
_______________________________________________ jetty-users mailing list [email protected] To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/jetty-users
