Thanks Walter. Yes, I saw your answer and fixed the issue per your suggestion.
The JavaDoc need to make this clear. The fact there is a close() on this class and the JavaDoc does not say "your program should have exactly as many HttpSolrClient objects as there are servers it talks to" is a prime candidate for missuses of the class. Steve On Sun, Jan 31, 2016 at 5:20 PM, Walter Underwood <wun...@wunderwood.org> wrote: > I already answered this. > > Move the creation of the HttpSolrClient outside the loop. Your code will > run much fast, because it will be able to reuse the connections. > > Put another way, your program should have exactly as many HttpSolrClient > objects as there are servers it talks to. If there is one Solr server, you > have one object. > > There is no leak in HttpSolrClient, you are misusing the class, massively. > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > > > On Jan 31, 2016, at 2:10 PM, Steven White <swhite4...@gmail.com> wrote: > > > > Thank you all for your feedback. > > > > This is code that I inherited and the example i gave is intended to > > demonstrate the memory leak which based on YourKit is > > on java/util/LinkedHashMap$Entry. In short, I'm getting core dumps with > > "Detail "java/lang/OutOfMemoryError" "Java heap space" received " > > > > Here is a more detailed layout of the code. This is a crawler that runs > > 24x7 without any recycle logic in place: > > > > init_data() > > > > while (true) > > { > > HttpSolrClient client = new HttpSolrClient(" > > http://localhost:8983/solr/core1 <http://192.168.202.129:8983/solr/core1 > >/"); > > <<<< this is real code > > > > see_if_we_have_new_data(); > > > > send_new_data_to_solr(); > > > > client.close(); <<<< this is real code > > > > sleep_for_a_bit(N); <<<< 'N' can be any positive int > > } > > > > By default, our Java program is given 4gb of ram "-Xmx4g" and N is set > for > > 5 min. We had a customer set N to 10 second and we started seeing core > > dumps with OOM. As I started to debug, I narrowed the OOM to > > HttpSolrClient per my original email. > > > > The follow up answers I got suggest that I move the construction of > > HttpSolrClient object outside the while loop which I did (but I also had > to > > move "client.close()" outside the loop) and the leak is gone. > > > > Give this, is this how HttpSolrClient is suppose to be used? If so, > what's > > the point of HttpSolrClient.close()? > > > > Another side question. I noticed HttpSolrClient has a setBaseUrl(). > Now, > > if I call it and give it "http://localhost:8983/solr/core1 > > <http://192.168.202.129:8983/solr/core1>/" (ntoice the "/" at the end) > next > > time I use HttpSolrClient to send Solr data, I get back 404. The fix is > to > > remove the ending "/". This is not how the constructor of HttpSolrClient > > behaves; HttpSolrClient will take the URL with or without "/". > > > > In summary, it would be good if someone can confirm f we have a memory > leak > > in HttpSolrClient if used per my example; if so this is a defect. Also, > > can someone confirm the fix I used for this issue: move the constructor > of > > HttpSolrClient outside the loop and reuse the existing object "client". > > > > Again, thank you all for the quick response it is much appreciated. > > > > Steve > > > > > > > > On Sat, Jan 30, 2016 at 1:24 PM, Erick Erickson <erickerick...@gmail.com > > > > wrote: > > > >> Assuming you're not really using code like above and it's a test > case.... > >> > >> What's your evidence that memory consumption goes up? Are you sure > >> you're not just seeing uncollected garbage? > >> > >> When I attached Java Mission Control to this program it looked pretty > >> scary at first, but the heap allocated after old generation garbage > >> collections leveled out to a steady state. > >> > >> > >> On Sat, Jan 30, 2016 at 9:29 AM, Walter Underwood < > wun...@wunderwood.org> > >> wrote: > >>> Create one HttpSolrClient object for each Solr server you are talking > >> to. Reuse it for all requests to that Solr server. > >>> > >>> It will manage a pool of connections and keep them alive for faster > >> communication. > >>> > >>> I took a look at the JavaDoc and the wiki doc, neither one explains > this > >> well. I don’t think they even point out what is thread safe. > >>> > >>> wunder > >>> Walter Underwood > >>> wun...@wunderwood.org > >>> http://observer.wunderwood.org/ (my blog) > >>> > >>> > >>>> On Jan 30, 2016, at 7:42 AM, Susheel Kumar <susheel2...@gmail.com> > >> wrote: > >>>> > >>>> Hi Steve, > >>>> > >>>> Can you please elaborate what error you are getting and i didn't > >> understand > >>>> your code above, that why initiating Solr client object is in loop. > In > >>>> general creating client instance should be outside the loop and a one > >> time > >>>> activity during the complete execution of program. > >>>> > >>>> Thanks, > >>>> Susheel > >>>> > >>>> On Sat, Jan 30, 2016 at 8:15 AM, Steven White <swhite4...@gmail.com> > >> wrote: > >>>> > >>>>> Hi folks, > >>>>> > >>>>> I'm getting memory leak in my code. I narrowed the code to the > >> following > >>>>> minimal to cause the leak. > >>>>> > >>>>> while (true) { > >>>>> HttpSolrClient client = new HttpSolrClient(" > >>>>> http://192.168.202.129:8983/solr/core1";); > >>>>> client.close(); > >>>>> } > >>>>> > >>>>> Is this a defect or an issue in the way I'm using HttpSolrClient? > >>>>> > >>>>> I'm on Solr 5.2.1 > >>>>> > >>>>> Thanks. > >>>>> > >>>>> Steve > >>>>> > >>> > >> > >
