Hi Otis,
The slowness was due to the JVM memory limit set by tomcat.. I have solved this problem. Initially I thought there might be memory leaking because I noticed the following behavior: In the peak of indexing, almost all 4GB of memory was used. Once indexing is done, the memory usage was about 3GB. If I delete all indexing, and shutdown solr, I still noticed that about 2 GB memory used, much more than the initial memory usage about 250M. I am not sure if I guess it right. Thanks. --- On Tue, 5/26/09, Otis Gospodnetic <otis_gospodne...@yahoo.com> wrote: > From: Otis Gospodnetic <otis_gospodne...@yahoo.com> > Subject: Re: Is it memory leaking in solr? > To: solr-user@lucene.apache.org > Date: Tuesday, May 26, 2009, 10:03 AM > > Jianbin, > > If you connect to that Java process with jconsole, do you > see a lot of garbage collection activity? > > What makes you think there is a memory leak? The > slowness? > > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > ----- Original Message ---- > > From: Jianbin Dai <djian...@yahoo.com> > > To: solr-user@lucene.apache.org > > Sent: Monday, May 25, 2009 1:05:43 PM > > Subject: Re: Is it memory leaking in solr? > > > > > > Again, indexing becomes extremely slow after indexed > 8m documents (about 25G of > > original file size). Here is the memory usage info of > my computer. Does this > > have anything to do with tomcat setting? Thanks. > > > > > > top - 08:09:53 up 7:22, 1 user, load > average: 1.03, 1.01, 1.00 > > Tasks: 78 total, 2 > running, 76 sleeping, 0 > stopped, 0 zombie > > Cpu(s): 49.9%us, 0.2%sy, 0.0%ni, > 49.8%id, 0.2%wa, 0.0%hi, 0.0%si, > 0.0%st > > Mem: 4044776k total, 3960740k > used, 84036k free, 42196k buffers > > Swap: 2031608k total, > 84k used, 2031524k free, > 2729892k cached > > > > PID USER PR > NI VIRT RES SHR S %CPU %MEM > TIME+ COMMAND > > > > > > 3322 root 21 0 > 1357m 1.0g 11m S 100 27.0 397:51.74 java > > > > > > > > --- On Mon, 5/25/09, Jianbin Dai wrote: > > > > > From: Jianbin Dai > > > Subject: Is it memory leaking in solr? > > > To: solr-user@lucene.apache.org, > noble.p...@gmail.com > > > Date: Monday, May 25, 2009, 1:27 AM > > > > > > I am using DIH to do indexing. After I indexed > about 8M > > > documents (took about 1hr40m), it used up almost > all memory > > > (4GB), and the indexing becomes extremely slow. > If I delete > > > all indexing and shutdown tomcat, it still shows > over 3gb > > > memory was used. Is it memory leaking? if it is, > then the > > > leaking is in solr indexing or DIH? > Thanks. > > > > > > > > > > > > > > > > >
