Re: How to index large set data

S >> is 1g of memory. I did several experiments, every time RES reaches 1g, the >> indexing process becomes extremely slow. Is this memory limit set by JVM? >> And how can I set the JVM memory when I use DIH through web command >> full-import? >> >> Thanks! >>

Re: How to index large set data

it set by JVM? > And how can I set the JVM memory when I use DIH through web command > full-import? > > Thanks! > > > JB > > > > > --- On Fri, 5/22/09, Noble Paul നോബിള്‍ नोब्ळ् > wrote: > > > From: Noble Paul നോബിള്‍ नोब्ळ् > > Subject: Re: How t

Re: How to index large set data

he indexing process becomes extremely slow. Is this memory limit set by JVM? And how can I set the JVM memory when I use DIH through web command full-import? Thanks! JB --- On Fri, 5/22/09, Noble Paul നോബിള്‍ नोब्ळ् wrote: > From: Noble Paul നോബിള്‍ नोब्ळ् > Subject: Re: How to

Re: How to index large set data

> > > > > --- On Fri, 5/22/09, Grant Ingersoll > wrote: > > > >> From: Grant Ingersoll > >> Subject: Re: How to index large set data > >> To: solr-user@lucene.apache.org > >> Date: Friday, May 22, 2009, 5:38 AM > >> Can you par

Re: How to index large set data

ri, 5/22/09, Grant Ingersoll wrote: > >> From: Grant Ingersoll >> Subject: Re: How to index large set data >> To: solr-user@lucene.apache.org >> Date: Friday, May 22, 2009, 5:38 AM >> Can you parallelize this?  I >> don't know that the DIH can handle it, >

Re: How to index large set data

fortuna! Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: Jianbin Dai > To: solr-user@lucene.apache.org > Sent: Friday, May 22, 2009 11:05:27 AM > Subject: Re: How to index large set data > > > I dont know exactly

Re: How to index large set data

If I do the xml parsing by myself and use embedded client to do the push, would it be more efficient than DIH? --- On Fri, 5/22/09, Grant Ingersoll wrote: > From: Grant Ingersoll > Subject: Re: How to index large set data > To: solr-user@lucene.apache.org > Date: Friday, May 22

Re: How to index large set data

I dont know exactly what is this 3G Ram buffer used. But what I noticed was both index size and file number were keeping increasing, but stuck in the commit. --- On Fri, 5/22/09, Otis Gospodnetic wrote: > From: Otis Gospodnetic > Subject: Re: How to index large set data > To:

Re: How to index large set data

in Dai > To: solr-user@lucene.apache.org; noble.p...@gmail.com > Sent: Friday, May 22, 2009 3:42:04 AM > Subject: Re: How to index large set data > > > about 2.8 m total docs were created. only the first run finishes. In my 2nd > try, > it hangs there forever at th

Re: How to index large set data

my solrconfig.xml >> >>   false >>   3000 >>   1000 >>   2147483647 >>   1 >>   false >> >> >> >> >> --- On Thu, 5/21/09, Noble Paul നോബിള്‍  नोब्ळ् >> wrote: >> >>> From: Noble Paul നോബിള്‍  नोब्ळ्

Re: How to index large set data

e and failed? 2. how can i speed up the indexing? Here is my solrconfig.xml false 3000 1000 2147483647 1 false --- On Thu, 5/21/09, Noble Paul നോബിള്‍ नो ब्ळ् wrote: From: Noble Paul നോബിള്‍ नोब्ळ् Subject: Re: How to index large set data To: solr-user@lucene.apach

Re: How to index large set data

i speed up the indexing? Here is my solrconfig.xml false 3000 1000 2147483647 1 false --- On Thu, 5/21/09, Noble Paul നോബിള്‍ नोब्ळ् wrote: > From: Noble Paul നോബിള്‍ नोब्ळ् > Subject: Re: How to index large set data > To: solr-user@lucene.apache.o

Re: How to index large set data

നോബിള്‍  नोब्ळ् > wrote: > >> From: Noble Paul നോബിള്‍  नोब्ळ् >> Subject: Re: How to index large set data >> To: solr-user@lucene.apache.org >> Date: Thursday, May 21, 2009, 9:57 PM >> check the status page of DIH and see >> if it is working properly. and &

Re: How to index large set data

: > From: Noble Paul നോബിള്‍ नोब्ळ् > Subject: Re: How to index large set data > To: solr-user@lucene.apache.org > Date: Thursday, May 21, 2009, 9:57 PM > check the status page of DIH and see > if it is working properly. and > if, yes what is the rate of indexing > > On Th

Re: How to index large set data

check the status page of DIH and see if it is working properly. and if, yes what is the rate of indexing On Thu, May 21, 2009 at 11:48 AM, Jianbin Dai wrote: > > Hi, > > I have about 45GB xml files to be indexed. I am using DataImportHandler. I > started the full import 4 hours ago, and it's sti

Re: How to index large set data

This isn't much data to go on. Do you have any idea what your throughput is?How many documents are you indexing? one 45G doc or 4.5 billion 10 character docs? Have you looked at any profiling data to see how much memory is being consumed? Are you IO bound or CPU bound? Best Erick On Thu, May 21,