If it's a contention between search and indexing, separate them via a query-slave and an index-master.
--cw On 8/9/07, David Whalen <[EMAIL PROTECTED]> wrote: > > What we're looking for is a way to inject *without* using > curl, or wget, or any other http-based communication. We'd > like for the HTTP daemon to only handle search requests, not > indexing requests on top of them. > > Plus, I have to believe there's a faster way to get documents > into solr/lucene than using curl.... > > _________________________________________________________________ > david whalen > senior applications developer > eNR Services, Inc. > [EMAIL PROTECTED] > 203-849-7240 > > > > -----Original Message----- > > From: Clay Webster [mailto:[EMAIL PROTECTED] > > Sent: Thursday, August 09, 2007 11:43 AM > > To: solr-user@lucene.apache.org > > Subject: Re: Any clever ideas to inject into solr? Without http? > > > > Condensing the loader into a single executable sounds right > > if you have performance problems. ;-) > > > > You could also try adding multiple <doc>s in a single post if > > you notice your problems are with tcp setup time, though if > > you're doing localhost connections that should be minimal. > > > > If you're already local to the solr server, you might check > > out the CSV slurper. http://wiki.apache.org/solr/UpdateCSV > > It's a little specialized. > > > > And then there's of course the question of "are you doing > > full re-indexing or incremental indexing of changes?" > > > > --cw > > > > > > On 8/9/07, Kevin Holmes <[EMAIL PROTECTED]> wrote: > > > > > > I inherited an existing (working) solr indexing script that > > runs like > > > this: > > > > > > > > > > > > Python script queries the mysql DB then calls bash script > > > > > > Bash script performs a curl POST submit to solr > > > > > > > > > > > > We're injecting about 1000 records / minute (constantly), > > frequently > > > pushing the edge of our CPU / RAM limitations. > > > > > > > > > > > > I'm in the process of building a Perl script to use DBI and > > > lwp::simple::post that will perform this all from a single script > > > (instead of 3). > > > > > > > > > > > > Two specific questions > > > > > > 1: Does anyone have a clever (or better) way to perform > > this process > > > efficiently? > > > > > > > > > > > > 2: Is there a way to inject into solr without using POST / > > curl / http? > > > > > > > > > > > > Admittedly, I'm no solr expert - I'm starting from someone else's > > > setup, trying to reverse-engineer my way out. Any input would be > > > greatly appreciated. > > > > > > > > >
