What we're looking for is a way to inject *without* using curl, or wget, or any other http-based communication. We'd like for the HTTP daemon to only handle search requests, not indexing requests on top of them.
Plus, I have to believe there's a faster way to get documents into solr/lucene than using curl.... _________________________________________________________________ david whalen senior applications developer eNR Services, Inc. [EMAIL PROTECTED] 203-849-7240 > -----Original Message----- > From: Clay Webster [mailto:[EMAIL PROTECTED] > Sent: Thursday, August 09, 2007 11:43 AM > To: solr-user@lucene.apache.org > Subject: Re: Any clever ideas to inject into solr? Without http? > > Condensing the loader into a single executable sounds right > if you have performance problems. ;-) > > You could also try adding multiple <doc>s in a single post if > you notice your problems are with tcp setup time, though if > you're doing localhost connections that should be minimal. > > If you're already local to the solr server, you might check > out the CSV slurper. http://wiki.apache.org/solr/UpdateCSV > It's a little specialized. > > And then there's of course the question of "are you doing > full re-indexing or incremental indexing of changes?" > > --cw > > > On 8/9/07, Kevin Holmes <[EMAIL PROTECTED]> wrote: > > > > I inherited an existing (working) solr indexing script that > runs like > > this: > > > > > > > > Python script queries the mysql DB then calls bash script > > > > Bash script performs a curl POST submit to solr > > > > > > > > We're injecting about 1000 records / minute (constantly), > frequently > > pushing the edge of our CPU / RAM limitations. > > > > > > > > I'm in the process of building a Perl script to use DBI and > > lwp::simple::post that will perform this all from a single script > > (instead of 3). > > > > > > > > Two specific questions > > > > 1: Does anyone have a clever (or better) way to perform > this process > > efficiently? > > > > > > > > 2: Is there a way to inject into solr without using POST / > curl / http? > > > > > > > > Admittedly, I'm no solr expert - I'm starting from someone else's > > setup, trying to reverse-engineer my way out. Any input would be > > greatly appreciated. > > > > >
