On Thursday 10 September 2009 01:47:38 pm Walter Underwood wrote:
> What kind of storage is used for the Solr index files? When I tested it, NFS
> was 100X slower than local disk.
I'm sorry - I misunderstood your question. The Solr indexes themselves are
stored on local disk. The documents are retrievable (for DIH) from NFS.
And, I started looking closer into this problem... both the box doing the
posts, and the solr box are around 90% idle while the indexing process is
running. And there is no I/O wait time.
I'm now looking into possible network slowness...
-Dan
>
> wunder
>
> -----Original Message-----
> From: Dan A. Dickey [mailto:[email protected]]
> Sent: Thursday, September 10, 2009 11:15 AM
> To: [email protected]
> Cc: Walter Underwood
> Subject: Re: Solr http post performance seems slow - help?
>
> On Thursday 10 September 2009 09:10:27 am Walter Underwood wrote:
> > How big are your documents?
>
> For the most part, I'm just indexing metadata that has been pulled from
> the documents. I think I have currently about 40 or so fields that I'm
> setting.
> When the document is an actual document - pdf, doc, etc... I use the DIH
> to extract stuff and also set the metadata then.
>
> > Is your index on local disk or network-
> > mounted disk?
>
> I'm basically pulling the metadata info from a database and the documents
> themselves are shared via NFS to the Solr indexer.
> -Dan
>
> >
> > wunder
> >
> > On Sep 10, 2009, at 6:39 AM, Yonik Seeley wrote:
> >
> > > On Thu, Sep 10, 2009 at 9:13 AM, Dan A. Dickey
> > > <[email protected]> wrote:
> > >> I'm posting documents to Solr using http (curl) from
> > >> C++/C code and am seeing approximately 3.3 - 3.4
> > >> documents per second being posted. Is this to be expected?
> > >
> > > No, that's very slow.
> > > Are you using libcurl, or actually forking a new process for every
> > > document?
> > > Are you committing on every document?
> > >
> > > If you can, using Java would make your life much easier since you
> > > could use the SolrJ client and it's binary protocol for indexing.
> > >
> > > -Yonik
> > > http://www.lucidimagination.com
> > >
> >
> >
>
>
--
Dan A. Dickey | Senior Software Engineer
Savvis
10900 Hampshire Ave. S., Bloomington, MN 55438
Office: 952.852.4803 | Fax: 952.852.4951
E-mail: [email protected]
