I am not familiar with perl so I cannot help you in how to do it
better in perl.The pseudo code should help.
You can do faster indexing if you post in multiple threads. If you
know java , use StreamingHttpSolrServer (in SolrJ client)
On Fri, Jul 10, 2009 at 4:28 PM, Shalin Shekhar
Mangar wrote:
>
On Fri, Jul 10, 2009 at 1:17 PM, Francis Yakin wrote:
> How you batching all documents in one curl call? Do you have a sample, so I
> can modify my script and try it again.
>
> Right now I do curl on each documents( I have 1000 docs on each folder and
> I have 1000 folders) using :
>
> curl http
ache.org
Subject: Re: Using curl comparing with using WebService::Solr
On Fri, Jul 10, 2009 at 11:50 AM, Francis Yakin wrote:
> I also commit too many I guess, since we have 1000 folders, so each loop
> will executed the load and commit.
> So 1000 loops with 1000 commits. I think it
On Fri, Jul 10, 2009 at 11:50 AM, Francis Yakin wrote:
> I also commit too many I guess, since we have 1000 folders, so each loop
> will executed the load and commit.
> So 1000 loops with 1000 commits. I think it will be help if I only commit
> once after the 1000 loops completed.
>
> Any inputs?
Yakin [mailto:fya...@liquid.com]
Sent: Thursday, July 09, 2009 11:13 PM
To: 'solr-user@lucene.apache.org'; 'noble.p...@gmail.com'
Subject: RE: Using curl comparing with using WebService::Solr
Yes, the xml files are in complete add format.
This is my code:
#!/usr/bin/perl
i
From: noble.p...@gmail.com [mailto:noble.p...@gmail.com] On Behalf Of Noble
Paul ??? ??
Sent: Thursday, July 09, 2009 10:40 PM
To: solr-user@lucene.apache.org
Subject: Re: Using curl comparing with using WebService::Solr
are these xml files in the solr add xml format?
When you post using curl,
I guess it would be fater if you run this program from a diffferent
box,.that way the disk IO will not be a bottleneck
2009/7/10 Noble Paul നോബിള് नोब्ळ् :
> are these xml files in the solr add xml format?
>
> When you post using curl, I guess it opens as many http connections as
> there are fil
are these xml files in the solr add xml format?
When you post using curl, I guess it opens as many http connections as
there are files. if you can write a small program to post all these
files in one request, you should be able to get better perf.
the following can be the pseudo-code
open connec
