Thank you for taking the time to do the test.
I have been doing similar tests using the post Tool (SimplePostTool) with
the real data and was able to get to about 10K documents/second.
I am considering using multiple files (one per client) ftp'd into a solr
node and then use a scheduled job to us
Are you suggesting that requests come into a service layer that identifies
which client is on which solrcloud and passes the request to that cloud?
Thank you
On Wed, Aug 19, 2015 at 1:13 PM, Toke Eskildsen
wrote:
> Troy Edwards wrote:
> > My average document size is 400 bytes
> > Number of doc
Toke Eskildsen wrote
> Use more than one cloud. Make them fully independent.
> As I suggested when you asked 4 days ago. That would
> also make it easy to scale: Just measure how much a
> single setup can take and do the math.
The goal is 250K documents/second.
I tried modifying the books.csv-ex
For Indexing 3.5 billion documents, you will not only run into bottleneck
with Solr but also at different places (data acquisition, solr document
object creation, submitting in bulk/batches to Solr).
This will require parallelizing the above operations at each of the above
steps which can get you
Ir you're sitting on HDFS anyway, you could use MapReduceIndexerTool. I'm not
sure that'll hit your rate, it spends some time copying things around.
If you're not on
HDFS, though, it's not an option.
Best,
Erick
On Wed, Aug 19, 2015 at 11:36 AM, Upayavira wrote:
>
>
> On Wed, Aug 19, 2015, at 07
On Wed, Aug 19, 2015, at 07:13 PM, Toke Eskildsen wrote:
> Troy Edwards wrote:
> > My average document size is 400 bytes
> > Number of documents that need to be inserted 25/second
> > (for a total of about 3.6 Billion documents)
>
> > Any ideas/suggestions on how that can be done? (use a cl
Troy Edwards wrote:
> My average document size is 400 bytes
> Number of documents that need to be inserted 25/second
> (for a total of about 3.6 Billion documents)
> Any ideas/suggestions on how that can be done? (use a client
> or uploadcsv or stream or data import handler)
Use more than on
On 8/19/2015 11:09 AM, Troy Edwards wrote:
> I have a requirement where I have to bulk insert a lot of documents in
> SolrCloud.
>
> My average document size is 400 bytes
> Number of documents that need to be inserted 25/second (for a total of
> about 3.6 Billion documents)
>
> Any ideas/sugges
I have been using the solrj client and get speeds of 1000 objects per
second. The size of my object is around 4 kb.
On Aug 19, 2015 12:09 PM, "Troy Edwards" wrote:
> I have a requirement where I have to bulk insert a lot of documents in
> SolrCloud.
>
> My average document size is 400 bytes
> Num
