Re: maximum number of shards per SolrCloud

2014-04-22 Thread YouPeng Yang
Hi Shawn Thanks a lot. It is greatly helpful. 2014-04-23 0:43 GMT+08:00 Shawn Heisey : > On 4/22/2014 10:02 AM, yypvsxf19870706 wrote: > >> I am curious of the influences when have more than 2G docs in a >> core.And we plan to have 5g docs/core. >> >> Please give me some suggesti

Re: maximum number of shards per SolrCloud

2014-04-22 Thread Shawn Heisey
On 4/22/2014 10:02 AM, yypvsxf19870706 wrote: I am curious of the influences when have more than 2G docs in a core.And we plan to have 5g docs/core. Please give me some suggestion about how to plan num of docs in a core ? One Solr core contains one Lucene index. It can't be divided

Re: maximum number of shards per SolrCloud

2014-04-22 Thread yypvsxf19870706
Hi I am curious of the influences when have more than 2G docs in a core.And we plan to have 5g docs/core. Please give me some suggestion about how to plan num of docs in a core ? Thanks. 发自我的 iPhone 在 2014-4-22,12:30,Erick Erickson 写道: > You're going to run into resource issues

Re: maximum number of shards per SolrCloud

2014-04-22 Thread Zhifeng Wang
Thanks, Erick. I wonder whether having multiple SolrClouds helps in this matter. If I have 5 SolrClouds, that will effectively boost the number of shards 5 times. Of course this introduces issues when I need to aggregate results from different SolrClouds. Thanks, Zhifeng On Tue, Apr 22, 2014 at

Re: maximum number of shards per SolrCloud

2014-04-21 Thread Erick Erickson
You're going to run into resource issues long before you hit 2G docs/node I suspect. I've seen from 50M t0 300M docs on a single node. Fortunately you may well be near the upper end of that since you're dealing with log files. Bottom line here is that you're off into largely uncharted territory wh

maximum number of shards per SolrCloud

2014-04-21 Thread Zhifeng Wang
Hi, We are facing a high incoming rate of usually small documents (logs). The incoming rate is initially assumed at 2K/sec but could reach as high as 20K/sec. So a year's worth of data could reach 60G (assuming the rate at 2K/sec) searchable documents. Since a single shard can contain no more tha