I don't recall anyone being able to get acceptable performance with a single index that large with solr/lucene. The conventional wisdom is that parallel searching across cores (or shards in SolrCloud) is the best way to handle index sizes in the "illions". So its of great interest how you did.
Anyone else gotten an index(es) with billions of documents to perform well? I'm greatly interested in how. On Mon, 2012-05-28 at 05:12 -0700, Nagendra Nagarajayya wrote: > It is a single node. I am trying to find out if the performance can be > referenced. > > Regarding information on Solr with RankingAlgorithm, you can find all > the information here: > > http://solr-ra.tgels.org > > On RankingAlgorithm: > > http://rankingalgorithm.tgels.org > > Regards, > - NN > > On 5/27/2012 4:50 PM, Li Li wrote: > > yes, I am also interested in good performance with 2 billion docs. how > > many search nodes do you use? what's the average response time and qps > > ? > > > > another question: where can I find related paper or resources of your > > algorithm which explains the algorithm in detail? why it's better than > > google site(better than lucene is not very interested because lucene > > is not originally designed to provide search function like google)? > > > > On Mon, May 28, 2012 at 1:06 AM, Darren Govoni<dar...@ontrenet.com> wrote: > >> I think people on this list would be more interested in your approach to > >> scaling 2 billion documents than modifying solr/lucene scoring (which is > >> already top notch). So given that, can you share any references or > >> otherwise substantiate good performance with 2 billion documents? > >> > >> Thanks. > >> > >> On Sun, 2012-05-27 at 08:29 -0700, Nagendra Nagarajayya wrote: > >>> Actually, RankingAlgorithm 1.4.2 has been scaled to more than 2 billion > >>> docs. With RankingAlgorithm 1.4.3, using the parameters > >>> age=latest&docs=number feature, you can retrieve the NRT inserted > >>> documents in milliseconds from such a huge index improving query and > >>> faceting performance and using very little resources ... > >>> > >>> Currently, RankingAlgorithm 1.4.3 is only available with Solr 4.0, and > >>> the NRT insert performance with Solr 4.0 is about 70,000 docs / sec. > >>> RankingAlgorithm 1.4.3 should become available with Solr 3.6 soon. > >>> > >>> Regards, > >>> > >>> Nagendra Nagarajayya > >>> http://solr-ra.tgels.org > >>> http://rankingalgorithm.tgels.org > >>> > >>> > >>> > >>> On 5/27/2012 7:32 AM, Darren Govoni wrote: > >>>> Hi, > >>>> Have you tested this with a billion documents? > >>>> > >>>> Darren > >>>> > >>>> On Sun, 2012-05-27 at 07:24 -0700, Nagendra Nagarajayya wrote: > >>>>> Hi! > >>>>> > >>>>> I am very excited to announce the availability of Solr 3.6 with > >>>>> RankingAlgorithm 1.4.2. > >>>>> > >>>>> This NRT supports now works with both RankingAlgorithm and Lucene. The > >>>>> insert/update performance should be about 5000 docs in about 490 ms with > >>>>> the MbArtists Index. > >>>>> > >>>>> RankingAlgorithm 1.4.2 has multiple algorithms, improved performance > >>>>> over the earlier releases, supports the entire Lucene Query Syntax, ± > >>>>> and/or boolean queries and can scale to more than a billion documents. > >>>>> > >>>>> You can get more information about NRT performance from here: > >>>>> http://solr-ra.tgels.org/wiki/en/Near_Real_Time_Search_ver_3.x > >>>>> > >>>>> You can download Solr 3.6 with RankingAlgorithm 1.4.2 from here: > >>>>> http://solr-ra.tgels.org > >>>>> > >>>>> Please download and give the new version a try. > >>>>> > >>>>> Regards, > >>>>> > >>>>> Nagendra Nagarajayya > >>>>> http://solr-ra.tgels.org > >>>>> http://rankingalgorithm.tgels.org > >>>>> > >>>>> ps. MbArtists index is the example index used in the Solr 1.4 Enterprise > >>>>> Book > >>>>> > >>>> > >>>> > >> > > >
