On 8/4/2011 12:38 AM, Bernd Fehling wrote:
Hi Shawn,
the 0.05 seconds for search time at peek times (3 qps) is my target
for Solr.
The numbers for solr are from Solr's statistic report page. So 39.5
seconds
average per request is definately to long and I have to change to
sharding.
Solr rep
solr-user@lucene.apache.org
Subject: Re: performance crossover between single index and sharding
Hi Shawn,
the 0.05 seconds for search time at peek times (3 qps) is my target for
Solr.
The numbers for solr are from Solr's statistic report page. So 39.5
seconds
average per request is definately to
> Sent: Thursday, August 04, 2011 2:39 AM
> To: solr-user@lucene.apache.org
> Subject: Re: performance crossover between single index and sharding
>
> Hi Shawn,
>
> the 0.05 seconds for search time at peek times (3 qps) is my target for
> Solr.
> The numbers for solr are
age-
> From: Markus Jelsma [mailto:markus.jel...@openindex.io]
> Sent: Tuesday, August 02, 2011 2:12 PM
> To: solr-user@lucene.apache.org
> Subject: Re: performance crossover between single index and sharding
>
> Hi Tom,
>
> Very interesting indeed! But i keep wondering wh
Hi Shawn,
the 0.05 seconds for search time at peek times (3 qps) is my target for Solr.
The numbers for solr are from Solr's statistic report page. So 39.5 seconds
average per request is definately to long and I have to change to sharding.
For FAST system the numbers for the search dispatcher ar
Replies inline.
On 8/3/2011 2:24 AM, Bernd Fehling wrote:
To show that I compare apples and oranges here are my previous FAST
Search setup:
- one master server (controlling, logging, search dispatcher)
- six index server (4.25 mio docs per server, 5 slices per index)
(searching and indexing a
OK, here is a brief on our sharded setup.
We have 10 shards, 3 per high-end Amazon machine. Majority of the searches
are done on 2 shards at most, that have the latest data in their indices. We
use logical sharding, not hash based. These two lead to a situation, where
given a user query that *will
On 02.08.2011 21:00, Shawn Heisey wrote:
...
I did try some early tests with a single large index. Performance was pretty
decent once it got warmed up, but I was worried about how it would
perform under a heavy load, and how it would cope with frequent updates. I
never really got very far with
On 8/2/2011 12:06 PM, Jonathan Rochkind wrote:
What's the reasoning behind having three shards on one machine,
instead of just combining those into one shard? Just curious. I had
been thinking the point of shards was to get them on different
machines, and there'd be no reason to have multiple
ust.org/blogs/large-scale-search/scaling-large-scale-search-500000-volumes-5-million-volumes-and-beyond
>>
>> -Original Message-
>> From: Markus Jelsma [mailto:markus.jel...@openindex.io]
>> Sent: Tuesday, August 02, 2011 12:33 PM
>> To: solr-user@lucene.apache.org
&g
justify buying 8 more machines:)
Tom
-Original Message-
From: Markus Jelsma [mailto:markus.jel...@openindex.io]
Sent: Tuesday, August 02, 2011 2:12 PM
To: solr-user@lucene.apache.org
Subject: Re: performance crossover between single index and sharding
Hi Tom,
Very interesting indeed! But
-search
>
> *
> http://www.hathitrust.org/blogs/large-scale-search/scaling-large-scale-sea
> rch-50-volumes-5-million-volumes-and-beyond
>
> -Original Message-
> From: Markus Jelsma [mailto:markus.jel...@openindex.io]
> Sent: Tuesday, August 02, 2011 12:33 PM
&
ale-search-50-volumes-5-million-volumes-and-beyond
-Original Message-
From: Markus Jelsma [mailto:markus.jel...@openindex.io]
Sent: Tuesday, August 02, 2011 12:33 PM
To: solr-user@lucene.apache.org
Subject: Re: performance crossover between single index and sharding
Actually, i do worr
-million-volumes-and-beyond
-Original Message-
From: Markus Jelsma [mailto:markus.jel...@openindex.io]
Sent: Tuesday, August 02, 2011 12:33 PM
To: solr-user@lucene.apache.org
Subject: Re: performance crossover between single index and sharding
Actually, i do worry about it. Would be
Actually, i do worry about it. Would be marvelous if someone could provide
some metrics for an index of many terabytes.
> [..] At some extreme point there will be diminishing
> returns and a performance decrease, but I wouldn't worry about that at all
> until you've got many terabytes -- I don't
That's a fantastic answer, Shawn.
To more directly answer Bernd's question: Bernard, shard your data once
you've done reasonable performance optimizations to your single core index
setup (see Chapter 9 of my book) and the query response time isn't meeting
your requirements in spite of this. Solr
On 8/2/2011 4:44 AM, Bernd Fehling wrote:
Is there any knowledge on this list about the performance
crossover between a single index and sharding and
when to change from a single index to sharding?
E.g. if index size is larger than 150GB and num of docs is
more than 25 mio. then it is better to
17 matches
Mail list logo
