On Fri, Jan 14, 2011 at 1:56 PM, supersoft wrote:
>
> The tests are performed with a selfmade program.
[...]
May I ask in what language is the program written in? The reason to
ask that is to eliminate the possibility that there is an issue with the
threading model, e.g., if you were using Python
On Thu, 2011-01-13 at 17:40 +0100, supersoft wrote:
> Although most of the queries are cache hits, the performance is still
> dependent of the number of simultaneous queries:
>
> 1 simultaneous query: 3437 ms (cache fails)
Average response time: 3437 ms
Throughput: 0.29 queries/sec
> 2 simultane
message in context:
http://lucene.472066.n3.nabble.com/Improving-Solr-performance-tp2210843p2254121.html
Sent from the Solr - User mailing list archive at Nabble.com.
On Thu, Jan 13, 2011 at 10:10 PM, supersoft wrote:
>
> On the one hand, I found really interesting those comments about the reasons
> for sharding. Documentation agrees you about why to split an index in
> several shards (big sizes problems) but I don't find any explanation about
> the inconvenien
ks in advance (and also thanks for previous comments)
--
View this message in context:
http://lucene.472066.n3.nabble.com/Improving-Solr-performance-tp2210843p2249108.html
Sent from the Solr - User mailing list archive at Nabble.com.
Any sources to cite for this statement? And are you talking about RAM
allocated to the JVM or available for OS cache?
> Not sure if this was mentioned yet, but if you are doing slave/master
> replication you'll need 2x the RAM at replication time. Just something to
> keep in mind.
>
> -mike
>
>
And I don't think I've seen anyone suggest a seperate core just for
Access Control Lists. I'm not sure what that would get you.
Perhaps a separate store that isn't Solr at all, in some cases.
On 1/10/2011 5:36 PM, Jonathan Rochkind wrote:
Access Control Lists
On 1/10/2011 5:03 PM, Dennis Gearon wrote:
What I seem to see suggested here is to use different cores for the things you
suggested:
different types of documents
Access Control Lists
I wonder how sharding would work in that scenario?
Sharding has nothing to do with that scenario at all.
Not sure if this was mentioned yet, but if you are doing slave/master
replication you'll need 2x the RAM at replication time. Just something to
keep in mind.
-mike
On Mon, Jan 10, 2011 at 5:01 PM, Toke Eskildsen wrote:
> On Mon, 2011-01-10 at 21:43 +0100, Paul wrote:
> > > I see from your other
PM
Subject: Re: Improving Solr performance
I see a lot of people using shards to hold "different types of documents", and
it almost always seems to be a bad solution. Shards are intended for
distributing a large index over multiple hosts -- that's it. Not for some kind
of federated sea
On Mon, 2011-01-10 at 21:43 +0100, Paul wrote:
> > I see from your other messages that these indexes all live on the same
> > machine.
> > You're almost certainly I/O bound, because you don't have enough memory for
> > the
> > OS to cache your index files. With 100GB of total index size, you'll
I see a lot of people using shards to hold "different types of
documents", and it almost always seems to be a bad solution. Shards are
intended for distributing a large index over multiple hosts -- that's
it. Not for some kind of federated search over multiple schemas, not
for access control.
No, it also depends on the queries you execute (sorting is a big consumer) and
the number of concurrent users.
> Is that a general rule of thumb? That it is best to have about the
> same amount of RAM as the size of your index?
>
> So, with a 5GB index, I should have between 4GB and 8GB of RAM
>
> I see from your other messages that these indexes all live on the same
> machine.
> You're almost certainly I/O bound, because you don't have enough memory for
> the
> OS to cache your index files. With 100GB of total index size, you'll get best
> results with between 64GB and 128GB of total R
ene.apache.org
Sent: Sun, January 9, 2011 4:34:08 PM
Subject: Re: Improving Solr performance
On 1/7/2011 2:57 AM, supersoft wrote:
> have deployed a 5-sharded infrastructure where: shard1 has 3124422 docs
> shard2 has 920414 docs shard3 has 602772 docs shard4 has 2083492 docs shard5
> has
On 1/7/2011 2:57 AM, supersoft wrote:
have deployed a 5-sharded infrastructure where: shard1 has 3124422 docs
shard2 has 920414 docs shard3 has 602772 docs shard4 has 2083492 docs shard5
has 11915639 docs Indexes total size: 100GB
The OS is Linux x86_64 (Fedora release 8) with vMem equal to 7872
Are you using the Solr caches? These are configured in solrconfig.xml
in each core. Make sure you have at least 50-100 configured for each
kind.
Also, use filter queries: a filter query describes a subset of
documents. When you do a bunch of queries against the same filter
query, the second and su
Making sure the index can fit in memory (you don't have to allocate that
much to Solr, just make sure it's available to the OS so it can cache it --
otherwise you are paging the hard drive, which is why you are probably IO
bound) has been the key to our performance. We recently opted to use less
RA
On Fri, 2011-01-07 at 10:57 +0100, supersoft wrote:
[5 shards, 100GB, ~20M documents]
...
[Low performance for concurrent searches]
> Using JConsole for monitoring the server java proccess I checked that Heap
> Memory and the CPU Usages don't reach the upper limits so the server
> shouldn't per
f
> simultaneous queries?
>
> Thanks in advance
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Improving-Solr-performance-tp2210843p2210843.html
> Sent from the Solr - User mailing list archive at Nabble.com.
.nabble.com/Improving-Solr-performance-tp2210843p2211305.html
Sent from the Solr - User mailing list archive at Nabble.com.
open a new mail conversation for that
-
Grijesh
--
View this message in context:
http://lucene.472066.n3.nabble.com/Improving-Solr-performance-tp2210843p2211300.html
Sent from the Solr - User mailing list archive at Nabble.com.
, Nb_indexed_fields_in_index,
...) ?
Regards,
---
Hong-Thai
-Message d'origine-
De : Grijesh.singh [mailto:pintu.grij...@gmail.com]
Envoyé : vendredi 7 janvier 2011 12:29
À : solr-user@lucene.apache.org
Objet : Re: Improving Solr performance
shards are used when index size become hug
response from all shards and incorporate all
responses in a single result and returns.
So if any of shards taking more time to response then your total response
time will affect
-
Grijesh
--
View this message in context:
http://lucene.472066.n3.nabble.com/Improving-Solr-performance
response from all shards and incorporate all
responses in a single result and returns.
So if any of shards taking more time to response then your total response
time will affect
-
Grijesh
--
View this message in context:
http://lucene.472066.n3.nabble.com/Improving-Solr-performance
1 - Yes, all the shards are in the same machine
2 - The machine RAM is 7.8GB and I assign 3.4GB to Solr server
3 - The shards sizes (GB) are 17, 5, 3, 11, 64
--
View this message in context:
http://lucene.472066.n3.nabble.com/Improving-Solr-performance-tp2210843p2211135.html
Sent from the Solr
Some questions-
1-Are all shards on same machine
2-What is your Ram Size
3-What are the size of index on each shards in GB
-
Grijesh
--
View this message in context:
http://lucene.472066.n3.nabble.com/Improving-Solr-performance-tp2210843p2210878.html
Sent from the Solr - User mailing list
nyone give me an approach of how I
should tune the instance for not being so hardly dependent of the number of
simultaneous queries?
Thanks in advance
--
View this message in context:
http://lucene.472066.n3.nabble.com/Improving-Solr-performance-tp2210842p2210842.html
Sent from the Solr - User ma
nyone give me an approach of how I
should tune the instance for not being so hardly dependent of the number of
simultaneous queries?
Thanks in advance
--
View this message in context:
http://lucene.472066.n3.nabble.com/Improving-Solr-performance-tp2210843p2210843.html
Sent from the Solr - User ma
29 matches
Mail list logo
