Thank you guys for your replies, Sorry that I forgot to mention that I have allocated 10 GB of memory to the Java Heap.
2013/12/26 Shawn Heisey <s...@elyograg.org> > On 12/26/2013 3:38 AM, Jilal Oussama wrote: > > Solr was hosted on an Amazon ec2 m1.large (2 vCPU with 4 ECU, 7.5 GB > memory > > & 840 GB storage) and contained several cores for different usage. > > > > When I manually executed a query through Solr Admin (a query containing > > 10~15 terms, with some of them having boosts over one field and limited > to > > one result without any sorting or faceting etc ....) it takes around 700 > > ms, and the Core contained 7 million documents. > > > > When the scripts are executed things get slower, my query takes 7~10s. > > > > Then what I did is to turn to SolrCloud expecting huge performance > increase. > > > > I installed it on a cluster of 5 Amazon ec2 c3.2xlarge instances (8 vCPU > > with 28 ECU, 15 GB memory & 160 SSD storage), then I created one > collection > > to contain the core I was querying, I sharded it to 25 shards (each node > > containing 5 shards without replication), each shards took 54 MB of > storage. > > > > Tested my query on the new SolrCloud, it takes 70 ms ! huge increase wich > > is very good ! > > > > Tested my scripts again (I have 30 scripts running at the same time), and > > as a surprise, things run fast for 5 seconds then it turns realy slow > again > > (query time ). > > > > I updated the solrconfig.xml to remove the query caches (I don't need > them > > since queries are very different and only 1 time queries) and changes the > > index memory to 1 GB, but only got a small increase (3~4s for each query > ?!) > > Your SolrCloud setup has 35 times as much CPU power (just basing this on > the ECU numbers) as your single-server setup, ten times as much memory, > and a lot more IOPS because you moved to SSD. A 10X increase in single > query performance is not surprising. > > You have not indicated how much memory is assigned to the java heap on > each server. I think that there are three possible problems happening > here, with a strong possibility that the third one is happening at the > same time as one of the other two: > > 1) Full garbage collections are too frequent because the heap is too small. > 2) Garbage collections take too long because the heap is very large and > GC is not tuned. > 3) Extremely high disk I/O because the OS disk cache is too small for > the index size. > > Some information on these that might be helpful: > > http://wiki.apache.org/solr/SolrPerformanceProblems > > The general solution for good Solr performance is to throw hardware, > especially memory, at the problem. It's worth pointing out that any > level of hardware investment has an upper limit on the total query > volume it can support. Running 30 test scripts at the same time will be > difficult for all but the most powerful and expensive hardware to deal > with, especially if every query is different. A five-server cloud where > each server has 8 CPU cores and 15GB of memory is pretty small, all > things considered. > > Thanks, > Shawn > >
