Thanks Shawn and Scott for your feedback. It is really appreciated. > -----Original Message----- > From: Shawn Heisey [mailto:s...@elyograg.org] > Sent: August-14-13 12:39 PM > To: solr-user@lucene.apache.org > Subject: Re: Huge discrepancy between QTime and ElapsedTime > > On 8/14/2013 9:09 AM, Jean-Sebastien Vachon wrote: > > I am running some benchmarks to tune our Solr 4.3 cloud and noticed > > that while the reported QTime is quite satisfactory (100 ms or so), > > the elapsed time is quite large (around 5 seconds). The collection > > contains 12.8M documents and the index size on disk is about 35 GB.. I > > have only one shard and 4 replicas (we intent to have 5 shards but > > wanted to see how Solr would perform with only one shard so that we > > could benefit from all Solr functions) > > As your other reply from Scott says, you may be dealing with the fact that > Solr must fetch stored field data from the index on disk and decompress it. > Solr 4.1 and later have compressed stored fields. There is no way other than > writing custom Solr code to turn off the compression. If the documents are > very large, the decompression step can be a big performance penalty. > > You have a VERY large field list - fl parameter. Have you tried just leaving > that > parameter off so that Solr will return all stored fields instead of > identifying > each field? This might not help at all, I'm just putting it out there as > something to try.
I will give it a try. > You also have grouping enabled. From what I understand, that can be slow. > If you turn that off, what happens to your elapsed times? Yes grouping is slow but not as much as bad as it was in Solr 1.4 which we are still using in production with a similar index (actually it has 17M documents on 6 shards but all on the same server). I expect the grouping to be much faster in 4.x than in 1.4 and I don't have this problem in 1.4. It's true however that I have some additional stored fields in my new setup. But this was done to limit the number of times I have to fetch the information from MySQL. > Your free RAM vs. index size is good, assuming that there's nothing else on > your Solr servers. With 12.8 million documents plus the use of grouping and > sorting, you might need a larger java heap. Try increasing it to 5GB as an > initial test and see if that makes any difference, either good or bad. > > Your email says you checked for huge GC, but without knowing exactly how > you checked, it's difficult to know what you would have actually found. I turned GC logging on and analyzed the resulting file. I've also taken a few heap dumps and all generations seem to be properly sized. I will give it a try to see if it affects the performance. > Thanks, > Shawn > > > ----- > Aucun virus trouvé dans ce message. > Analyse effectuée par AVG - www.avg.fr > Version: 2013.0.3392 / Base de données virale: 3209/6563 - Date: 09/08/2013
