On 1/8/2015 8:57 AM, Andrew Butkus wrote: > We have 4gb usage (because the shards are split by 100 each shard is approx. > 2.8gb on disk), we have allocated 14gb min and 16gb max of ram to solr, so it > has plenty to use (the ram in the dashboard never goes above about 8gb - so > still plenty ). > > I've managed to reproduce the issue with shards= parameter, and think I have > proven the disk cache issue to not be the problem > > I'm simply querying the same shard, on the same server, multiple times (so > the shards index should always be in memory and never loaded from disk)? > > All but the last query are low ms ...
Extrapolating what Jack was saying on his reply ... with 100 shards and 4 replicas, you have 400 cores that are each about 2.8GB. That results in a total index size of just over a terabyte, with 140GB of index data on each of the eight servers. Assuming you have only one Solr instance per server, an ideal setup would have enough RAM for that 140GB of index plus the 16GB max heap, so 156GB of RAM. Because the ideal setup is rarely a strict requirement unless the query load is high, if you have 128 GB of RAM per server, then I would not be worried about performance. If you have less than that, then I would be worried. The behavior with the same shard listed multiple times is a little strange. That behavior could indicate problems with garbage collection pauses -- as Solr is building the memory structures necessary to compose the final response, it might fill up one of the heap generations to its current size limit and each subsequent allocation might require a significant garbage collection, stopping the world while it happens, but not freeing up any significant amount of memory in that particular heap generation. Have you tuned your garbage collection? If not, that is a likely suspect. If you run with the latest Oracle Java, you can use my settings and probably see good GC performance: https://wiki.apache.org/solr/ShawnHeisey Further down on the page is a good set of CMS parameters for earlier Java versions, if you can't run the latest. Thanks, Shawn
