Thanks guys for your responses. That's a very very large cache size. It is likely to use a VERY large amount of heap, and autowarming up to 4096 entries at commit time might take many *minutes*. Each filterCache entry is maxDoc/8 bytes. On an index core with 70 million documents, each filterCache entry is at least 8.75 million bytes. Multiply that by 16384, and a completely full cache would need about 140GB of heap memory. 4096 entries will require 35GB. I don't think this cache is actually storing that many entries, or you would most certainly be running into OutOfMemoryError exceptions.
True, however, I have tried with the default filtercache at the beginning but the problem was still there. So, I don't think that is how I should increase the performance of my Solr. Moreover, as you mentioned, when I change the configuration, I should be running out of memory but that did not happen. Do you think my Solr has not taken the latest configs? I have restarted the Solr btw. Lately I have been trying different ways to improve this and I have created a brand new index on the same machine using 2 shards and it had few entries (about 5) and the performance was booming, I got the results back in 42 ms sometimes. What concerns me is that may be I am loading too much into one index so that is why this is killing the performance. Is there a recommended index size/document number and size that I should be looking at to tune this? Any other ideas other than increasing the memory size as I have already tried this? Regards, Salman On Thu, Oct 22, 2015 at 9:18 AM, Toke Eskildsen <t...@statsbiblioteket.dk> wrote: > On Wed, 2015-10-14 at 10:17 +0200, Jan Høydahl wrote: > > I have not benchmarked various number of segments at different sizes > > on different HW etc, so my hunch could very well be wrong for Salman’s > case. > > I don’t know how frequent updates there is to his data either. > > > > Have you done #segments benchmarking for your huge datasets? > > Only informally. However, the guys at UKWA run a similar scale index and > have done multiple segment-count-oriented tests. They have not published > a report, but there are measurements & graphs at > https://github.com/ukwa/shine/tree/master/python/test-logs > > - Toke Eskildsen, State and University Library, Denmark > > >
