Thank you Shawn for reading all of my previous entries and for a detailed answer.
To clarify, the third shard is used to store the recently added/updated data. Two main big cores take very long to replicate ( when a full replication is required) so the third one helps us to return the newly indexed documents quickly. It gets deleted every hour after we replicate the two other cores with last hour's of new/changed data. This third core is very small. As you said, with that big index and distributed queries , searches were too slow.So we tried to use the filtercache to speed up the queries. Filtercache was big as we have thousands of different filters. other caches were not very helpful as queries are not repetitive and there is heavy add/update to the index. So we have to use bigger heap size. Now,with that big heap size GC pauses was horrible, so we moved to Zing jvm. Zing jvm is now using 134 G of heap and does not have those big pauses but it also does not leave much memory for OS. I am now testing with small heap, small filter cache ( just the basic filters) and lot of memory available for OS disk cache. If that does not work, I am thinking of breaking my index down into small pieces. -- View this message in context: http://lucene.472066.n3.nabble.com/long-QTime-for-big-index-tp4037635p4037781.html Sent from the Solr - User mailing list archive at Nabble.com.
