Thanks Erick and Kshitij. Would try both the options and see what works best.
Regards Ankush Khanna On Fri, 9 Sep 2016 at 16:33 Erick Erickson <erickerick...@gmail.com> wrote: > The soft commit interval governs opening new > searchers, which should be "warmed" in order > load up caches. Mu guess is that you're not doing much > warming and thus seeing long search times. > > Most attachments are stripped by the mail server, > if you want people to see the images put them up somewhere > and provide a link is the usual practice. > > Have you examined the options here? > https://wiki.apache.org/solr/SolrPerformanceFactors > > Best, > Erick > > On Fri, Sep 9, 2016 at 2:10 AM, kshitij tyagi > <kshitij.shopcl...@gmail.com> wrote: > > Hi Ankush, > > > > As you are updating highly on one of the cores, hard commit will play a > > major role. > > > > Reason: During hard commits solr merges your segments and this is a time > > taking process. > > > > During merging of segments indexing of documents gets affected i.e. gets > > slower. > > > > Try figuring out the right number of segments you need to have and focus > on > > analysing the merge process of solr when you are updating high amount of > > data. > > > > You will need to find the correct time for hard commits and the required > > number of segments for the collection. > > > > Hope this helps. > > > > > > > > On Fri, Sep 9, 2016 at 2:13 PM, Ankush Khanna <a.kha...@rebuy.com> > wrote: > > > >> Hello, > >> > >> We are running some test for improving our solr performance. > >> > >> We have around 15 collections on our solr cluster. > >> But we are particularly interested in one collection holding high > amount of > >> documents. ( > >> https://gist.github.com/AnkushKhanna/9a472bccc02d9859fce07cb0204862da) > >> > >> Issue: > >> We see that there are high response time from the collection, for the > same > >> queries, when user load or update load is increased. > >> > >> What are we aiming for: > >> Low response time (lower than 3 sec) in high update/traffic. > >> > >> Current collection, production: > >> * Solr Cloud, 2 Shards 2 Replicas > >> * Indexed: 5.4 million documents > >> * 45 indexed fields per document > >> * Soft commit: 5 seconds > >> * Hard commit: 10 minutes > >> > >> Test Setup: > >> * Indexed: 3 million documents > >> * Rest is same as in production > >> * Using gatling to mimic behaviour of updates and user traffic > >> > >> Finding: > >> We see the problem occurring more often when: > >> * query size is greater than 2000 characters (we can limit the search to > >> 2000 characters, but is there a solution to do this without limiting the > >> size) > >> * there is high updates going on > >> * high user traffic > >> > >> Some settings I explored: > >> * 1 Shard and 3 Replicas > >> * Hard commit: 5 minutes (Referencing > >> https://lucidworks.com/blog/2013/08/23/understanding- > >> transaction-logs-softcommit-and-commit-in-sorlcloud/ > >> ) > >> > >> With both the above solutions we see some improvements, but not drastic. > >> (Attach images) > >> > >> I would like to have more insights into the following questions: > >> * Why is there an improvement with lowering the hard commit time, would > it > >> interesting to explore with lower hard commit time. > >> > >> Can some one provide some other pointer I could explore. > >> > >> Regards > >> Ankush Khanna > >> >
