On Fri, Oct 26, 2018 at 9:25 PM Sofiya Strochyk <s...@interlogic.com.ua> wrote:
> Hi everyone, > > We have a SolrCloud setup with the following configuration: > > - 4 nodes (3x128GB RAM Intel Xeon E5-1650v2, 1x64GB RAM Intel Xeon > E5-1650v2, 12 cores, with SSDs) > - One collection, 4 shards, each has only a single replica (so 4 > replicas in total), using compositeId router > - Total index size is about 150M documents/320GB, so about 40M/80GB > per node > - Zookeeper is on a separate server > - Documents consist of about 20 fields (most of them are both stored > and indexed), average document size is about 2kB > - Queries are mostly 2-3 words in the q field, with 2 fq parameters, > with complex sort expression (containing IF functions) > - We don't use faceting due to performance reasons but need to add it > in the future > - Majority of the documents are reindexed 2 times/day, as fast as the > SOLR allows, in batches of 1000-10000 docs. Some of the documents are also > deleted (by id, not by query) > - autoCommit is set to maxTime of 1 minute with openSearcher=false and > autoSoftCommit maxTime is 30 minutes with openSearcher=true. Commits from > clients are ignored. > - Heap size is set to 8GB. > > Target query rate is up to 500 qps, maybe 300, and we need to keep > response time at <200ms. But at the moment we only see very good search > performance with up to 100 requests per second. Whenever it grows to about > 200, average response time abruptly increases to 0.5-1 second. (Also it > seems that request rate reported by SOLR in admin metrics is 2x higher than > the real one, because for every query, every shard receives 2 requests: one > to obtain IDs and second one to get data by IDs; so target rate for SOLR > metrics would be 1000 qps). > > During high request load, CPU usage increases dramatically on the SOLR > nodes. It doesn't reach 100% but averages at 50-70% on 3 servers and about > 93% on 1 server (random server each time, not the smallest one). > > The documentation mentions replication to spread the load between the > servers. We tested replicating to smaller servers (32GB RAM, Intel Core > i7-4770). However, when we tested it, the replicas were going out of sync > all the time (possibly during commits) and reported errors like "PeerSync > Recovery was not successful - trying replication." Then they proceed with > replication which takes hours and the leader handles all requests > singlehandedly during that time. Also both leaders and replicas started > encountering OOM errors (heap space) for unknown reason. Heap dump analysis > shows that most of the memory is consumed by [J (array of long) type, my > best guess would be that it is "_version_" field, but it's still unclear > why it happens. Also, even though with replication request rate and CPU > usage drop 2 times, it doesn't seem to affect mean_ms, stddev_ms or p95_ms > numbers (p75_ms is much smaller on nodes with replication, but still not as > low as under load of <100 requests/s). > > Garbage collection is much more active during high load as well. Full GC > happens almost exclusively during those times. We have tried tuning GC > options like suggested here > <https://wiki.apache.org/solr/ShawnHeisey#CMS_.28ConcurrentMarkSweep.29_Collector> > and it didn't change things though. > > My questions are > > - How do we increase throughput? Is replication the only solution? > > 1. Increase the CPU speed 2. Increase the heap size (& tune the GC) 3. Replication 4. Have one more node on the hardware server (if cpu is not reaching 100%) > > - > - if yes - then why doesn't it affect response times, considering that > CPU is not 100% used and index fits into memory? > - How to deal with OOM and replicas going into recovery? > > 1. Increase the heap size 2. Memory debug to check for memory leaks (rare) > > - Is memory or CPU the main problem? (When searching on the internet, > i never see CPU as main bottleneck for SOLR, but our case might be > different) > > 1. Could be both > > - > - Or do we need smaller shards? Could segments merging be a problem? > - How to add faceting without search queries slowing down too much? > - How to diagnose these problems and narrow down to the real reason in > hardware or setup? > > 1. I would first tune all the software (OS, JVM, Solr) & benchmark the current hardware setup 2. Then i would play around with the hardware to check performance benefits > > > Any help would be much appreciated. > Increase in response time of 1 sec when you bump up the load indicates Queuing happening in your setup. (Since CPU is not 100% utilised, it most likely indicates memory-disk-network or software problem) Last, what is the nature of your request. Are the queries the same? Or they are very random? Random queries would need more tuning than if the queries the same. > Thanks! > -- > > *Sofiia Strochyk * > > > s...@interlogic.com.ua > [image: InterLogic] > www.interlogic.com.ua > > [image: Facebook icon] <https://www.facebook.com/InterLogicOfficial> [image: > LinkedIn icon] <https://www.linkedin.com/company/interlogic> > Deepak "The greatness of a nation can be judged by the way its animals are treated. Please consider stopping the cruelty by becoming a Vegan" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" Make In India : http://www.makeinindia.com/home
