On 12/14/2016 1:28 AM, forest_soup wrote: > We are doing index on the same http endpoint. But as we have shardnum=1 and > replicafactor=1, so each collection only has one core. So there should no > distributed update/query, as we are using solrj's CloudSolrClient which will > get the target URL of the solrnode when requesting to each collection. > > For the questions: > * What is the total physical memory in the machine? > 128GB > > * What is the max heap on each of the two Solr processes? > 32GB for each > > * What is the total index size in each Solr process? > Each Solr node(process) has 16 cores. 130GB for each solr core. So totally >> 2000G for each solr node.
This means that you have approximately 64GB left for your OS after deducting the heap sizes, which it must use for itself and for OS disk caching. With nearly 2 terabytes of index data on the machine, 64GB is nowhere near enough for good performance. The server will be VERY busy whenever there is query activity, so the CPU spike is what I would expect. For that much index data, I would hope to have somewhere between 512GB and 2 terabytes of memory. Adding machines and/or increasing memory in each machine would make your performance better and reduce CPU load. https://wiki.apache.org/solr/SolrPerformanceProblems > * What is the total tlog size in each Solr process? > 25m for each core. So totally 400m for each solr node. > > <updateLog> > <str name="dir">${solr.ulog.dir:}</str> > <int > name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}</int> > <int name="numRecordsToKeep">10000</int> > <int name="maxNumLogsToKeep">100</int> > </updateLog> Compared to the amount of index data, 400MB is tiny, but this will take a long time to process on restart. You might want to consider lowering the amount of data that the update log keeps so restarts are faster. > * What are your commit characteristics like -- both manual and automatic. > > <autoCommit> > <maxDocs>10000</maxDocs> > <maxTime>${solr.autoCommit.maxTime:59000}</maxTime> > <openSearcher>false</openSearcher> > </autoCommit> > <autoSoftCommit> > <maxDocs>5000</maxDocs> > <maxTime>${solr.autoSoftCommit.maxTime:31000}</maxTime> > </autoSoftCommit> I would personally remove the "maxDocs" portion of these settings and do the automatic commits based purely on time. For the amount of data you're handling, those are very low maxDocs numbers, and could result in very frequent commits when you index. The time values are lower than I would prefer, but are probably OK. The number of collections should be no problem. If there were hundreds or thousands, that might be different. Thanks, Shawn
