Ah, so there are ~560 shards per node and not all nodes are indexing at the same time. Why is that? You can have better throughput if indexing on all nodes. If happy with shard size, you can create new collection with 49 shards every 2h and have everything the same and index on all nodes.
Back to main question: what is the heap utilisation? When you restart node what is heap utilisation? Do you see any errors in your logs? Do you see any errors in ZK logs? Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > On 27 Feb 2018, at 13:22, 苗海泉 <mseaspr...@gmail.com> wrote: > > Thanks for you reply again. > I just said that you may have some misunderstanding, we have 49 solr nodes, > each collection has 25 shards, each shard has only one replica of the data, > there is no copy, and I reduce the part of the cache. If you need the > metric data, I can check Come out to tell you, in addition we are only > additional system, there will not be any change action. > > 2018-02-27 20:05 GMT+08:00 Emir Arnautović <emir.arnauto...@sematext.com>: > >> Hi, >> It is hard to tell without looking more into your metrics. It seems to me >> that you are reaching limits of your cluster. I would doublecheck if memory >> is the issue. If I got it right, you have ~1120 shards per node. It takes >> some heap just to keep them open. If you have some caches enabled and if it >> is append only system, old shards will keep caches until reloaded. >> Probably will not make much diff, but with 25x2=50 shards and 49 nodes, >> one node will need to handle double indexing load. >> >> Emir >> -- >> Monitoring - Log Management - Alerting - Anomaly Detection >> Solr & Elasticsearch Consulting Support Training - http://sematext.com/ >> >> >> >>> On 27 Feb 2018, at 12:54, 苗海泉 <mseaspr...@gmail.com> wrote: >>> >>> In addition, we found that the rate was normal when the number of >>> collections was kept below 936 and the speed was slower and slower at >> 984. >>> Therefore, we could only temporarily delete the older collection, but now >>> we need more Online collection, there has been no good way to confuse us >>> for a long time, very much hope to give a solution to the problem of >> ideas, >>> greatly appreciated >>> >>> 2018-02-27 19:46 GMT+08:00 苗海泉 <mseaspr...@gmail.com>: >>> >>>> Thank you for reply. >>>> One collection has 25 shard one replica, one solr node has about 5T on >>>> desk. >>>> GC is checked ,and modify as follow : >>>> SOLR_JAVA_MEM="-Xms32768m -Xmx32768m " >>>> GC_TUNE=" \ >>>> -XX:+UseG1GC \ >>>> -XX:+PerfDisableSharedMem \ >>>> -XX:+ParallelRefProcEnabled \ >>>> -XX:G1HeapRegionSize=8m \ >>>> -XX:MaxGCPauseMillis=250 \ >>>> -XX:InitiatingHeapOccupancyPercent=75 \ >>>> -XX:+UseLargePages \ >>>> -XX:+AggressiveOpts \ >>>> -XX:+UseLargePages" >>>> >>>> 2018-02-27 19:27 GMT+08:00 Emir Arnautović < >> emir.arnauto...@sematext.com>: >>>> >>>>> Hi, >>>>> To get more complete picture, can you tell us how many shards/replicas >> do >>>>> you have per collection? Also what is index size on disk? Did you >> check GC? >>>>> >>>>> BTW, using 32GB heap prevents you from using compressed oops, resulting >>>>> in less memory available than 31GB. >>>>> >>>>> Thanks, >>>>> Emir >>>>> -- >>>>> Monitoring - Log Management - Alerting - Anomaly Detection >>>>> Solr & Elasticsearch Consulting Support Training - >> http://sematext.com/ >>>>> >>>>> >>>>> >>>>>> On 27 Feb 2018, at 11:36, 苗海泉 <mseaspr...@gmail.com> wrote: >>>>>> >>>>>> I encountered a more serious problem in the process of using solr. We >>>>> use >>>>>> the solr version is 6.0, our daily amount of data is about 500 billion >>>>>> documents, create a collection every hour, the online collection of >> more >>>>>> than a thousand, 49 solr nodes. If the collection in less than 800, >> the >>>>>> speed is still very fast, if the collection of the number of 1100 or >> so, >>>>>> the construction of solr index will drop sharply, one of the original >>>>>> program speed of about 2-3 million TPS, Dropped to only a few hundred >> or >>>>>> even tens of TPS, who have encountered a similar situation, there is >> no >>>>>> good idea to find this issue. By the way, solr a node memory we >> assigned >>>>>> 32G,We checked the memory, cpu, disk IO, network IO occupancy is no >>>>>> problem, belong to the normal state. Which friend encountered a >> similar >>>>>> problem, please inform the solution, thank you very much. >>>>> >>>>> >>>> >>>> >>>> -- >>>> ============================== >>>> 联创科技 >>>> 知行如一 >>>> ============================== >>>> >>> >>> >>> >>> -- >>> ============================== >>> 联创科技 >>> 知行如一 >>> ============================== >> >> > > > -- > ============================== > 联创科技 > 知行如一 > ==============================
