Hi Jaan, It can be several things: caches fieldCache/fieldValueCache - it can be that you you are missing doc values on some fields that are used for faceting/sorting/functions and that uninverted field structures are eating your memory. filterCache - you’ve changed setting for filter caches and set it to some large value heavy queries return a lot of documents facet on high cardinality fields deep pagination
HTH, Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > On 27 Oct 2020, at 08:48, Jaan Arjasepp <jaan.arjas...@coop.ee> wrote: > > Hello, > > We have been using SOLR for quite some time. We used 6.0 and now we did a > little upgrade to our system and servers and we started to use 8.6.1. > We use it on a Windows Server 2019. > Java version is 11 > Basically using it in a default setting, except giving SOLR 2G of heap. It > used 512, but it ran out of memory and stopped responding. Not sure if it was > the issue. When older version, it managed fine with 512MB. > SOLR is not in a cloud mode, but in solo mode as we use it internally and it > does not have too many request nor indexing actually. > Document sizes are not big, I guess. We only use one core. > Document stats are here: > Num Docs: 3627341 > Max Doc: 4981019 > Heap Memory Usage: 434400 > Deleted Docs: 1353678 > Version: 15999036 > Segment Count: 30 > > The size of index is 2.66GB > > While making upgrade we had to modify one field and a bit of code that uses > it. Thats basically it. It works. > If needed more information about background of the system, I am happy to help. > > > But now to the issue I am having. > If SOLR is started, at first 40-60 minutes it works just fine. CPU is not > high, heap usage seem normal. All is good, but then suddenly, the heap usage > goes crazy, going up and down, up and down and CPU rises to 50-60% of the > usage. Also I noticed over the weekend, when there are no writing usage, the > CPU remains low and decent. I can try it this weekend again to see if and how > this works out. > Also it seems to me, that after 4-5 days of working like this, it stops > responding, but needs to be confirmed with more heap also. > > Heap memory usage via JMX and jconsole -> > https://drive.google.com/file/d/1Zo3B_xFsrrt-WRaxW-0A0QMXDNscXYih/view?usp=sharing > As you can see, it starts of normal, but then goes crazy and it has been like > this over night. > > This is overall monitoring graphs, as you can see CPU is working hard or > hardly working. -> > https://drive.google.com/file/d/1_Gtz-Bi7LUrj8UZvKfmNMr-8gF_lM2Ra/view?usp=sharing > VM summary can be found here -> > https://drive.google.com/file/d/1FvdCz0N5pFG1fmX_5OQ2855MVkaL048w/view?usp=sharing > And finally to have better and quick overview of the SOLR executing > parameters that I have -> > https://drive.google.com/file/d/10VCtYDxflJcvb1aOoxt0u3Nb5JzTjrAI/view?usp=sharing > > If you can point me what I have to do to make it work, then I appreciate it a > lot. > > Thank you in advance. > > Best regards, > Jaan > >
