Hi Erick,
Thanks for this information, I will look into it.
Main changes were regarding parsing the results JSON got from solr, not the
queries or updates.
Jaan
P.S. configuration change about requestParser was not it.
-----Original Message-----
From: Erick Erickson <erickerick...@gmail.com>
Sent: 27 October 2020 15:03
To: solr-user@lucene.apache.org
Subject: Re: SOLR uses too much CPU and GC is also weird on Windows server
Jean:
The basic search uses an “inverted index”, which is basically a list of terms
and the documents they appear in, e.g.
my - 1, 4, 9, 12
dog - 4, 8, 10
So the word “my” appears in docs 1, 4, 9 and 12, and “dog” appears in 4, 8, 10.
Makes it easy to search for my AND dog for instance, obviously both appear in
doc 4.
But that’s a lousy structure for faceting, where you have a list of documents
and are trying to find the terms it has to count them up. For that, you want to
“uninvert” the above structure,
1 - my
4 - my dog
8 - dog
9 - my
10 - dog
12 - my
From there, it’s easy to say “count the distinct terms for docs 1 and 4 and put
them in a bucket”, giving facet counts like
my (2)
dog (1)
If docValues=true, then the second structure is built at index time and
occupies memory at run time out in MMapDirectory space, i.e. _not_ on the heap.
If docValues=false, the second structure is built _on_ the heap when it’s
needed, adding to GC, memory pressure, CPU utilization etc.
So one theory is that when you upgraded your system (and you did completely
rebuild your corpus, right?) you inadvertently changed the docValues property
for one or more fields that you facet, group, sort, or use function queries on
and Solr is doing all the extra work of uninverting the field that it didn’t
have to before.
To answer that, you need to go through your schema and insure that
docValues=true is set for any field you facet, group, sort, or use function
queries on. If you do change this value, you need to blow away your index so
there are no segments and index all your documents again.
But that theory has problems:
1> why should Solr run for a while and then go crazy? It’d have to be
1> that the query that
triggers uninversion is uncommon.
2> docValues defaults to true for simple types in recent schemas.
2> Perhaps you pulled
over an old definition from your former schema?
One other thing: you mention a bit of custom code you needed to change. I
always try to investigate that first. Is it possible to
1> reproduce the problem no a non-prod system
2> see what happens if you take the custom code out?
Best,
Erick
> On Oct 27, 2020, at 4:42 AM, Emir Arnautović <emir.arnauto...@sematext.com>
> wrote:
>
> Hi Jaan,
> It can be several things:
> caches
> fieldCache/fieldValueCache - it can be that you you are missing doc values on
> some fields that are used for faceting/sorting/functions and that uninverted
> field structures are eating your memory.
> filterCache - you’ve changed setting for filter caches and set it to
> some large value heavy queries return a lot of documents facet on high
> cardinality fields deep pagination
>
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection Solr &
> Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
>> On 27 Oct 2020, at 08:48, Jaan Arjasepp <jaan.arjas...@coop.ee> wrote:
>>
>> Hello,
>>
>> We have been using SOLR for quite some time. We used 6.0 and now we did a
>> little upgrade to our system and servers and we started to use 8.6.1.
>> We use it on a Windows Server 2019.
>> Java version is 11
>> Basically using it in a default setting, except giving SOLR 2G of heap. It
>> used 512, but it ran out of memory and stopped responding. Not sure if it
>> was the issue. When older version, it managed fine with 512MB.
>> SOLR is not in a cloud mode, but in solo mode as we use it internally and it
>> does not have too many request nor indexing actually.
>> Document sizes are not big, I guess. We only use one core.
>> Document stats are here:
>> Num Docs: 3627341
>> Max Doc: 4981019
>> Heap Memory Usage: 434400
>> Deleted Docs: 1353678
>> Version: 15999036
>> Segment Count: 30
>>
>> The size of index is 2.66GB
>>
>> While making upgrade we had to modify one field and a bit of code that uses
>> it. Thats basically it. It works.
>> If needed more information about background of the system, I am happy to
>> help.
>>
>>
>> But now to the issue I am having.
>> If SOLR is started, at first 40-60 minutes it works just fine. CPU is not
>> high, heap usage seem normal. All is good, but then suddenly, the heap usage
>> goes crazy, going up and down, up and down and CPU rises to 50-60% of the
>> usage. Also I noticed over the weekend, when there are no writing usage, the
>> CPU remains low and decent. I can try it this weekend again to see if and
>> how this works out.
>> Also it seems to me, that after 4-5 days of working like this, it stops
>> responding, but needs to be confirmed with more heap also.
>>
>> Heap memory usage via JMX and jconsole ->
>> https://drive.google.com/file/d/1Zo3B_xFsrrt-WRaxW-0A0QMXDNscXYih/vie
>> w?usp=sharing As you can see, it starts of normal, but then goes
>> crazy and it has been like this over night.
>>
>> This is overall monitoring graphs, as you can see CPU is working hard
>> or hardly working. ->
>> https://drive.google.com/file/d/1_Gtz-Bi7LUrj8UZvKfmNMr-8gF_lM2Ra/vie
>> w?usp=sharing VM summary can be found here ->
>> https://drive.google.com/file/d/1FvdCz0N5pFG1fmX_5OQ2855MVkaL048w/vie
>> w?usp=sharing And finally to have better and quick overview of the
>> SOLR executing parameters that I have ->
>> https://drive.google.com/file/d/10VCtYDxflJcvb1aOoxt0u3Nb5JzTjrAI/vie
>> w?usp=sharing
>>
>> If you can point me what I have to do to make it work, then I appreciate it
>> a lot.
>>
>> Thank you in advance.
>>
>> Best regards,
>> Jaan
>>
>>
>
