Jean:
The basic search uses an “inverted index”, which is basically a list of terms
and the documents they appear in, e.g.
my - 1, 4, 9, 12
dog - 4, 8, 10
So the word “my” appears in docs 1, 4, 9 and 12, and “dog” appears in 4, 8, 10.
Makes
it easy to search for
my AND dog
for instance, obviously both appear in doc 4.
But that’s a lousy structure for faceting, where you have a list of documents
and are trying to
find the terms it has to count them up. For that, you want to “uninvert” the
above structure,
1 - my
4 - my dog
8 - dog
9 - my
10 - dog
12 - my
From there, it’s easy to say “count the distinct terms for docs 1 and 4 and put
them in a bucket”,
giving facet counts like
my (2)
dog (1)
If docValues=true, then the second structure is built at index time and
occupies memory at
run time out in MMapDirectory space, i.e. _not_ on the heap.
If docValues=false, the second structure is built _on_ the heap when it’s
needed, adding to
GC, memory pressure, CPU utilization etc.
So one theory is that when you upgraded your system (and you did completely
rebuild your
corpus, right?) you inadvertently changed the docValues property for one or
more fields that you
facet, group, sort, or use function queries on and Solr is doing all the extra
work of
uninverting the field that it didn’t have to before.
To answer that, you need to go through your schema and insure that
docValues=true is
set for any field you facet, group, sort, or use function queries on. If you do
change
this value, you need to blow away your index so there are no segments and index
all your documents again.
But that theory has problems:
1> why should Solr run for a while and then go crazy? It’d have to be that the
query that
triggers uninversion is uncommon.
2> docValues defaults to true for simple types in recent schemas. Perhaps you
pulled
over an old definition from your former schema?
One other thing: you mention a bit of custom code you needed to change. I
always try to
investigate that first. Is it possible to
1> reproduce the problem no a non-prod system
2> see what happens if you take the custom code out?
Best,
Erick
> On Oct 27, 2020, at 4:42 AM, Emir Arnautović <emir.arnauto...@sematext.com>
> wrote:
>
> Hi Jaan,
> It can be several things:
> caches
> fieldCache/fieldValueCache - it can be that you you are missing doc values on
> some fields that are used for faceting/sorting/functions and that uninverted
> field structures are eating your memory.
> filterCache - you’ve changed setting for filter caches and set it to some
> large value
> heavy queries
> return a lot of documents
> facet on high cardinality fields
> deep pagination
>
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
>> On 27 Oct 2020, at 08:48, Jaan Arjasepp <jaan.arjas...@coop.ee> wrote:
>>
>> Hello,
>>
>> We have been using SOLR for quite some time. We used 6.0 and now we did a
>> little upgrade to our system and servers and we started to use 8.6.1.
>> We use it on a Windows Server 2019.
>> Java version is 11
>> Basically using it in a default setting, except giving SOLR 2G of heap. It
>> used 512, but it ran out of memory and stopped responding. Not sure if it
>> was the issue. When older version, it managed fine with 512MB.
>> SOLR is not in a cloud mode, but in solo mode as we use it internally and it
>> does not have too many request nor indexing actually.
>> Document sizes are not big, I guess. We only use one core.
>> Document stats are here:
>> Num Docs: 3627341
>> Max Doc: 4981019
>> Heap Memory Usage: 434400
>> Deleted Docs: 1353678
>> Version: 15999036
>> Segment Count: 30
>>
>> The size of index is 2.66GB
>>
>> While making upgrade we had to modify one field and a bit of code that uses
>> it. Thats basically it. It works.
>> If needed more information about background of the system, I am happy to
>> help.
>>
>>
>> But now to the issue I am having.
>> If SOLR is started, at first 40-60 minutes it works just fine. CPU is not
>> high, heap usage seem normal. All is good, but then suddenly, the heap usage
>> goes crazy, going up and down, up and down and CPU rises to 50-60% of the
>> usage. Also I noticed over the weekend, when there are no writing usage, the
>> CPU remains low and decent. I can try it this weekend again to see if and
>> how this works out.
>> Also it seems to me, that after 4-5 days of working like this, it stops
>> responding, but needs to be confirmed with more heap also.
>>
>> Heap memory usage via JMX and jconsole ->
>> https://drive.google.com/file/d/1Zo3B_xFsrrt-WRaxW-0A0QMXDNscXYih/view?usp=sharing
>> As you can see, it starts of normal, but then goes crazy and it has been
>> like this over night.
>>
>> This is overall monitoring graphs, as you can see CPU is working hard or
>> hardly working. ->
>> https://drive.google.com/file/d/1_Gtz-Bi7LUrj8UZvKfmNMr-8gF_lM2Ra/view?usp=sharing
>> VM summary can be found here ->
>> https://drive.google.com/file/d/1FvdCz0N5pFG1fmX_5OQ2855MVkaL048w/view?usp=sharing
>> And finally to have better and quick overview of the SOLR executing
>> parameters that I have ->
>> https://drive.google.com/file/d/10VCtYDxflJcvb1aOoxt0u3Nb5JzTjrAI/view?usp=sharing
>>
>> If you can point me what I have to do to make it work, then I appreciate it
>> a lot.
>>
>> Thank you in advance.
>>
>> Best regards,
>> Jaan
>>
>>
>
