Thank you Shawn, I'm definitely checking out those recommendations, but what I
cannot explain is how this worked fine for the last 3 months and then suddenly
this issue started happening.
On our application, customers expect that when a record is created, that record
should be available on search immediately (that's why the auto Soft commit of 1
second), what can you recommend for a situation like this?
This is the configuration we have so far:
<autoCommit>
<openSearcher>false</openSearcher>
<maxTime>60000</maxTime>
</autoCommit>
<autoSoftCommit>
<maxTime>1000</maxTime>
</autoSoftCommit>
MATIAS LAINO | DIRECTOR OF PASSARE REMOTE DEVELOPMENT
[email protected] | +54 11-6357-2143
-----Original Message-----
From: Shawn Heisey <[email protected]>
Sent: Tuesday, November 29, 2022 5:52 PM
To: [email protected]
Subject: Re: Very High CPU when indexing
On 11/29/22 13:35, Matias Laino wrote:
> it takes about 10 minutes for those records to be available on search,
> our commit strategy has been hard commit every 60 seconds and soft
> commit every 1 second.
Don't do that. Chances are that the commits will take longer than 1 second to
complete, so if you have autoSoftCommit set to 1 second, you'll have lots of
commits all trying to happen at the same time. I think that probably explains
your issues. That problem tends to snowball out of control.
If you've got autoSoftCommit set to 1 second and it takes ten minutes before
updates are visible, that is definitely an indication that your commits take
much longer than 1 second to complete and you need to do them less frequently.
The autoSoftCommit maxTime should generally be at least twice what autoCommit
is set to. It is completely unrealistic to expect document changes to be
visible in 1 second.
The autoCommit config should have openSearcher set to false.
I recommend not using maxDocs in autoCommit or autoSoftCommit. The maxTime
setting is much more predictable.
Other possibilities that would make the commit issues worse: 66 million
documents might require a heap size larger than 4GB for good performnace. Also
it is extremely unlikely that the size of the index data on disk will even come
close to fitting into the remaining 4GB of free memory. If the system cannot
cache the index data effectively, performance will be terrible. Generally
speaking, you're going to need a lot more memory than 8GB.
https://cwiki.apache.org/confluence/display/solr/solrperformanceproblems
Thanks,
Shawn
