On 11/29/22 13:35, Matias Laino wrote:
it takes about 10 minutes for those records to be available on search,
our commit strategy has been hard commit every 60 seconds and soft
commit every 1 second.
Don't do that. Chances are that the commits will take longer than 1
second to complete, so if you have autoSoftCommit set to 1 second,
you'll have lots of commits all trying to happen at the same time. I
think that probably explains your issues. That problem tends to
snowball out of control.
If you've got autoSoftCommit set to 1 second and it takes ten minutes
before updates are visible, that is definitely an indication that your
commits take much longer than 1 second to complete and you need to do
them less frequently.
The autoSoftCommit maxTime should generally be at least twice what
autoCommit is set to. It is completely unrealistic to expect document
changes to be visible in 1 second.
The autoCommit config should have openSearcher set to false.
I recommend not using maxDocs in autoCommit or autoSoftCommit. The
maxTime setting is much more predictable.
Other possibilities that would make the commit issues worse: 66 million
documents might require a heap size larger than 4GB for good
performnace. Also it is extremely unlikely that the size of the index
data on disk will even come close to fitting into the remaining 4GB of
free memory. If the system cannot cache the index data effectively,
performance will be terrible. Generally speaking, you're going to need a
lot more memory than 8GB.
https://cwiki.apache.org/confluence/display/solr/solrperformanceproblems
Thanks,
Shawn
