On 10/12/2018 8:46 AM, root23 wrote:
We are having an issue where we are seeing latency in updates. We are on
solr 6.
The new documents are reflected right away but updates to existing document
take sometime from 30 seconds to couple of minutes.
This is some relevant things from our solrconfig.
Our autosoft commit time is 500ms.(I know its low.)
<autoSoftCommit>
<maxTime>500</maxTime>
</autoSoftCommit>
That interval is WAY too low. The starting point I recommend for
autoSoftCommit is 60000 -- one minute. Depending on everything you're
dealing with, you might want to go up or down from there. The number
you ultimately end up with should be based on two things:
* How long it actually takes to open and warm a new searcher.
* A REALISTIC assessment of how quickly users must be able to see index
changes.
<ramBufferSizeMB>2048</ramBufferSizeMB>
You're just throwing away memory here. The default buffer size is
100MB. Some quick testing by Solr developers a long time ago indicated
that values above 128MB do not improve performance. Unless your Solr
documents are HUGE, you're not going to gain anything by making it
bigger, except a need for a very large heap -- especially when
allocating a buffer size measured in gigabytes.
<int name="maxThreadCount">8</int>
<int name="maxMergeCount">16</int>
For most situations, the merge scheduler doesn't need more than 1 for
threads and 6 for merges. Since you're on SSD, increasing threads won't
cause problems, but in my opinion you shouldn't go higher than about 3
or 4. It is extremely unlikely that you'll ever have 16 merges
scheduled simultaneously ... I've never seen more than about 4, and
that's from indexing MILLIONS of documents.
Also in our logs i see following .
Lucene flush took 0.003 seconds with 2 segments flushed and 0.022 MB flushed
SolrMetricsEventListener.java:65 - Lucene flush took 0.001 seconds with 1
segments flushed and 0.009 MB flushed
SolrMetricsEventListener.java:65 - Lucene flush took 0.006 seconds with 2
segments flushed and 0.057 MB flushed
SolrMetricsEventListener.java:65 - Lucene flush took 0.008 seconds with 5
segments flushed and 0.046 MB flushed
so if you see we are flushing too often and very small amount . May be its
because of the low autosoft commit time ?
Yes, I think the small flushes would indeed be from your low
autoSoftCommit time. Small segments aren't necessarily a problem, in
the end that means there will be a lot more segment merging, but with
small segments, the lower level merges will be quick. But it does mean
that the same data can be merged more times than necessary.
Can someone explain to me what is going on. Are these things related.
And why is that new document addition is fast but updates take time. I know
updates have to do a delete first. Is that a reason ?
I have no idea why there would be a discrepancy between those actions.
At the Lucene level, the work done is pretty much identical either way.
My best guess about why you would have ANY kind of latency issues is the
amount of time needed to open a new searcher, as Erick was talking
about. Can you share your entire solrconfig.xml file? Attachments
rarely make it to the list -- use a paste website or a file sharing website.
Thanks,
Shawn
