Oh, and the other thing that occurs is if too many merges stack up, incoming updates are blocked which may be another source of delays. But still, updates to existing docs and adding new docs shouldn't be any different.
Best, Erick On Fri, Oct 12, 2018 at 10:42 AM Shawn Heisey <apa...@elyograg.org> wrote: > > On 10/12/2018 8:46 AM, root23 wrote: > > We are having an issue where we are seeing latency in updates. We are on > > solr 6. > > The new documents are reflected right away but updates to existing document > > take sometime from 30 seconds to couple of minutes. > > > > This is some relevant things from our solrconfig. > > > > Our autosoft commit time is 500ms.(I know its low.) > > <autoSoftCommit> > > <maxTime>500</maxTime> > > </autoSoftCommit> > > That interval is WAY too low. The starting point I recommend for > autoSoftCommit is 60000 -- one minute. Depending on everything you're > dealing with, you might want to go up or down from there. The number > you ultimately end up with should be based on two things: > > * How long it actually takes to open and warm a new searcher. > * A REALISTIC assessment of how quickly users must be able to see index > changes. > > > <ramBufferSizeMB>2048</ramBufferSizeMB> > > You're just throwing away memory here. The default buffer size is > 100MB. Some quick testing by Solr developers a long time ago indicated > that values above 128MB do not improve performance. Unless your Solr > documents are HUGE, you're not going to gain anything by making it > bigger, except a need for a very large heap -- especially when > allocating a buffer size measured in gigabytes. > > > <int name="maxThreadCount">8</int> > > <int name="maxMergeCount">16</int> > > For most situations, the merge scheduler doesn't need more than 1 for > threads and 6 for merges. Since you're on SSD, increasing threads won't > cause problems, but in my opinion you shouldn't go higher than about 3 > or 4. It is extremely unlikely that you'll ever have 16 merges > scheduled simultaneously ... I've never seen more than about 4, and > that's from indexing MILLIONS of documents. > > > Also in our logs i see following . > > Lucene flush took 0.003 seconds with 2 segments flushed and 0.022 MB flushed > > > > SolrMetricsEventListener.java:65 - Lucene flush took 0.001 seconds with 1 > > segments flushed and 0.009 MB flushed > > > > SolrMetricsEventListener.java:65 - Lucene flush took 0.006 seconds with 2 > > segments flushed and 0.057 MB flushed > > > > SolrMetricsEventListener.java:65 - Lucene flush took 0.008 seconds with 5 > > segments flushed and 0.046 MB flushed > > > > so if you see we are flushing too often and very small amount . May be its > > because of the low autosoft commit time ? > > Yes, I think the small flushes would indeed be from your low > autoSoftCommit time. Small segments aren't necessarily a problem, in > the end that means there will be a lot more segment merging, but with > small segments, the lower level merges will be quick. But it does mean > that the same data can be merged more times than necessary. > > > Can someone explain to me what is going on. Are these things related. > > And why is that new document addition is fast but updates take time. I know > > updates have to do a delete first. Is that a reason ? > > I have no idea why there would be a discrepancy between those actions. > At the Lucene level, the work done is pretty much identical either way. > My best guess about why you would have ANY kind of latency issues is the > amount of time needed to open a new searcher, as Erick was talking > about. Can you share your entire solrconfig.xml file? Attachments > rarely make it to the list -- use a paste website or a file sharing website. > > Thanks, > Shawn >
