Shawn, Mikhail, Chris,
Thank you all for your feedback.
Unfortunately I cannot try your recommendations right away - this week is
busy.
Will post my results here next week.
Regards,
Denis
On Tue, Apr 24, 2018 at 11:33 AM Shawn Heisey wrote:
> On 4/24/2018 6:30 AM, Chris Ulicny wrote:
> > I ha
Hi Denis,
Merge works on segments and depending on merge strategy it is triggered
separately so there is no some queue between update executor and merge threads.
Re SPM - I am using it on a daily bases for most of my consulting work and if
you have SPM app you can invite me to it and I’ll take a
On 4/24/2018 6:30 AM, Chris Ulicny wrote:
I haven't worked with AWS, but recently we tried to move some of our solr
instances to a cloud in Google's Cloud offering, and it did not go well.
All of our problems ended up stemming from the fact that the I/O is
throttled. Any complicated enough query
Denis,
Can you enable infoSteam
https://lucene.apache.org/solr/guide/6_6/indexconfig-in-solrconfig.html#IndexConfiginSolrConfig-OtherIndexingSettings
and examine logs about throttling?
And what if you try without auto-commit?
On Tue, Apr 24, 2018 at 12:37 AM, Denis Demichev wrote:
> I conducted
I haven't worked with AWS, but recently we tried to move some of our solr
instances to a cloud in Google's Cloud offering, and it did not go well.
All of our problems ended up stemming from the fact that the I/O is
throttled. Any complicated enough query would require too many disk reads
to return
I conducted another experiment today with local SSD drives, but this did
not seem to fix my problem.
Don't see any extensive I/O in this case:
Device:tpskB_read/skB_wrtn/skB_readkB_wrtn
xvda 1.7688.83 5.521256191 77996
xvdb
When all indexing threads are occupied merging, incoming updates block
until at least one thread frees up IIUC.
The fact that you're not opening searchers doesn't matter as far as
merging is concerned, that happens regardless on hard commits.
Bumping your ram buffer up to 2G is usually unnecessar
Mikhail,
I see what you're saying. Thank you for the clarification.
Yes, there's no single line in the client code that contains a commit
statement.
The only thing I do: solr.add(collectionName, dataToSend); where solr is a
SolrClient.
Autocommits are set up on the server side for 2 minutes and th
Threads are hanging on merge io throthling
at
org.apache.lucene.index.MergePolicy$OneMergeProgress.pauseNanos(MergePolicy.java:150)
at
org.apache.lucene.index.MergeRateLimiter.maybePause(MergeRateLimiter.java:148)
at
org.apache.lucene.index.MergeRateLimiter.pause(MergeRa
Erick,
Thank you for your quick response.
I/O bottleneck: Please see another screenshot attached, as you can see disk
r/w operations are pretty low or not significant.
iostat==
Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz
avgqu-sz await r_await w_await svc
Have you changed any of the merge policy parameters? I doubt it but just asking.
My guess: your I/O is your bottleneck. There are a limited number of
threads (tunable) that are used for background merging. When they're
all busy, incoming updates are queued up. This squares with your
statement that
11 matches
Mail list logo
