If you flood the system with updates, eventually you run out of merge threads and then updates block until the merge is done. It's may be that at this point you get into some deadlocked state, updates should slow down but not deadlock.
How are you updating? Are you stringing together a zillion clients (or threads)? Are you _not_ waiting for a return before sending more docs? If updates are blocking due to merging threads and your clients are waiting for the response from Solr before sending more docs, it should be self-limiting. Some things you can do to index quickly 1> batch updates together. I usually send packets of 1,000 updates, some experimentation will help you decide a good number. 2> make very sure you aren't committing from your clients. There's some discussion of rate-limiting Solr to avoid this kind of thing, but no definitive plans. Best, Erick On Sat, Feb 10, 2018 at 2:57 AM, Shawn Heisey <apa...@elyograg.org> wrote: > On 2/9/2018 8:20 PM, mmb1234 wrote: >> >> Ran /solr/58f449cec94a2c75-core-248/admin/luke at 7:05pm PST >> >> It showed "lastModified: 2018-02-10T02:25:08.231Z" indicating commit >> blocked >> for about 41 mins. >> Hard commit is set as 10secs in solrconfig.xml >> >> Other cores are also now blocked. >> >> https://jstack.review analysis of the thread dump says "Potential >> Deadlock" >> for three 3 cores. > > > The deadlock is not detected by analysis on the fastthread.io website. I > don't know if that's because jstack.review is a better tool or not. > > I do notice that you are using custom Solr components from Amazon and > VMWare. There is another user writing a thread on this list right now who > is also experiencing a lot of blocked threads. That user is ALSO running > custom components in Solr. I suspect there may be bugs in the custom > components. > >> Thread dump and top output are from that condition are available at >> >> https://github.com/mohsinbeg/datadump/tree/master/solr58f449cec94a2c75_core_248/solr58f449cec94a2c75_core_248 > > > Those screenshots were of the "htop" program. I very specifically said to > NOT use htop -- it must be the actual "top" program. > > Thanks, > Shawn >
