Re: solr hangs

I've seen this error happen when there are, well, too many threads simultaneously active and the stack space exhausts available memory, here's some discussion: https://stackoverflow.com/questions/36898701/how-does-java-jvm-allocate-stack-for-each-thread With your settings, you can _create_ 65K th

Re: solr hangs

Hi Shawn, The software config of the server is: OS: CentOS 7 Solr Version: 6.5 The system sends delete requests to the Solr server, the requests are around 5k to 6k.The exception occurred from time to time, and we have to restart the node to continue the service. The max threads setting os t

Re: solr hangs

On 8/7/2017 12:35 AM, hawk@139.com wrote: > Below is the OOM exception. > > 2017-08-07 12:45:48.446 WARN (qtp33524623-4275) [c:taoke s:shard2 > r:core_node4 x:taoke_shard2_replica1] o.e.j.u.t.QueuedThreadPool > java.lang.OutOfMemoryError: unable to create new native thread The last line that

Re: solr hangs

Below is the OOM exception. 2017-08-07 12:45:48.446 WARN (qtp33524623-4275) [c:taoke s:shard2 r:core_node4 x:taoke_shard2_replica1] o.e.j.u.t.QueuedThreadPool java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thr

Re: solr hangs

On 8/6/2017 10:29 PM, hawk@139.com wrote: > We found the problem is caused by the delete command. The request is used to > delete document by id. > > url --> http://10.91.1.120:8900/solr/taoke/update?&commit=true&wt=json > body --> {"delete":["20ec36ade0ca4da3bcd78269e2300f6f"]} > > When

Re: solr hangs

We found the problem is caused by the delete command. The request is used to delete document by id. url --> http://10.91.1.120:8900/solr/taoke/update?&commit=true&wt=json body --> {"delete":["20ec36ade0ca4da3bcd78269e2300f6f"]} When we send over 3000 requests, the Solr starts to give OOM ex

Re: solr hangs

You have several possibilities here: 1> you're hitting a massive GC pause that's timing out. You can turn on GC logging and analyze if that's the case. 2> your updates are getting backed up. At some point it's possible that the index writer blocks until merges are done IIUC. Does this ever happen

Re: solr hangs

Hi Eric, I am using the restful api directly. In our application, system issues the http request directly to Solr. ${solr.autoCommit.maxTime:15000} 1 true Thanks Hawk > On 6 Aug 2017, at 11:10 AM, Erick Erickson wrote: > > How are you updating 50K docs? SolrJ

Re: solr hangs

How are you updating 50K docs? SolrJ? If so are you using CloudSolrClient? What are your commit settings? Details matter. Best, Erick On Sat, Aug 5, 2017 at 6:19 PM, hawk wrote: > Hi All, > > I have encountered one problem of Solr. In our environment, we setup 2 Solr > nodes, every hour we will

Re: Solr hangs / LRU operations are heavy on cpu

We use filter very heavily because we run an e-commerce site which has a lot of faceting and drill downs configured at different paths on the store .. We are using master slave replication and we use slaves to support higher qps. filterCache : Concurrent LFU Cache(maxSize=1, initialSize

Re: Solr hangs / LRU operations are heavy on cpu

Are you faceting? That can sometimes use one of the caches (just glanced at stack trace...) as entries are pushed into and removed from the cache during the same request. Shot in the dark. Best, Erick On Fri, Mar 20, 2015 at 12:17 PM, Yonik Seeley wrote: > The document cache is not really going

Re: Solr hangs / LRU operations are heavy on cpu

The document cache is not really going to be taking up time here. How many concurrent requests (threads) are you testing with here? One thing I've seen over the years is a false sense of what is taking up time when benchmarks with a lot of threads are used. The reason is that when there are a lot

Re: Solr hangs / LRU operations are heavy on cpu

: we have quite a problem with Solr. We are running it in a config 6x3, and : suddenly solr started to hang, taking all the available cpu on the nodes. : : In the threads dump noticed things like this can eat lot of CPU time : : :- org.apache.solr.search.LRUCache.put​(LRUCache.java:116) :

Re: Solr hangs / LRU operations are heavy on cpu

Hello Shawn, In that case it makes it a bit strange the behavior as it was noticed. LRU was heavy on the CPU in threads dump, and I don't have any reasonable explanation for that. However switch to LFU seemingly solved the case. -- Best regards, Sergeymail

Re: Solr hangs / LRU operations are heavy on cpu

On 3/19/2015 8:49 PM, Umesh Prasad wrote: > It might be because LRUCache by default will try to evict its entries on > each call to put and putAll. LRUCache is built on top of java's > LinkedHashMap. Check the javadoc of removeEldestEntry >

Re: Solr hangs / LRU operations are heavy on cpu

Hello Umesh, Thank you, indeed that gave positive results so far. we changed completely to LFU. Today it went quite okay. We wait till it shows more stability and then work out the optimal cache size. Below is a summary of the changes. - - - - + + + + + -- Best regards, Sergey

Re: Solr hangs / LRU operations are heavy on cpu

It might be because LRUCache by default will try to evict its entries on each call to put and putAll. LRUCache is built on top of java's LinkedHashMap. Check the javadoc of removeEldestEntry

Re: Solr hangs on distributed updates

> As of 4.10, commits/optimize etc are executed in parallel. Excellent - thanks. On Tue, Dec 16, 2014 at 6:51 AM, Shalin Shekhar Mangar < shalinman...@gmail.com> wrote: > > On Tue, Dec 16, 2014 at 11:34 AM, Peter Keegan > wrote: > > > > > A distributed update is streamed to all available replicas

Re: Solr hangs on distributed updates

On Tue, Dec 16, 2014 at 11:34 AM, Peter Keegan wrote: > > > A distributed update is streamed to all available replicas in parallel. > > Hmm, that's not what I'm seeing with 4.6.1, as I tail the logs on leader > and replicas. Mark Miller comments on this last May: > > > http://mail-archives.apache.

Re: Solr hangs on distributed updates

> A distributed update is streamed to all available replicas in parallel. Hmm, that's not what I'm seeing with 4.6.1, as I tail the logs on leader and replicas. Mark Miller comments on this last May: http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201404.mbox/%3CetPan.534d8d6d.74b0dc51.

Re: Solr hangs on distributed updates

On Mon, Dec 15, 2014 at 8:41 PM, Peter Keegan wrote: > > If a timeout occurs, does the distributed update then go to the next > replica? > A distributed update is streamed to all available replicas in parallel. > > On Fri, Dec 12, 2014 at 3:42 PM, Shalin Shekhar Mangar < > shalinman...@gmail.co

Re: Solr hangs on distributed updates

If a timeout occurs, does the distributed update then go to the next replica? On Fri, Dec 12, 2014 at 3:42 PM, Shalin Shekhar Mangar < shalinman...@gmail.com> wrote: > > Sorry I should have specified. These timeouts go inside the > section and apply for inter-shard update requests only. The socke

Re: Solr hangs on distributed updates

I added distribUpdateConnTimeout and distribUpdateSoTimeout to solr.xml and the commit did timeout.(btw, is there any way to view solr.xml in the admin console?). Also, although we do have an init.d start/stop script for Solr, the 'stop' command was not executed during shutdown because there was n

Re: Solr hangs on distributed updates

The AMIs are Red Hat (not Amazon's) and the instances are properly sized for the environment (t1.micro for ZK, m3.xlarge for Solr). I do plan to add hooks for a clean shutdown of Solr when the VM is shut down, but if Solr takes too long, AWS may clobber it anyway. One frustrating part of auto scali

Re: Solr hangs on distributed updates

> The Solr leader should stop sending requests to the stopped replica once > that replica's live node is removed from ZK (after session expiry). Fwiw, here's the Zookeeper log entry for a graceful shutdown of the Solr replica: 2014-12-12 15:04:21,304 [myid:2] - INFO [ProcessThread(sid:2 cport:81

Re: Solr hangs on distributed updates

: No, I wasn't aware of these. I will give that a try. If I stop the Solr : jetty service manually, things recover fine, but the hang occurs when I : 'stop' or 'terminate' the EC2 instance. The Zookeeper leader reports a I don't know squat about AWS Auto-Scaling, (and barely anything about AWS)

Re: Solr hangs on distributed updates

Sorry I should have specified. These timeouts go inside the section and apply for inter-shard update requests only. The socket and connection timeout inside the shardHandlerFactory section apply for inter-shard search requests. On Fri, Dec 12, 2014 at 8:38 PM, Peter Keegan wrote: > Btw, are the

Re: Solr hangs on distributed updates

Okay, that should solve the hung threads on the leader. When you stop the jetty service then it is a graceful shutdown where existing requests finish before the searcher thread pool is shutdown completely. A EC2 terminate probably just kills the processes and leader threads just wait due to a lack

Re: Solr hangs on distributed updates

Btw, are the following timeouts still supported in solr.xml, and do they only apply to distributed search? ${socketTimeout:0} ${connTimeout:0} Thanks, Peter On Fri, Dec 12, 2014 at 3:14 PM, Peter Keegan wrote: > No, I wasn't aware of these. I will give that a try. If I stop the S

Re: Solr hangs on distributed updates

No, I wasn't aware of these. I will give that a try. If I stop the Solr jetty service manually, things recover fine, but the hang occurs when I 'stop' or 'terminate' the EC2 instance. The Zookeeper leader reports a 15-sec timeout from the stopped node, and expires the session, but the Solr leader n

Re: Solr hangs on distributed updates

Do you have distribUpdateConnTimeout and distribUpdateSoTimeout set to reasonable values in your solr.xml? These are the timeouts used for inter-shard update requests. On Fri, Dec 12, 2014 at 2:20 PM, Peter Keegan wrote: > We are running SolrCloud in AWS and using their auto scaling groups to sp

Re: Solr Hangs During Updates for over 10 minutes

d >> >>8 replicas with each shard at 100gb and more load and more indexing >> >>requests without this problem but we send docs in batches here and all >> >>fields are stored. Where as the trouble index has only 1 or 2 stored >> >>fields and only send docs 1 at a tim

Re: Solr Hangs During Updates for over 10 minutes

On 7/10/2013 6:57 AM, Jed Glazner wrote: So, while it's 'just as risky' as you say, it's 'less risky' than a new version of java and is possible to implement without downtime. I believe that if you update one node at a time, there should be no downtime. I've not actually tried this, so it wou

Re: Solr Hangs During Updates for over 10 minutes

On 7/10/2013 6:57 AM, Jed Glazner wrote: So we'll do what we can quickly to see if we can 'band-aid' the problem until we can upgrade to solr 4.4 Speaking of band-aids - does anyone know of a way to change the socket timeout/connection timeout for distributed updates? If you need to change Htt

Re: Solr Hangs During Updates for over 10 minutes

013 at 4:44 PM, Jed Glazner >>>>wrote: >>>> >> Hi Shawn, >>>> >> >>>> >> I have been trying to duplicate this problem without success for >>>>the >>>> >>last 2 weeks which is one reason I'm getting flustered

Re: Solr Hangs During Updates for over 10 minutes

gt;> >> >>> >> We do have a story to upgrade but that is still weeks if not months >>> >>before that gets rolled out to production. >>> >> >>> >> We have another cluster running the same version but with 8 shards >>>and >>>

Re: Solr Hangs During Updates for over 10 minutes

ade but that is still weeks if not months >> >>before that gets rolled out to production. >> >> >> >> We have another cluster running the same version but with 8 shards >>and >> >>8 replicas with each shard at 100gb and more load and more indexing &

Re: Solr Hangs During Updates for over 10 minutes

s and > >>8 replicas with each shard at 100gb and more load and more indexing > >>requests without this problem but we send docs in batches here and all > >>fields are stored. Where as the trouble index has only 1 or 2 stored > >>fields and only send docs 1 at a

Re: Solr Hangs During Updates for over 10 minutes

as the trouble index has only 1 or 2 stored >>fields and only send docs 1 at a time. >> >> Could that have anything to do with it? >> >> Jed >> >> >> Von Samsung Mobile gesendet >> >> >> >> Ursprüngliche Nachricht ---

Re: Solr Hangs During Updates for over 10 minutes

thing to do with it? > > Jed > > > Von Samsung Mobile gesendet > > > > Ursprüngliche Nachricht > Von: Shawn Heisey > Datum: 07.09.2013 18:33 (GMT+01:00) > An: solr-user@lucene.apache.org > Betreff: Re: Solr Hangs During Updates for over 10 minu

Re: Solr Hangs During Updates for over 10 minutes

On 7/9/2013 9:50 AM, Jed Glazner wrote: I'll give you the high level before delving deep into setup etc. I have been struggeling at work with a seemingly random problem when solr will hang for 10-15 minutes during updates. This outage always seems to immediately be proceeded by an EOF excepti

Re: Solr Hangs on startup

What are the last few lines of the Solr log? No errors, exceptions, or warnings? -- Jack Krupansky -Original Message- From: Cool Techi Sent: Friday, June 14, 2013 4:49 AM To: solr-user@lucene.apache.org Subject: Solr Hangs on startup Hi, We are using solr4.3 cloud setup, but for som

Re: solr hangs

Thanks for the response. I have given a size of 8gb for the instance and has only around few thousands of documents (with 15 fields each having small amount of data)..apparently the problem is the process (solr jetty instance) is consuming lots of threads...one time it consumed around 50k threads a

Re: solr hangs

You wrote that you can see such error "OutOfMemoryError". I had such problems when my caches were to big. It means that there is no more free memory in JVM and probably full gc starts running. How big is your Java heap? Maybe cache sizes in yout solr are to big according to your JVM settings. -- R

Re: SOLR hangs - update timeout - please help

Working for a week now, no signs of fatigue. Many thanks for all the hints R -- View this message in context: http://lucene.472066.n3.nabble.com/SOLR-hangs-update-timeout-please-help-tp3863851p3899004.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: SOLR hangs - update timeout - please help

Hi, To anyone still interested in this subject: after disabling windows nio handler in Jetty SOLR became more stable - currently it's been working for 3 days without any hanging or slowdown. I'll post next update in few days. -- View this message in context: http://lucene.472066.n3.nabble.com/SO

Re: SOLR hangs - update timeout - please help

Oops... my previous replies accidentally went off-list. I'll cut-n-paste below. OK, so it looks like there is probably no bug here - it's simply that commits can sometimes take a long time and updates were blocked during that time (and would have succeeded eventually except the jetty timeout was

Re: SOLR hangs - update timeout - please help

Yonik, I didn't say there was an update request active at the moment the thread dump was made, only that previous update requests failed with a timeout. So maybe this is the missing piece. I didn't enable nio with Jetty, probably it's there by default. Disabling it is the next thing to check. If so

Re: SOLR hangs - update timeout - please help

On Thu, Mar 29, 2012 at 1:50 PM, Rafal Gwizdala wrote: > Below i'm pasting the thread dump taken when the update was hung (it's also > attached to the first message of this topic) Interesting... It looks like there's only one thread in solr code (the one generating the thread dump). The stack tr

Re: SOLR hangs - update timeout - please help

More memory is not necessarily better, it can lead to longer, more intense garbage collections that cause things to stop. You might also consider lowering your memory allocation, but 2G is really not all that much so I somewhat doubt it's a problem but thought I'd mention it. Best Erick On Thu, M

Re: SOLR hangs - update timeout - please help

Guys, thanks for all the suggestions I will be trying them, one at a time. Imho it's too early to give up and look for another tool, I'll try to work on configuration and see what happens. The NRT looks quite promising, there are also tons of config options to change. As for now, I have made the up

Re: SOLR hangs - update timeout - please help

If you must have real-time search, you might look at systems that are designed to do that. MarkLogic isn't free, but it is fast and real-time. You can use their no-charge Express license for development and prototyping: http://developer.marklogic.com/express OK, back to Solr. wunder Search Guy

Re: SOLR hangs - update timeout - please help

On Thu, Mar 29, 2012 at 4:24 AM, Lance Norskog wrote: > 5-7 seconds- there's the problem. If you want to have documents > visible for search within that time, you want to use the trunk and > "near-real-time" search. A hard commit does several hard writes to the > disk (with the fsync() system call

Re: SOLR hangs - update timeout - please help

Have you tried using Solr 3.5 with RankingAlgorithm 1.4.1 ? Has NRT support and is very fast, updates about 5000 documents in about 490 ms (while updating 1m docs in batches of 5k). You can get more info from here: http://solr-ra.tgels.com/wiki/en/Near_Real_Time_Search_ver_3.x Regards, Nagen

Re: SOLR hangs - update timeout - please help

On 3/29/2012 2:49 AM, Rafal Gwizdala wrote: That's bad news. If 5-7 seconds is not safe then what is the safe interval for updates? Near real-time is not for me as it works only when querying by document Id - this doesn't solve anything in my case. I just want the index to be updated in real-time

Re: SOLR hangs - update timeout - please help

Could be garbage collection. Could be larger and larger merges. At some point your commit will cause all segments to be merged. It's likely that what's happening is you need to hit the "magic combination" of events, particularly the problem of too many warming searchers. So, look at your log files

Re: SOLR hangs - update timeout - please help

That's bad news. If 5-7 seconds is not safe then what is the safe interval for updates? Near real-time is not for me as it works only when querying by document Id - this doesn't solve anything in my case. I just want the index to be updated in real-time, 30-40 seconds delay is acceptable but not mu

Re: SOLR hangs - update timeout - please help

5-7 seconds- there's the problem. If you want to have documents visible for search within that time, you want to use the trunk and "near-real-time" search. A hard commit does several hard writes to the disk (with the fsync() system call). It does not run smoothly at that rate. It is no surprise tha

Re: SOLR hangs - update timeout - please help

Lance, I know there are many variables that's why I'm asking where to start and what to check. Updates are sent every 5-7 seconds, each update contains between 1 and 50 docs. Commit is done every time (on each update). Currently queries aren't very frequent - about 1 query every 3-5 seconds, but th

Re: SOLR hangs - update timeout - please help

How often are updates? And when are commits? How many CPUs? How much query load? There are so many variables. Check the mailing list archives and Solr issues, there might be a similar problem already discussed. Also, attachments do not work with Apache mailing lists. (Well, ok, they work for direc

Re: Solr Hangs

+91-9901768202 > About Me: http://about.me/rohitg > > -Original Message- > From: simon [mailto:mtnes...@gmail.com] > Sent: 02 September 2011 14:03 > To: solr-user@lucene.apache.org > Subject: Re: Solr Hangs > > That error has nothing to do with Solr - it looks as thou

RE: Solr Hangs

t why solr hangs and is there a way to automatically kill and restart. Regards, Rohit Mobile: +91-9901768202 About Me: http://about.me/rohitg -Original Message- From: simon [mailto:mtnes...@gmail.com] Sent: 02 September 2011 14:03 To: solr-user@lucene.apache.org Subject: Re: Solr Hangs

Re: Solr Hangs

That error has nothing to do with Solr - it looks as though you are trying to start the JVM with a heap size that is too big for the available physical memory. -Simon On Fri, Sep 2, 2011 at 2:15 AM, Rohit wrote: > Hi All, > > > > I am using Solr 3.0 and have 4 cores build into it with the follo

Re: Solr Hangs up after couple of hours

It would be very useful if you can take a threads dump while Solr is hanging. That will give indication where/why Solr is hanging. Bill On Mon, Aug 23, 2010 at 9:32 PM, Manepalli, Kalyan < kalyan.manepa...@orbitz.com> wrote: > Hi all, > I am facing a peculiar problem with Solr querying. D