Re: Checking in on Solr Progress

2020-03-02 Thread Jason Gerlowski
Very low-tech and manual, but worth mentioning... If there's a particularly large core that's doing a full recovery, and you have access to the disk itself you can navigate to the relevant directory for that core and run something like "watch -n 10 ls -lah" or "watch -n 10 du -sh ." to see how the

Re: Checking in on Solr Progress

2020-02-07 Thread Walter Underwood
I wrote some Python that checks CLUSTERSTATUS and reports replica status to Telegraf. Great for charts and alerts, but it only shows status, not progress. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Feb 7, 2020, at 7:58 AM, Erick Erickson wrote:

Re: Checking in on Solr Progress

2020-02-07 Thread Erick Erickson
I was wondering about using metrics myself. I confess I didn’t look to see what was already there either ;) Actually, using metrics might be easiest all told, but I also confess I have no clue what it takes to build a new metric in. Nor how to use the same (?) collection process for the 5 situa

Re: Checking in on Solr Progress

2020-02-07 Thread Jan Høydahl
Could we expose some high level recovery info as part of metrics api? Then people could track number of cores recovering, recovery time, recovery phase, number of recoveries failed etc, and also build alerts on top of that. Jan Høydahl > 6. feb. 2020 kl. 19:42 skrev Erick Erickson : > > There

Re: Checking in on Solr Progress

2020-02-06 Thread Erick Erickson
There’s actually a crying need for this, but there’s nothing that’s there yet, basically you have to look at the log files and try to figure it out. Actually I think this would be a great thing to work on, but it’d be pretty much all new. If you’d like, you can create a Solr Improvement Proposa

Re: Checking in on Solr Progress

2020-02-06 Thread dj-manning
Erick Erickson wrote > When you say “look”, where are you looking from? Http requests? SolrJ? The > admin UI? I'm open to looking form anywhere - http request, or the admin UI, or following a log if possible. My objective for this ask would be to human interactively follow/watch solr's recovery

Re: Checking in on Solr Progress

2020-02-06 Thread Erick Erickson
When you say “look”, where are you looking from? Http requests? SolrJ? The admin UI? Zookeeper is always the keeper of the state, so when the replica is “active” _AND_ the replica’s node is in the “live_nodes” hive it’s up. The Collections API CLUSTERSTATUS can help here if you’re not using Sol

Checking in on Solr Progress

2020-02-05 Thread dj-manning
Hi - I'm wondering if you would be able to point me in the right direction - I'm looking for the best way to check solr recover progress and status. I've seen a replica fall into recovery and I was wondering where I should look to monitor progress. Thank you in advance. -- Sent from: https://