Re: Solr using all available CPU and becoming unresponsive

__ From: Michael Gibney Sent: Monday, January 11, 2021 8:30 PM To: solr-user@lucene.apache.org Subject: Re: Solr using all available CPU and becoming unresponsive Hi Jeremy, Can you share your analysis chain configs? (SOLR-13336 can manifest in a similar way, and w

Re: Solr using all available CPU and becoming unresponsive

nd SynonymGraphFilterFactory from the query section > (and maybe the StopFilterFactory from the index section as well)? > > Thanks again, > Jeremy > > > From: Michael Gibney > Sent: Monday, January 11, 2021 8:30 PM > To: solr-user@lucene.apache.o

Re: Solr using all available CPU and becoming unresponsive

e query section (and maybe the StopFilterFactory from the index section as well)? Thanks again, Jeremy From: Michael Gibney Sent: Monday, January 11, 2021 8:30 PM To: solr-user@lucene.apache.org Subject: Re: Solr using all available CPU and becoming unresponsive

Re: Solr using all available CPU and becoming unresponsive

Hi Jeremy, Can you share your analysis chain configs? (SOLR-13336 can manifest in a similar way, and would affect 7.3.1 with a susceptible config, given the right (wrong?) input ...) Michael On Mon, Jan 11, 2021 at 5:27 PM Jeremy Smith wrote: > Hello all, > We have been struggling with an i

Re: SOLR USING 100% percent CPU and not responding after a while

Hi, Show us more graphs. Is the GC working hard? Any of the JVM mem pools at or near 100%? SPM for Solr is your friend for long term monitoring/alerting/trends, jconsole and visualvm for a quick look. Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Suppor

RE: SOLR USING 100% percent CPU and not responding after a while

I have the same problem, please look at the image: And this is on idle. Index size is about 90Gb. Solr 4.4.0. Memory is not an issue, there's a lot. RAID 10 (15000RPM rapid hdd). -- View this message in context: http://luce

RE: SOLR USING 100% percent CPU and not responding after a while

Hi Biva, Any luck on this? Even we are facing same issue with exactly same configuration and setup. Any inputs will help a lot. -- View this message in context: http://lucene.472066.n3.nabble.com/SOLR-USING-100-percent-CPU-and-not-responding-after-a-while-tp4021359p4083234.html Sent from the

Re: solr - using fq parameter does not retrieve an answer

Thanks. On Mon, Aug 5, 2013 at 4:57 PM, Shawn Heisey wrote: > On 8/5/2013 2:35 AM, Mysurf Mail wrote: > > When I query using > > > > http://localhost:8983/solr/vault/select?q=*:* > > > > I get reuslts including the following > > > > > > ... > > ... > > 7 > > ... > > > > > > Now I try

Re: solr - using fq parameter does not retrieve an answer

On 8/5/2013 2:35 AM, Mysurf Mail wrote: > When I query using > > http://localhost:8983/solr/vault/select?q=*:* > > I get reuslts including the following > > > ... > ... > 7 > ... > > > Now I try to get only that row so I add to my query fq=VersionNumber:7 > > http://localhost:8983/so

Re: solr - using fq parameter does not retrieve an answer

Is VersionNumber an "indexed" field, or just "stored"? -- Jack Krupansky -Original Message- From: Mysurf Mail Sent: Monday, August 05, 2013 4:35 AM To: solr-user@lucene.apache.org Subject: solr - using fq parameter does not retrieve an answer When I query using http://localhost:8

Re: Solr using a ridiculous amount of memory

16, 2013 9:36 AM To: solr-user@lucene.apache.org Subject: Re: Solr using a ridiculous amount of memory It was interesting to read this post. I had similar issue on Solr v4.2.1. The nature of our document is that it has huge multiValued fields and we were able to knock off out server in about 30mu

Re: Solr using a ridiculous amount of memory

It was interesting to read this post. I had similar issue on Solr v4.2.1. The nature of our document is that it has huge multiValued fields and we were able to knock off out server in about 30muns We then found a bug "Lucene-4995" which was causing all the problem. Applying the patch has helped a

Re: Solr using a ridiculous amount of memory

John: If you'd like to add your experience to the Wiki, create an ID and let us know what it is and we'll add you to the contributors list. Unfortunately we had problems with spam pages to we added this step. Make sure you include your logon in the request. Thanks, Erick On Fri, Jun 14, 2013 at

Re: Solr using a ridiculous amount of memory

On Fri, 2013-06-14 at 14:55 +0200, John Nielsen wrote: > Sorry for not getting back to the list sooner. Time not important, only feedback important (apologies to Fifth Element). > After some major refactoring, our 15 cores have now turned into ~500 cores > and our memory consumption has dropped d

Re: Solr using a ridiculous amount of memory

Sorry for not getting back to the list sooner. It seems like I finally solved the memory problems by following Toke's instruction of splitting the cores up into smaller chunks. After some major refactoring, our 15 cores have now turned into ~500 cores and our memory consumption has dropped dramati

Re: Solr using a ridiculous amount of memory

Hmmm. There has been quite a bit of work lately to support a couple of things that might be of interest (4.3, which Simon cut today, probably available to all mid next week at the latest). Basically, you can choose to pre-define all the cores in solr.xml (so-called "old style") _or_ use the new-sty

Re: Solr using a ridiculous amount of memory

> You are missing an essential part: Both the facet and the sort > structures needs to hold one reference for each document > _in_the_full_index_, even when the document does not have any values in > the fields. > Wow, thank you for this awesome explanation! This is where the penny dropped for me.

Re: Solr using a ridiculous amount of memory

On Thu, 2013-04-18 at 11:59 +0200, John Nielsen wrote: > Yes, thats right. No search from any given client ever returns > anything from another client. Great. That makes the 1 core/client solution feasible. [No sort & facet warmup is performed] [Suggestion 1: Reduce the number of sort fields by

Re: Solr using a ridiculous amount of memory

> > > http://172.22.51.111:8000/solr/default1_Danish/search > > [...] > > > &fq=site_guid%3a(10217) > > This constraints to hits to a specific customer, right? Any search will > only be in a single customer's data? > Yes, thats right. No search from any given client ever returns anything from anot

Re: Solr using a ridiculous amount of memory

On Thu, 2013-04-18 at 08:34 +0200, John Nielsen wrote: > [Toke: Can you find the facet fields in any of the other caches?] > Yes, here it is, in the field cache: > http://screencast.com/t/mAwEnA21yL > Ah yes, mystery solved, my mistake. > http://172.22.51.111:8000/solr/default1_Danish/search

Re: Solr using a ridiculous amount of memory

> That was strange. As you are using a multi-valued field with the new setup, they should appear there. Yes, the new field we use for faceting is a multi valued field. > Can you find the facet fields in any of the other caches? Yes, here it is, in the field cache: http://screencast.com/t/mAwEnA

RE: Solr using a ridiculous amount of memory

Whopps. I made some mistakes in the previous post. Toke Eskildsen [t...@statsbiblioteket.dk]: > Extrapolating from 1.4M documents and 180 clients, let's say that > there are 1.4M/180/5 unique terms for each sort-field and that their > average length is 10. We thus have > 1.4M*log2(1500*10*8) + 1

RE: Solr using a ridiculous amount of memory

John Nielsen [j...@mcb.dk]: > I never seriously looked at my fieldValueCache. It never seemed to get used: > http://screencast.com/t/YtKw7UQfU That was strange. As you are using a multi-valued field with the new setup, they should appear there. Can you find the facet fields in any of the other

Re: Solr using a ridiculous amount of memory

> I am surprised about the lack of "UnInverted" from your logs as it is logged on INFO level. Nope, no trace of it. No mention either in Logging -> Level from the admin interface. > It should also be available from the admin interface under collection/Plugin / Stats/CACHE/fieldValueCache. I neve

RE: Solr using a ridiculous amount of memory

John Nielsen [j...@mcb.dk] wrote: > I managed to get this done. The facet queries now facets on a multivalue > field as opposed to the dynamic field names. > Unfortunately it doesn't seem to have done much difference, if any at all. I am sorry to hear that. > documents = ~1.400.000 > references

Re: Solr using a ridiculous amount of memory

I managed to get this done. The facet queries now facets on a multivalue field as opposed to the dynamic field names. Unfortunately it doesn't seem to have done much difference, if any at all. Some more information that might help: The JVM memory seem to be eaten up slowly. I dont think that the

Re: Solr using a ridiculous amount of memory

Might be obvious, but just in case - remember that you'll need to re-index your content once you've added docValues to your schema, in order to get the on-disk files to be created. Upayavira On Mon, Mar 25, 2013, at 03:16 PM, John Nielsen wrote: > I apologize for the slow reply. Today has been ki

Re: Solr using a ridiculous amount of memory

I did a search. I have no occurrence of "UnInverted" in the solr logs. > Another explanation for the large amount of memory presents itself if > you use a single index: If each of your clients facet on at least one > fields specific to the client ("client123_persons" or something like > that), the

Re: Solr using a ridiculous amount of memory

On Mon, 2013-04-15 at 10:25 +0200, John Nielsen wrote: > The FieldCache is the big culprit. We do a huge amount of faceting so > it seems right. Yes, you wrote that earlier. The mystery is that the math does not check out with the description you have given us. > Unfortunately I am super swamped

Re: Solr using a ridiculous amount of memory

Yes and no, The FieldCache is the big culprit. We do a huge amount of faceting so it seems right. Unfortunately I am super swamped at work so I have precious little time to work on this, which is what explains my silence. Out of desperation, I added another 32G of memory to each server and increa

Re: Solr using a ridiculous amount of memory

On Sun, 2013-03-24 at 09:19 +0100, John Nielsen wrote: > Our memory requirements are running amok. We have less than a quarter of > our customers running now and even though we have allocated 25GB to the JVM > already, we are still seeing daily OOM crashes. Out of curiosity: Did you manage to pinp

Re: Solr using a ridiculous amount of memory

I apologize for the slow reply. Today has been killer. I will reply to everyone as soon as I get the time. I am having difficulties understanding how docValues work. Should I only add docValues to the fields that I actually use for sorting and faceting or on all fields? Will the docValues magic

Re: Solr using a ridiculous amount of memory

nt: Sunday, March 24, 2013 2:00 PM To: solr-user@lucene.apache.org Subject: Re: Solr using a ridiculous amount of memory Just to get started, do you hit OOM quickly with a few expensive queries, or is it after a number of hours and lots of queries? Does Java heap usage seem to be growing linearly

RE: Solr using a ridiculous amount of memory

Toke Eskildsen [t...@statsbiblioteket.dk]: > If your whole index has 10M documents, which each has 100 values > for each field, with each field having 50M unique values, then the > memory requirement would be more than > 10M*log2(100*10M) + 100*10M*log2(50M) bit ~= 340MB/field ~= > 1.6GB for face

RE: Solr using a ridiculous amount of memory

From: John Nielsen [j...@mcb.dk]: > The index is about 35GB on disk with each register between 15k and 30k. > (This is simply the size of a full xml reply of one register. I'm not sure > how to measure it otherwise.) > Our memory requirements are running amok. We have less than a quarter of > our

Re: Solr using a ridiculous amount of memory

On Sun, Mar 24, 2013 at 4:19 AM, John Nielsen wrote: > Schema with DocValues attempt at solving problem: > http://pastebin.com/Ne23NnW4 > Config: http://pastebin.com/x1qykyXW > This schema isn't using docvalues, due to a typo in your config. it should not be DocValues="true" but docValues="true"

Re: Solr using a ridiculous amount of memory

Just to get started, do you hit OOM quickly with a few expensive queries, or is it after a number of hours and lots of queries? Does Java heap usage seem to be growing linearly as queries come in, or are there big spikes? How complex/rich are your queries (e.g., how many terms, wildcards, fac

Re: SOLR USING 100% percent CPU and not responding after a while

copied the memory chart here. It shows that most of the memory was > being used for cache. > Thank you > > > -Original Message- > From: Rafał Kuć [mailto:r@solr.pl] > Sent: Tuesday, November 20, 2012 11:25 AM > To: solr-user@lucene.apache.org > Subject: Re: SOLR USI

Re: SOLR USING 100% percent CPU and not responding after a while

esday, November 20, 2012 11:25 AM > To: solr-user@lucene.apache.org > Subject: Re: SOLR USING 100% percent CPU and not responding after a while > Hello! > The first thing one should ask is what is the cause of the 100% CPU > utilization. It can be an issue with the memory or your

Re: Solr using very high I/O

Do you commit often? If so, try committing less often :) /Martin On Wed, Dec 7, 2011 at 12:16 PM, Adrian Fita wrote: > Hi. I experience an issue where Solr is using huge ammounts of I/O. > Basically it uses the whole HDD continously, leaving nothing to the > other processes. Solr is called by a

Re: Solr using very high I/O

Can u share more info: like what is your H/W infra, CPU, RAM, HDD?? >From where you pick the records/documents to index; RDBMS, Files, Network?? Regards Pravesh -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-using-very-high-I-O-tp3567076p3569903.html Sent from the Solr

Re: Solr: using to index large folders recursively containing lots of different documents, and querying over the web

Solr itself does all three things. There is no need for Nutch- that is needed for crawling web sites, not file systems (as the original question specifies). Solr operates as a web service, running in any Java servlet container. Detecting changes to files is more tricky: there is no implementation

Re: Solr: using to index large folders recursively containing lots of different documents, and querying over the web

Nutch can crawl the file system as well. Nutch 1.x can also provide search but this is delegated to Solr in Nutch 2.x. Solr can provide the search and Nutch can provide Solr with content from your intranet. On Friday 14 January 2011 13:17:52 Cathy Hemsley wrote: > Hi, > Thanks for suggesting thi

Re: Solr: using to index large folders recursively containing lots of different documents, and querying over the web

On Fri, 2011-01-14 at 13:05 +0100, Cathy Hemsley wrote: > I hope you can help. We are migrating our intranet web site management > system to Windows 2008 and need a replacement for Index Server to do the > text searching. I am trying to establish if Lucene and Solr is a feasible > replacement, bu

Re: Solr: using to index large folders recursively containing lots of different documents, and querying over the web

Please visit the Nutch project. It is a powerful crawler and can integrate with Solr. http://nutch.apache.org/ > Hi Solr users, > > I hope you can help. We are migrating our intranet web site management > system to Windows 2008 and need a replacement for Index Server to do the > text searching

Re: Solr using 1500 threads - is that normal?

Glad to help. Do be aware that there are several config values that influence the commit frequency, they might also be relevant. Best Erick On Thu, Jul 29, 2010 at 5:11 AM, Christos Constantinou < ch...@simpleweb.co.uk> wrote: > Eric, > > Thank you very much for the indicators! I had a closer lo

Re: Solr using 1500 threads - is that normal?

Eric, Thank you very much for the indicators! I had a closer look at the commit intervals and it seems that the application is gradually increasing the commits to almost once per second after some time - something that was hidden in the massive amount of queries in the log file. I have changed

Re: Solr using 1500 threads - is that normal?

Your commits are very suspect. How often are you making changes to your index? Do you have autocommit on? Do you commit when updating each document? Committing too often and consequently firing off warmup queries is the first place I'd look. But I agree with dc tech, 1,500 is wy more than I wou

Re: Solr using 1500 threads - is that normal?

1,500 threads seems extreme by any standards so there is something happening in your install. Even with appservers for web apps, typically 100 would be a fair # of threads. On 7/28/10, Christos Constantinou wrote: > Hi, > > Solr seems to be crashing after a JVM exception that new threads cannot

RE: Solr Using

Do these JSP pages compile under another servlet container? If the JSP pages have Java .15 or Java 1.6 syntax features, they will not compile under Jboss 4.0.2. The jboss 4.0.2 jsp compiler does the Java 1.4 language. I ran into this problem moving from a new tomcat to an older jboss. -Origin

Re: Solr Using

What is the syntax error? Which JSP? Please give the stack trace too. On Wed, Sep 24, 2008 at 12:24 PM, Dinesh Gupta <[EMAIL PROTECTED]>wrote: > > Which version of tomcat required. > > I installed jboss4.0.2 which have tomcat5.5.9. > > JSP pages are not going to compile. > > Its giving syntax er

Re: Solr Using

>catalogues=null; > > >Document document = new ProductDocument().toDocument(pbi, >productAllCategories, productAllCatalogues, > productDirectCategories, categoryNames); > >categoryNames =null; >pbi=null; >produc

RE: Solr Using

ames =null; pbi=null; productAllCatalogues =null; productAllCategories =null; productDirectCategories=null; categoryNames=null; long time = System.currentTimeMillis() - start; if (time > longestIndexTime) { longestIndex

Re: Solr Using

Dinesh, Please have a look at the Solr tutorial first. Then have a look at the new DataImportHandler - there is a very detailed page about it on the Wiki. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: Dinesh Gupta <[EMAIL PROTECTED]> >

Re: SOLR using sort but not sorting

: : now I add sort=last-name asc : : this DOES change the ordering of the docs but it's not exactly : alphabetically. Lucene sorting can't work on a field with more then one indexed term per document, this was breifly covered in the "sort" param docs, but i have beefed up the info http