Re: SolrCloud 6.6.2 suddenly crash due to slow queries and Log4j issue

Shawn, According to the log4j description ( https://bz.apache.org/bugzilla/show_bug.cgi?id=57714), the issue is related to lock during appenders collection process. In addition to CONSOLE and file appenders in the default log4j.properties, my customer added 2 extra FileAppender dedicated to all r

Re: SolrCloud 6.6.2 suddenly crash due to slow queries and Log4j issue

Hi Shawn, Thank you for your response. You are confirming my diagnosis. This is in fact a 8 nodes cluster with one single collection with 4 shards and 1 replica (8 cores). 4 Gb heap and 90 Gb Ram When no issue occurs nearly 50% of the heap is used. Num Docs in collection : 10.000.000 Num Do

Re: SolrCloud 6.6.2 suddenly crash due to slow queries and Log4j issue

On 10/18/2020 3:22 AM, Dominique Bejean wrote: A few months ago, I reported an issue with Solr nodes crashing due to the old generation heap growing suddenly and generating OOM. This problem occurred again this week. I have threads dumps for each minute during the 3 minutes the problem occured. I

SolrCloud 6.6.2 suddenly crash due to slow queries and Log4j issue

doExecute(AbstractHttpClient.java:816) - waiting to lock <0x00070087b910> (a org.apache.solr.util.stats.InstrumentedHttpClient) === 15h58 -> Young Gen heap full -> OOM The global scenario is (6 solr nodes) 1/ The problem starts with very slow queries (30 seconds to 2 minutes) o

Re: Slow queries until core is reindexed

I'm pretty sure we found the problem. It's related to memory. Sometimes Windows seems to unmap index files from memory, because other processes need it. To force Windows to map index files again, we need to rebuild the index. We can clearly see this behaviour with tools like RAMMap. With servers

RE: Log slow queries to SQL Database using Log4j2 (JDBC)

ginal Message- From: Walter Underwood Sent: Dienstag, 26. Mai 2020 02:06 To: solr-user@lucene.apache.org Subject: Re: Log slow queries to SQL Database using Log4j2 (JDBC) I would back up and do this a different way, with off-the-shelf parts. Send the logs to syslog or your favorite log aggregator.

Re: Log slow queries to SQL Database using Log4j2 (JDBC)

I would back up and do this a different way, with off-the-shelf parts. Send the logs to syslog or your favorite log aggregator. From there, configure something that puts them into an ELK stack (Elasticsearch, Logstash, Kibana). A commercial version of this is logz.io . Traditi

Log slow queries to SQL Database using Log4j2 (JDBC)

Hi everyone, For our Solr instance I have the requirement that all queries should be logged, so that we can later on analyze, which search texts were queried most often. Were using solr 8.3.1 using the official docker image, hosted on Azure. My approach for implementing this, was now to confi

Slow queries until core is reindexed

Hi all, We have a solr 8.4.1 server running on Windows. (Very simple setup.) 16GB RAM / JVM-Mem set to 4GB Solr host 4 cores. (2 GB + 1GB + 75MB + 75MB) Full data import every night. No delta import. This server is used for tests by 2 people. (very low request rate) We have an issue we don't unde

Re: CommonTerms & slow queries

> > All great advice thanks Michael, have an excellent weekend! Testing the > common grams > -Craig

Re: CommonTerms & slow queries

Solr 8.0.0, single instance, single core, 50m records (38gb > index) > > > on one SSD, 96gb ram, 16 cores CPU > > > > > > Most queries run very very fast <1 sec however we have noticed queries > > > containing "common" words are quite slow

Re: CommonTerms & slow queries

t; > on one SSD, 96gb ram, 16 cores CPU > > > > Most queries run very very fast <1 sec however we have noticed queries > > containing "common" words are quite slow sometimes 10+sec , currently > using > > edismax with 2 text_general fields,. qf, and pf, qs=0,ps

Re: CommonTerms & slow queries

rds are quite slow sometimes 10+sec , currently using > edismax with 2 text_general fields,. qf, and pf, qs=0,ps=0 > > I came across these which describe the issue. > > https://www.hathitrust.org/blogs/large-scale-search/slow-queries-and-common-words-part-2 > > > https

CommonTerms & slow queries

l fields,. qf, and pf, qs=0,ps=0 I came across these which describe the issue. https://www.hathitrust.org/blogs/large-scale-search/slow-queries-and-common-words-part-2 https://lucene.apache.org/core/5_5_3/queries/org/apache/lucene/queries/CommonTermsQuery.html Test queries with issues : 1. th

Re: slow queries

> "\n1.0 >> >> > = (MATCH) MatchAllDocsQuery, product of:\n 1.0 = queryNorm\n", " >> >> > Product:15295602": "\n1.0 = (MATCH) MatchAllDocsQuery, product of:\n >> 1.0 >> >> = >> >> > queryNorm\n", "Pro

Re: slow queries

Query, product of:\n 1.0 = queryNorm\n", " > >> > Product:50827742": "\n1.0 = (MATCH) MatchAllDocsQuery, product of:\n > 1.0 > >> = > >> > queryNorm\n", "Product:5771722": "\n1.0 = (MATCH) MatchAllDocsQuery, > >> >

Re: slow queries

:\n 1.0 >> = >> > queryNorm\n", "Product:5771722": "\n1.0 = (MATCH) MatchAllDocsQuery, >> > product of:\n 1.0 = queryNorm\n", "Product:3245678": "\n1.0 = (MATCH) >> > MatchAllDocsQuery, product of:\n 1.0 = queryNorm\n", "Product:13

Re: slow queries

t; queryNorm\n" }, "QParser": "ExtendedDismaxQParser", "altquerystring": > null, > > "boost_queries": null, "parsed_boost_queries": [], "boostfuncs": null, " > > filter_queries": [ "{!cost=1 cache=true}typ

Re: slow queries

[ "{!cost=1 cache=true}type_s:Product AND > is_valid_b:true", > "{!cost=50 cache=true}in_languages_t:de", "{!cost=99 > cache=false}(shipping_country_codes_mt: (DE OR EURO OR EUR OR ALL)) AND > (cents_ri: [* TO 3000])" ], "parsed_filter_queries": [

Re: slow queries

gt;false >> > >> > >> > >> > >> >${solr.autoSoftCommit.maxTime:60} >> > >> > >> > and the following stats for filters: >> > >> > lookups = 3602 >> > hi

Re: slow queries

$2@169ee0fd) > > > > > > > >${solr.autoCommit.maxTime:15000} > >false > > > > > > > > > >${solr.autoSoftCommit.maxTime:60} > > > > > > and the following stats for filters: > > > > loo

Re: slow queries

ND is_valid_b:true", "{!cost=50 cache=true}in_languages_t:de", "{!cost=99 cache=false}(shipping_country_codes_mt: (DE OR EURO OR EUR OR ALL)) AND (cents_ri: [* TO 3000])" ], "parsed_filter_queries": [ "+type_s:Product +is_valid_b:true", "in_langua

Re: slow queries

utoCommit.maxTime:15000} >false > > > > >${solr.autoSoftCommit.maxTime:60} > > > and the following stats for filters: > > lookups = 3602 > hits = 3148 > hit ratio = 0.87 > inserts = 455 > evictions = 400 > size = 63 > warmupTime = 770 > > *Problem: *a lot of

Re: slow queries

lr.autoCommit.maxTime:15000} >false > > > > >${solr.autoSoftCommit.maxTime:60} > > > and the following stats for filters: > > lookups = 3602 > hits = 3148 > hit ratio = 0.87 > inserts = 455 > evictions = 400

Re: slow queries

ely to be a magic bullet. But are these slow queries constant or intermittent? In other words, are all queries of this general form slow or just the first few? In particular is the first query that mentions sorting on this field slow but subsequent ones faster? In that case consider adding a

slow queries

) ${solr.autoCommit.maxTime:15000} false ${solr.autoSoftCommit.maxTime:60} and the following stats for filters: lookups = 3602 hits = 3148 hit ratio = 0.87 inserts = 455 evictions = 400 size = 63 warmupTime = 770 *Problem: *a lot of slow queries, for example: {q=*:*&

Re: Slow queries

rformance, > SHould I use an independant tomcat engine? > > rgds, > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Slow-queries-tp4172032p4173092.html > Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slow queries

, -- View this message in context: http://lucene.472066.n3.nabble.com/Slow-queries-tp4172032p4173092.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slow queries

ne? > Is there any thing else that can be done to the solr instance for example > deviding the collection? > > rgds, > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Slow-queries-tp4172032p4172039.html > Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slow queries

; deviding the collection? > > rgds, > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Slow-queries-tp4172032p4172039.html > Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slow queries

this message in context: http://lucene.472066.n3.nabble.com/Slow-queries-tp4172032p4172039.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slow queries

solution? regards, -- View this message in context: http://lucene.472066.n3.nabble.com/Slow-queries-tp4172032.html Sent from the Solr - User mailing list archive at Nabble.com.

Slow queries

optimize the collection and speed up querying it is it normal with this volume of data? is sharding is a good solution? regards, -- View this message in context: http://lucene.472066.n3.nabble.com/Slow-queries-tp4172032.html Sent from the Solr - User mailing list archive at Nabble.com.

RE: Slow queries for common terms

mailto:erickerick...@gmail.com] Sent: Monday, March 25, 2013 10:04 PM To: solr-user@lucene.apache.org Subject: Re: Slow queries for common terms take a look here: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html looking at memory consumption can be a bit tricky to interpret wi

Re: Slow queries for common terms

ce when I can give it say 10GB of RAM? > > Thanks, this has been tremendously helpful. > > David > > > -Original Message- > From: Tom Burton-West [mailto:tburt...@umich.edu] > Sent: Saturday, March 23, 2013 1:38 AM > To: solr-user@lucene.apache.org > Su

RE: Slow queries for common terms

Thanks, this has been tremendously helpful. David -Original Message- From: Tom Burton-West [mailto:tburt...@umich.edu] Sent: Saturday, March 23, 2013 1:38 AM To: solr-user@lucene.apache.org Subject: Re: Slow queries for common terms Hi David and Jan, I wrote the blog post, and David, you are

Re: Slow queries for common terms

Hi David and Jan, I wrote the blog post, and David, you are right, the problem we had was with phrase queries because our positions lists are so huge. Boolean queries don't need to read the positions lists. I think you need to determine whether you are CPU bound or I/O bound.It is possible

Re: Slow queries for common terms

proper load testing. Right now I'm just trying to get it to "work" > for a few users. If you could elaborate a bit on your thinking I'd be quite > grateful. > > David > > > -Original Message- > From: Jan Høydahl [mailto:jan....@cominvent.com] >

RE: Slow queries for common terms

to "work" for a few users. If you could elaborate a bit on your thinking I'd be quite grateful. David -Original Message- From: Jan Høydahl [mailto:jan@cominvent.com] Sent: Thursday, March 21, 2013 8:01 PM To: solr-user@lucene.apache.org Subject: Re: Slow queries for

Re: Slow queries for common terms

Hi David, I happen to know CommonTermsQuery added recently. But now sure how to use it. http://lucene.apache.org/core/4_1_0/queries/org/apache/lucene/queries/CommonTermsQuery.html --- On Thu, 3/21/13, David Parks wrote: > From: David Parks > Subject: Slow queries for common term

Slow queries for common terms

I've got a query that takes 15 seconds to return whenever I have the term "book" in a query that isn't cached. That's a pretty common term in our search index. We're indexing about 120 GB of text data. We only store terms and IDs, no document data, and the disk is virtually unused, it's all CPU

Re: Slow queries for common terms

;category book" is getting included, which seems logical and correct. > > > > -Original Message----- > From: Jan Høydahl [mailto:jan@cominvent.com] > Sent: Thursday, March 21, 2013 5:43 PM > To: solr-user@lucene.apache.org > Subject: Re: Slow queries for common terms >

RE: Slow queries for common terms

common phrase "category book" is getting included, which seems logical and correct. -Original Message- From: Jan Høydahl [mailto:jan@cominvent.com] Sent: Thursday, March 21, 2013 5:43 PM To: solr-user@lucene.apache.org Subject: Re: Slow queries for common terms Hi, I

Re: Slow queries for common terms

Hi, I think you can start by reading this blog http://www.hathitrust.org/blogs/large-scale-search/slow-queries-and-common-words-part-2 and try out the approach using a dictionary of the most common words in your index. You don't say how many documents, avg. doc size, the IDF value of

Slow queries for common terms

I've got a query that takes 15 seconds to return whenever I have the term "book" in a query that isn't cached. That's a pretty common term in our search index. We're indexing about 120 GB of text data. We only store terms and IDs, no document data, and the disk is virtually unused, it's all CPU tim

Re: Dropping slow queries

low search in the shard, network issues between the shards, etc. thanks -- View this message in context: http://lucene.472066.n3.nabble.com/Dropping-slow-queries-tp4043074p4043592.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Dropping slow queries

Am I missing something? which answer do you mean? And actually, just to clarify, the question was how can I force the partial results and drop the results from the slow shard? -- View this message in context: http://lucene.472066.n3.nabble.com/Dropping-slow-queries-tp4043074p4043549.html Sent

Re: Dropping slow queries

Where Influence Isn’t a Game On Wed, Feb 27, 2013 at 9:29 AM, adm1n wrote: > Anybody? > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Dropping-slow-queries-tp4043074p4043387.html > Sent from the Solr - User mailing list archive at Nabble.com.

Dropping slow queries

this message in context: http://lucene.472066.n3.nabble.com/Dropping-slow-queries-tp4043074.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Very slow queries

: > Hello everyone, > > All of a sudden, I am experiencing some very slow queries with solr. I have > 13GB of indexed documents, each averaging 50-100kb. They have an id key, so I > expect to be getting results really fast if I execute > id:7cd6cb99fd239c1d743a51bb85a48f790f4a6d3c as

Very slow queries

Hello everyone, All of a sudden, I am experiencing some very slow queries with solr. I have 13GB of indexed documents, each averaging 50-100kb. They have an id key, so I expect to be getting results really fast if I execute id:7cd6cb99fd239c1d743a51bb85a48f790f4a6d3c as the query with no other