Hi everyone,
We had an OOM event earlier this morning. This has caused one of our shards
to lose all it's replicas and it's leader is still in a down state. We have
restarted the Java process (solr) and it's still in a down state. Logs
below:
```
Feb 25, 2021 @ 11:46:43.000 2
SinceI changed heap size to 10G, I found that solr always uses around
6G-6.5G. Just wondering where I can set to limit memory usage, for example,
I just want to give solr 6G.
On Sun, Jan 24, 2021 at 1:51 PM Luke wrote:
> looks like the solr-8983-console.log was overridden after I restarted Solr
looks like the solr-8983-console.log was overridden after I restarted Solr
with 10G memory, I cannot find it anymore.
as for how I install and start solr, I did as below
1. download binary file(8.7.0)
2. change configuration in solr.in.sh(setup external zk)
3. start it by ./bin/solr start &
Thank
On 1/23/2021 6:41 PM, Luke wrote:
I don't see any log in solr.log, but there is OutOfMemory error in
solr-8983-console.log file.
Do you have the entire text of that exception? Can you share it? That
is the real information that I am after here.
I only asked how Solr was installed and start
ctions with 1node and 1 replica, however, there is not much data at
> all, just 100 documents.
> >
> > My server is 32 G memory and 4 core cpu, ssd drive 300g
> >
> > It was ok when i created 5 collections. It got oom killed when 10
> collections are created. Please
collections. It got oom killed when 10 collections
are created. Please, no data in new collections.
What version of Solr? How is it installed and started? What OS? What
Java version?
Do you have the actual OutOfMemoryError text? If I remember correctly
from my own reading, there are eight
Hi there,
I use default settings to start solr , I set heap to 6G, I created 10
collections with 1node and 1 replica, however, there is not much data at all,
just 100 documents.
My server is 32 G memory and 4 core cpu, ssd drive 300g
It was ok when i created 5 collections. It got oom killed
ckson
wrote: A sort on anything can cause an OOM… That said, _all_ fields defined
in your Solr schema should have docValues set to true if you sort, group, use
function queries or facet on them. What’s happening is the docValues structure
is being synthesized at runtime on the heap. In recent
A sort on anything can cause an OOM… That said, _all_ fields defined in your
Solr schema should have docValues set to true if you sort, group, use function
queries or facet on them. What’s happening is the docValues structure is being
synthesized at runtime on the heap. In recent Solr releases
y query in which there is any
sort on id field. But instead I find few queries in which we are sorting on
_docid_ lucene internal id.
Can sort on _docid_ cause OOM?
And if I will enable docValues for uniqueKey(id) will that solve my problem?
Regards,SanjaySent from Yahoo Mail on Android
;
> Hi Isabelle
> Thanks for your input.
> In fact SolR returns 30 results out of this queries. Why does it behave in a
> way that causes OOM ? Also the commands, they are SQL commands and solr would
> parse it as normal character …
>
> Thanks
>
>
> > On 10 Jun 2020,
Hi Isabelle
Thanks for your input.
In fact SolR returns 30 results out of this queries. Why does it behave in a
way that causes OOM ? Also the commands, they are SQL commands and solr would
parse it as normal character …
Thanks
> On 10 Jun 2020, at 22:50, Isabelle Giguere
> wrote:
&
solr-user@lucene.apache.org
Objet : [EXTERNAL] - SolR OOM error due to query injection
Hi,
Environment: SolR 6.6.2, with org.apache.solr.solr-core:6.1.0. This setup has
been running for at least 4 years without having OutOfMemory error. (it is
never too late for an OOM…)
This week, our searc
Hi,
Environment: SolR 6.6.2, with org.apache.solr.solr-core:6.1.0. This setup has
been running for at least 4 years without having OutOfMemory error. (it is
never too late for an OOM…)
This week, our search tool has been attacked via ‘sql injection’ like, and that
led to an OOM. These
e heap dump
> taken within hour before (we have hourly heap generation) the OOM did not
> have more than 150 to 160 threads. So it doesn't look like it happens due
> to running out of threads. Rather suspecting it happens because there is no
> native memory?.
>
> Thanks,
&
Thanks for your reply . Sure will take a look at the docker host log. But
even when we got "unable to create new native thread" error , the heap dump
taken within hour before (we have hourly heap generation) the OOM did not
have more than 150 to 160 threads. So it doesn't look
imes no
> exceptions .
> When it says "unable to create native thread" error , we got below
> exceptions as we use cdcr. To eliminate cdcr from this issue , we disabled
> CDCR also. But we still get OOM.
>
> WARN (cdcr-update-log-synchronizer-93-thread-1) [
, we got below
exceptions as we use cdcr. To eliminate cdcr from this issue , we disabled
CDCR also. But we still get OOM.
WARN (cdcr-update-log-synchronizer-93-thread-1) [ ]
o.a.s.h.CdcrUpdateLogSynchronizer Caught unexpected exception
java.lang.OutOfMemoryError: unable to create new native
Raji, how that "OOM for solr occur in every 5 days." exactly looks like?
What is the error message? Where it's occurring exactly?
On Thu, Apr 30, 2020 at 1:30 AM Raji N wrote:
> Thanks so much Jan. Will try your suggestions , yes we are also running
> solr inside docke
lly the index
> files grow large, and there is something strange going on in the way Docker
> handles this, leading to OOM, not for Java heap but for the process.
>
> I have no definitive answer, but so far my research has found a few
> possible settings
>
> Set
unloading of the index files, and Solr will access them as if they
were in a huge virtual memory pool. Naturally the index files grow large, and
there is something strange going on in the way Docker handles this, leading to
OOM, not for Java heap but for the process.
I have no definitive answer
What does the message look like, exactly, from solr.log ?
On Wed, Apr 29, 2020 at 1:27 PM Raji N wrote:
>
> Thank you for your reply. When OOM happens somehow it doesn't generate
> dump file. So we have hourly heaps running to diagnose this issue. Heap is
> around 700MB and t
Thank you for your reply. When OOM happens somehow it doesn't generate
dump file. So we have hourly heaps running to diagnose this issue. Heap is
around 700MB and threads around 150. But 29GB of native memory is used up,
it is consumed by java.io.DirectBufferR (27GB major consumption
On 4/29/2020 2:07 AM, Raji N wrote:
Has anyone encountered off-heap OOM. We are thinking of reducing heap
further and increasing the hardcommit interval . Any other suggestions? .
Please share your thoughts.
It sounds like it's not heap memory that's running out.
When the OutOfMemo
leaves off-
heap as 25GB. We have set
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
max memory size (kbytes, -m) unlimited
OOM for solr occur in every 5 days. When we examined heapdumps
We have defined a “search feed” as a file of JSONL objects, one per line.
The feed files can be stored in S3, reloaded, sent to two clusters, etc.
Each destination can keep its own log of failures and retries. We’ve been
doing this for full batch feeds and incrementals for a few years.
We’ve been
One approach could be to buffer the messages in Kafka before pushing to
Solr.
And then use "Kafka mirror" to replicate the messages to the other DC.
Now both DCs' Kafka pipelines are in sync by the mirror and you can run
storm/spark/flink etc jobs to consume local Kafka and publish to local Solr
cl
Hi Eric,
What are you recommendations for SolrCloud DR strategy.
Thanks,
Raji
On Sun, Mar 29, 2020 at 6:25 PM Erick Erickson
wrote:
> I don’t recommend CDCR at this point, I think there better approaches.
>
> The root problem is that CDCR uses tlog files as a queueing mechanism.
> If the conne
Thanks Eric. I don't seeing anywhere that CDCR is not recommended for
production use. Took the thread dump. Seeing about 140 CDCR threads
cdcr-replicator-219-thread-8" #787 prio=5 os_prio=0 tid=0x7f7c34009000
nid=0x50a waiting on condition [0x7f7ec871b000]
java.lang.Thread.State: WAIT
I don’t recommend CDCR at this point, I think there better approaches.
The root problem is that CDCR uses tlog files as a queueing mechanism.
If the connection between the DCs is broken for any reason, the tlogs grow
without limit. This could probably be fixed, but a better alternative is to
use s
Is CDCR even recommended to be used in production?
Or it was abandoned before it could become production ready ?
Thanks
SG
On Sun, Mar 29, 2020 at 5:18 AM Erick Erickson
wrote:
> What that error usually means is that there are a zillion threads running.
>
> Try taking a thread dump. It’s _prob
What that error usually means is that there are a zillion threads running.
Try taking a thread dump. It’s _probable_ that it’s CDCR, but
take a look at the thread dump to see if you have lots of
threads that are running. Any by “lots” here, I mean 100s of threads
that reference the same component,
Hi All,
We running solrcloud 7.6 (with the patch #
https://issues.apache.org/jira/secure/attachment/12969150)/SOLR-11724.patchon
production on 7 hosts in containers. The container memory is 48GB , heap
is 24GB.
ulimit -v
unlimited
ulimit -m
unlimited
We don't have any custom code in solr. We
> cores with the same name) because someone did a backup of an existing one ...
>
> Thank you for your support!
> - Torsten
>
> -Ursprüngliche Nachricht-
> Von: Erick Erickson
> Gesendet: Freitag, 6. März 2020 16:22
> An: solr-user@lucene.apache.org
>
Nachricht-
Von: Erick Erickson
Gesendet: Freitag, 6. März 2020 16:22
An: solr-user@lucene.apache.org
Betreff: Re: Problem with Solr 7.7.2 after OOM
Is it still giving you OOMs? That was the original problem statement. If not,
then you need to look at your Solr logs to see what error is
; Gesendet: Freitag, 6. März 2020 09:33
> An: solr-user@lucene.apache.org
> Betreff: AW: Problem with Solr 7.7.2 after OOM
>
> I set the heap to 8g but this doesn't have any effect and the problem is
> still the same.
>
> ~# ps -eaf | grep solr
> solr
@lucene.apache.org
Betreff: AW: Problem with Solr 7.7.2 after OOM
I set the heap to 8g but this doesn't have any effect and the problem is still
the same.
~# ps -eaf | grep solr
solr 3176 1 0 08:50 ?00:00:00 /lib/systemd/systemd --user
solr 3177 3176 0
26388 62793615034128 796 764324
15488956
Swap:969960 0 969960
~#
Torsten
-Ursprüngliche Nachricht-
Von: Jörn Franke
Gesendet: Donnerstag, 5. März 2020 17:31
An: solr-user@lucene.apache.org
Betreff: Re: Problem with Solr 7.7.2 after OOM
Just keep in mind that the total memory should be much more than the heap to
leverage Solr file caches. If you have 8 GB heap probably at least 16 gb total
memory make sense to be available on the machine .
> Am 05.03.2020 um 16:58 schrieb Walter Underwood :
>
>
>>
>> On Mar 5, 2020, at 4:29
> On Mar 5, 2020, at 4:29 AM, Bunde Torsten wrote:
>
> -Xms512m -Xmx512m
Your heap is too small. Set this to -Xms8g -Xmx8g
In solr.in.sh, that looks like this:
SOLR_HEAP=8g
wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/ (my blog)
Hello @all,
I've got a problem with Solr (version 7.7.2) since some days.
On March, 4th I got an OOM and the log file just says the following:
Running OOM killer script for process 12351 for Solr on port 8983
Killed process 12351
So I started the service again and the service is running
Hi,
We are running another 24 hour test with 8GB JVM and so far it is also
> running flawlessly.
If this is the case, as Erick mentioned, the failures were probably due to
long GC pauses. During couple of my stress testings, I had found that
decreasing JVM helps sometimes (it makes GC more frequ
Yes 18% of total physical RAM. The failures in G1GC and CMS setup did seem to
be from pause the world.
We are using Solr Docker image which is using G1GC by default and we tuned
with G1GC. Even with tuning the performance test failed after about 8 hours.
With ZGC we had consistent 12 and 24 hour p
People are certainly interested. You’re running on the bleeding edge of
technology, you’re very brave ;).
I’m not quite sure how to interpret “memory utilization stays around 18%”.
18% of total physical RAM or heap? I’m assuming the former..
I’m curious, how did CMS and G1GC fail? It’s perfectly
We are currently running performance tests with Solr 8.2/OpenJDK11/ZGC. We've
ran multiple successful 12 hour tests and are currently running 24 hour
tests. There are three nodes which are 4 cores and 28GB memory, JVM is 16GB.
We are getting max ~780 Page Per Second with max of ~8,000 users/min. CP
Robert:
My concern with fixing by adding memory is that it may just be kicking the can
down the road. Assuming there really is some leak eventually they’ll accumulate
and you’ll hit another OOM. If that were the case, I’d expect a cursory look at
your memory usage to just keep increasing over
ough while the re-index is happening.
>
> I've managed to work around the issue on my dev box by upping the the memory
> for solr to 16G, and haven't had an OOM since doing that, but I'm hesitant to
> push these changes to our AWS-hosted production instances since
zero searches
coming through while the re-index is happening.
I've managed to work around the issue on my dev box by upping the the memory
for solr to 16G, and haven't had an OOM since doing that, but I'm hesitant to
push these changes to our AWS-hosted production instances since
How many fields do you wind up having? It looks on a quick glance like
it depends on the values of fields. While I’ve seen Solr/Lucene handle
indexes with over 1M different fields, it’s unsatisfactory.
What I’m wondering is if you are adding a zillion different fields to your
docs as time passes a
above, and called the scripts from
the updateRequestProcessorChain and tested, and everything seemed great.
However when I ran the bulk of our 9 million records through the indexing
process, solr would repeatedly, unceremoniously throw a OOM error and
terminate. Usua
en log entries for Full GC attempts, but the JVM crashes with OOM long
before the Full GC could do anything.
The goal for good GC tuning is to avoid full GCs ever being needed. It
cannot be prevented entirely, especially when humongous allocations are
involved ... but a well-tuned GC should not
answer your previous questions:
1. We run completely stock Solr, not custom code, no plugins.
Regardless, we never had such OOMs with Solr 4.x or Solr 6.x
2. It seems that Full GC is never triggered. In some cases in the past
I've seen log entries for Full GC attempts, but the JV
On 10/14/2019 7:18 AM, Vassil Velichkov (Sensika) wrote:
After the migration from 6.x to 7.6 we kept the default GC for a couple of
weeks, than we've started experimenting with G1 and we've managed to achieve
less frequent OOM crashes, but not by much.
Changing your GC settings
Hi Shawn,
My answers are in-line below...
Cheers,
Vassil
-Original Message-
From: Shawn Heisey
Sent: Monday, October 14, 2019 3:56 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr 7.6 frequent OOM with Java 9, G1 and large heap sizes - any
tests with Java 13 and the new ZGC?
On
t all shards will be rebalanced (we've added 6 more) and will
contain up to 100-120M documents (14.31MB + overhead should be < 16MB), so
hopefully this will help us to alleviate the OOM crashes.
It doesn't sound to me like your filterCache can cause OOM. The total
size of 256 filte
me time tonight all shards will be rebalanced (we've added 6 more) and
will contain up to 100-120M documents (14.31MB + overhead should be < 16MB), so
hopefully this will help us to alleviate the OOM crashes.
Cheers,
Vassil
-Original Message-
From: Erick Erickson
Sent: Monday
tios between JVM Heap / OS RAM (up to 128GB /
> 256GB) and we have the same Java Heap OOM crashes.
> For example, a BitSet of 160M documents is > 16MB and when we look at the G1
> logs, it seems it never discards the humongous allocations, so they keep
> piling. Forcing a full
Hi Everyone,
Since we’ve upgraded our cluster (legacy sharding) from Solr 6.x to Solr 7.6 we
have frequent OOM crashes on specific nodes.
All investigations (detailed below) lead to a hard-coded limitation in the G1
garbage collector and the Java Heap is exhausted due to too many filterCache
Thanks Jörn,
Yep, we are rebalancing the cluster to keep up to ~100M documents per shard,
but that's not quite optimal in our use-case.
We've tried with various ratios between JVM Heap / OS RAM (up to 128GB / 256GB)
and we have the same Java Heap OOM crashes.
For example, a BitS
Hi Everyone,
>
> Since we’ve upgraded our cluster (legacy sharding) from Solr 6.x to Solr 7.6
> we have frequent OOM crashes on specific nodes.
>
> All investigations (detailed below) lead to a hard-coded limitation in the G1
> garbage collector. The Java Heap is exhaust
Hi Everyone,
Since we’ve upgraded our cluster (legacy sharding) from Solr 6.x to Solr 7.6 we
have frequent OOM crashes on specific nodes.
All investigations (detailed below) lead to a hard-coded limitation in the G1
garbage collector. The Java Heap is exhausted due to too many filterCache
Recently we had a few Japanese queries that killed our production Solrcloud
instance. Our schemas support multiple languages, with language specific search
fields.
This query and similar ones caused OOM errors in Solr:
モノクローナル抗ニコチン性アセチルコリンレセプター(??7サブユニット)抗体 マウス宿主抗体
The query doesn’t match
On Mon, 2018-09-17 at 17:52 +0200, Vincenzo D'Amore wrote:
> org.apache.solr.common.SolrException: Error while processing facet
> fields:
> java.lang.OutOfMemoryError: Java heap space
>
> Here the complete stacktrace:
> https://gist.github.com/freedev/a14aa9e6ae33fc3ddb2f02d602b34e2b
>
> I suppos
On 9/17/2018 9:52 AM, Vincenzo D'Amore wrote:
recently I had few Java OOM in my Solr 4.8.1 instance.
Here the configuration I have.
The only part of your commandline options that matters for OOM is the
max heap. Which is 16GB for your server. Note, you should set the min
heap and max
Hi there,
recently I had few Java OOM in my Solr 4.8.1 instance.
Here the configuration I have.
-Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Dsolr.log=/opt/tomcat/logs
-DzkHost=ep-1:2181,ep-2:2181,ep-3
Has anyone ever been successful in processing 150M records using the
Suggester Component? The make of the component, please comment.
On Tue, Jun 26, 2018 at 1:37 AM, Ratnadeep Rakshit
wrote:
> The site_address field has all the address of United states. Idea is to
> build something similar to Go
The site_address field has all the address of United states. Idea is to
build something similar to Google Places autosuggest.
Here's an example query: curl "
http://localhost/solr/addressbook/suggest?suggest.q=1054%20club&wt=json";
Response:
{
"responseHeader": {
"status": 0,
"QTime": 3125,
"par
I didn't get any answer to my questions ( unless you meant you have 25
millions of different values for those fields ...)
Please read again my answer and elaborate further.
Do you problem happen for the 2 different suggesters ?
Cheers
-
---
Alessandro Benedetti
Search Consultant
Anyone from the Solr team who can shed some more light?
On Tue, Jun 12, 2018 at 8:13 PM, Ratnadeep Rakshit
wrote:
> I observed that the build works if the data size is below 25M. The moment
> the records go beyond that, this OOM error shows up. Solar itself shows 56%
> usage of 2
I observed that the build works if the data size is below 25M. The moment
the records go beyond that, this OOM error shows up. Solar itself shows 56%
usage of 20GB space during the build. So, is there some settings I need to
change to handle larger data size?
On Tue, Jun 12, 2018 at 3:17 PM
Hi,
first of all the two different suggesters you are using are based on
different data structures ( with different memory utilisation) :
- FuzzyLookupFactory -> FST ( in memory and stored binary on disk)
- AnalyzingInfixLookupFactory -> Auxiliary Lucene Index
Both the data structures should be v
gt;
>> >
>> >
>> > mySuggester1
>> > FuzzyLookupFactory
>> > suggester_fuzzy_dir
>> >
>> >
>> >
>> > DocumentDictionaryFactory
>> > site_address
>> > suggestType
>> > property_metadata
>> > false
>> > false
>> >
>> >
>> > mySuggester2
>> > AnalyzingInfixLookupFactory
>> > suggester_infix_dir
>> >
>> > DocumentDictionaryFactory
>> > site_address_other
>> > suggestType
>> > property_metadata
>> > false
>> > false
>> >
>> >
>> >
>> > The handler is defined like so -
>> >
>> > > startup="lazy" >
>> >
>> > true
>> > 10
>> > mySuggester1
>> > mySuggester2
>> > false
>> > explicit
>> >
>> >
>> > suggest
>> >
>> >
>> >
>> > *Problem Statement*
>> >
>> > Every time I try to build the suggest index using the suggest.build=true
>> > url parameter, I end up with an OutOfMemory error. I have no clue how I
>> can
>> > make this work with the current setup. Can anyone explain why this is
>> > happening? And how can I fix this issue?
>> > *StackOverflow:*
>> > https://stackoverflow.com/questions/50802122/solr-suggest-
>> component-and-outofmemory-error
>> >
>>
>> Can you explain the nature of the OOM? Not all OOMs are due to heap
>> exhaustion...
>>
>> -chris
>>
>>
>>
>
> true
> > 10
> > mySuggester1
> > mySuggester2
> > false
> > explicit
> >
> >
> > suggest
> >
> >
> >
> > *Problem Statement*
> >
> > Every time I try to build the suggest index using the sug
d the suggest index using the suggest.build=true
> url parameter, I end up with an OutOfMemory error. I have no clue how I can
> make this work with the current setup. Can anyone explain why this is
> happening? And how can I fix this issue?
> *StackOverflow:*
> https://stackoverflow.com/questions/50802122/solr-suggest-component-and-outofmemory-error
>
Can you explain the nature of the OOM? Not all OOMs are due to heap
exhaustion...
-chris
signature.asc
Description: OpenPGP digital signature
I am using the Solr Suggester component in Solr 5.5 with a lot of address
data. My Machine has allotted 20Gb RAM for solr and the machine has 32GB
RAM in total.
I have an address book core with the following vitals -
"numDocs"=153242074
"segmentCount"=34
"size"=30.29 GB
My solrconfig.xml looks s
On 4/11/2018 9:23 AM, Adam Harrison-Fuller wrote:
> In addition, here is the GC log leading up to the crash.
>
> https://www.dropbox.com/s/sq09d6hbss9b5ov/solr_gc_log_20180410_1009.zip?dl=0
I pulled that log into the http://gceasy.io website. This is a REALLY
nice way to look at GC logs. I do sti
I'm going to share how I've debugged a similar OOM crash and solving it had
nothing to do with increasing heap.
https://risdenk.github.io/2017/12/18/ambari-infra-solr-ranger.html
This is specifically for Apache Ranger and how to fix it but you can treat
it just like any application
.@wunderwood.org
> >> http://observer.wunderwood.org/ (my blog)
> >>
> >> > On Apr 11, 2018, at 6:29 AM, Sujay Bawaskar
> >> wrote:
> >> >
> >> > What is directory factory defined in solrconfig.xml? Your JVM heap
> >> should
&g
Just as a side note, when Solr goes OOM and kills itself, and if you're
running HDFS, you are guaranteed to have write.lock files left over. If
you're running lots of shards/replicas, you may have many files that you
need to go into HDFS and delete before restarting.
-Joe
On 4/
> be tuned up with respect to that.
>> How solr is being use, is it more updates and less query or less updates
>> more queries?
>> What is OOM error? Is it frequent GC or Error 12?
>>
>> On Wed, Apr 11, 2018 at 6:05 PM, Adam Harrison-Fuller <
>> aharrison-fu
ned in solrconfig.xml? Your JVM heap
>> should
>> > be tuned up with respect to that.
>> > How solr is being use, is it more updates and less query or less
>> updates
>> > more queries?
>> > What is OOM error? Is it frequent GC or Error 12?
&
h respect to that.
> > How solr is being use, is it more updates and less query or less updates
> > more queries?
> > What is OOM error? Is it frequent GC or Error 12?
> >
> > On Wed, Apr 11, 2018 at 6:05 PM, Adam Harrison-Fuller <
> > aharrison-ful...@mintel.co
)
> On Apr 11, 2018, at 6:29 AM, Sujay Bawaskar wrote:
>
> What is directory factory defined in solrconfig.xml? Your JVM heap should
> be tuned up with respect to that.
> How solr is being use, is it more updates and less query or less updates
> more queries?
> What is OOM err
Our Solr cloud nodes are having issues throwing OOM exceptions under load.
> This issue has only started manifesting itself over the last few months
> during which time the only change I can discern is an increase in index
> size. They are running Solr 5.5.2 on OpenJDK version "1.8.0_
What is directory factory defined in solrconfig.xml? Your JVM heap should
be tuned up with respect to that.
How solr is being use, is it more updates and less query or less updates
more queries?
What is OOM error? Is it frequent GC or Error 12?
On Wed, Apr 11, 2018 at 6:05 PM, Adam Harrison
please?
>>
>> Regards
>>
>>
>> 2018-04-11 12:01 GMT+02:00 Adam Harrison-Fuller <
>> aharrison-ful...@mintel.com
>>> :
>>
>>> Hey all,
>>>
>>> I was wondering if I could get some JVM/GC tuning advice to resolve an
some JVM/GC tuning advice to resolve an
> > issue that we are experiencing.
> >
> > Full disclaimer, I am in no way a JVM/Solr expert so any advice you can
> > render would be greatly appreciated.
> >
> > Our Solr cloud nodes are having issues throwing OOM excep
experiencing.
>
> Full disclaimer, I am in no way a JVM/Solr expert so any advice you can
> render would be greatly appreciated.
>
> Our Solr cloud nodes are having issues throwing OOM exceptions under load.
> This issue has only started manifesting itself over the last few month
Hey all,
I was wondering if I could get some JVM/GC tuning advice to resolve an
issue that we are experiencing.
Full disclaimer, I am in no way a JVM/Solr expert so any advice you can
render would be greatly appreciated.
Our Solr cloud nodes are having issues throwing OOM exceptions under load
We put nginx servers in front of our three solr stand alone servers and
three node gallera cluster, it works very well and the amount of control it
gives you is really helpful.
On Tue, Dec 19, 2017 at 10:58 AM, Walter Underwood
wrote:
> > On Dec 19, 2017, at 7:38 AM, Toke Eskildsen wrote:
> >
>
> On Dec 19, 2017, at 7:38 AM, Toke Eskildsen wrote:
>
> Let's say we change Solr, so that it does not re-issue queries that
> caused nodes to fail. Unfortunately that does not solve your problem as
> the user will do what users do on an internal server error: Press
> reload.
That would work, be
e the Solr cloud to maintain
a blacklist of queries that causes nodes to fail. But if it is paging
related, the user might try pressing "next" instead and then the query
will be different from the previous one, but still cause OOM. So maybe
a mechanism for detecting multiple OOM-trigge
Hi Susheel,
If a single query can cause node to fail and if retry cause replicas to be
affected (still to be confirmed) then preventing retry logic on Solr side can
only partially solve that issue - retry logic can exist on client side and it
will result in replicas’ OOM. Again, not sure if
UNSUBSCRIBE
On Mon, Dec 18, 2017 at 12:57 PM Susheel Kumar
wrote:
> Technically I agree Shawn with you on fixing OOME cause, Infact it is not
> an issue any more but I was testing for HA when planing for any failures.
> Same time it's hard to convince Business folks that HA wouldn't be there in
Technically I agree Shawn with you on fixing OOME cause, Infact it is not
an issue any more but I was testing for HA when planing for any failures.
Same time it's hard to convince Business folks that HA wouldn't be there in
case of OOME.
I think the best option is to enable timeAllowed for now.
T
On 12/18/2017 9:01 AM, Susheel Kumar wrote:
> Any thoughts on how one can provide HA in these situations.
As I have said already a couple of times today on other threads, there
are *exactly* two ways to deal with OOME. No other solution is possible.
1) Configure the system to allow the process t
Shawn/Emir - its the Java heap space issue. I can see in GCViewer sudden
heap utilization and finally Full GC lines and oom killer script killing
the solr.
What I wonder is if there is retry from coordinating node which is causing
this OOM query to spread to next set of replica's then how c
Susheel Kumar wrote:
>
> Yes, Emir. If I repeat the query, it will spread to other nodes but that's
> not the case. This is my test env and i am deliberately executing the
> query with very high offset and wildcard to cause OOM but executing only
> one time.
>
> So it
On 12/18/2017 7:36 AM, Susheel Kumar wrote:
Yes, Emir. If I repeat the query, it will spread to other nodes but that's
not the case. This is my test env and i am deliberately executing the
query with very high offset and wildcard to cause OOM but executing only
one time.
So it shou
Yes, Emir. If I repeat the query, it will spread to other nodes but that's
not the case. This is my test env and i am deliberately executing the
query with very high offset and wildcard to cause OOM but executing only
one time.
So it shouldn't spread to other replica sets and at the
1 - 100 of 457 matches
Mail list logo
