Total memory of server is 256 GB and in this server below application running Application1 50 GB Application2 30 GB Application3 8 GB Application4 2 GB Solr shard1 64 GB Solr shard2 replica 64 GB
Note: Solr shard2 and shard1 replica running on another server. Normally 35 to 40 GB memory constant usage in one solr instance so we keep the 64 GB. We are using NRT. How big are your indexes on disk? - [Shard1-115 Gb, shard2 replica-96 GB] [shard1 replica-114 GB, shard2-100GB] How many docs per replica? - Approx 30959714 docs How many replicas per host? - One server has one shard and one replica. Regards, Vishal Sent from Outlook<http://aka.ms/weboutlook> ________________________________ From: Erick Erickson <erickerick...@gmail.com> Sent: Friday, February 14, 2020 4:00 AM To: solr-user@lucene.apache.org <solr-user@lucene.apache.org> Subject: Re: Replica is going into recovery in Solr 6.1.0 What Walter said. Also, you _must_ leave quite a bit of free RAM for the OS due to Lucene using MMapDirectory space, see: https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html Basically until you can get your GC pauses under control, you’ll have an unstable collection. How big are your indexes on disk? How many docs per replica? How many replicas per host? Best, Erick > On Feb 13, 2020, at 5:16 PM, Walter Underwood <wun...@wunderwood.org> wrote: > > You have a 64GB heap. That is extremely unusual. You can only do that if the > instance has 80 GB or more of RAM. If you don’t have enough RAM, the JVM will > start using swap space and cause extremely long GC pauses. > > How much RAM do you have? > > How did you choose these GC settings? > > We have been using these settings with Java 8 in prod for three years with no > GC problems. > > SOLR_HEAP=8g > # Use G1 GC -- wunder 2017-01-23 > # Settings from https://wiki.apache.org/solr/ShawnHeisey > GC_TUNE=" \ > -XX:+UseG1GC \ > -XX:+ParallelRefProcEnabled \ > -XX:G1HeapRegionSize=8m \ > -XX:MaxGCPauseMillis=200 \ > -XX:+UseLargePages \ > -XX:+AggressiveOpts \ > “ > > If you don’t have a very, very good reason for your GC settings, use these > instead. > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > >> On Feb 12, 2020, at 10:47 PM, vishal patel <vishalpatel200...@outlook.com> >> wrote: >> >> My configuration: >> >> -XX:+AggressiveOpts -XX:ConcGCThreads=12 -XX:G1HeapRegionSize=33554432 >> -XX:G1ReservePercent=20 -XX:InitialHeapSize=68719476736 >> -XX:InitiatingHeapOccupancyPercent=10 -XX:+ManagementServer >> -XX:MaxHeapSize=68719476736 -XX:ParallelGCThreads=36 >> -XX:+ParallelRefProcEnabled -XX:PrintFLSStatistics=1 -XX:+PrintGC >> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps >> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC >> -XX:+PrintTenuringDistribution -XX:ThreadStackSize=256 -XX:+UseG1GC >> -XX:-UseLargePages -XX:-UseLargePagesIndividualAllocation >> -XX:+UseStringDeduplication >> >> Sent from Outlook<http://aka.ms/weboutlook> >> ________________________________ >> From: Rajdeep Sahoo <rajdeepsahoo2...@gmail.com> >> Sent: Thursday, February 13, 2020 10:03 AM >> To: solr-user@lucene.apache.org <solr-user@lucene.apache.org> >> Subject: Re: Replica is going into recovery in Solr 6.1.0 >> >> What is your memory configuration >> >> On Thu, 13 Feb, 2020, 9:46 AM vishal patel, <vishalpatel200...@outlook.com> >> wrote: >> >>> Is there anyone looking at this? >>> >>> Sent from Outlook<http://aka.ms/weboutlook> >>> ________________________________ >>> From: vishal patel <vishalpatel200...@outlook.com> >>> Sent: Wednesday, February 12, 2020 3:45 PM >>> To: solr-user@lucene.apache.org <solr-user@lucene.apache.org> >>> Subject: Replica is going into recovery in Solr 6.1.0 >>> >>> I am using solr version 6.1.0, Java 8 version and G1gc on production. We >>> have 2 shards and each shard has 1 replica. Suddenly one replica is going >>> into recovery mode and Requests become slow in our production. >>> I have analyzed that minor GC max pause time was 1 min 6 sec 800 ms on >>> that time and also multiple times minor GC pauses. >>> >>> My logs : >>> >>> https://drive.google.com/file/d/158z3nzLsnHGouyRnXgfzCjwD4iadgKSp/view?usp=sharing >>> >>> https://drive.google.com/file/d/1E4jyffvIWVJB7EeEMXBXyqaK2ZfAA8kk/view?usp=sharing >>> >>> I do not know why long GC pause time happened. In our platform heavy >>> searching and indexing is performed. >>> long GC pause times happen due to searching or indexing? >>> If GC pause time long then why replica is going into recovery? can we set >>> the waiting time of update request? >>> what is the minimum GC pause time for going into recovery mode? >>> >>> It is useful for my problem? : >>> https://issues.apache.org/jira/browse/SOLR-9310 >>> >>> Regards, >>> Vishal Patel >>> >>> Sent from Outlook<http://aka.ms/weboutlook> >>> >
