On Mon, Jun 29, 2020 at 1:49 PM David Hastings <hastings.recurs...@gmail.com> wrote:
> little nit picky note here, use 31gb, never 32. Good to know. Just now I got this output from bin/solr status: "solr_home":"/opt/solr/server/solr", "version":"7.7.2 d4c30fc2856154f2c1fefc589eb7cd070a415b94 - janhoy - 2019-05-28 23:37:48", "startTime":"2020-06-29T17:35:13.966Z", "uptime":"0 days, 1 hours, 32 minutes, 7 seconds", "memory":"9.3 GB (%57.9) of 16 GB"} That's the highest memory use I've seen. Not sure if this indicates 16GB isn't enough. Then I ran it again a couple minutes later and it was down to 598.3 MB. I wonder what accounts for these wide swings. I can't imagine if a few users are doing searches, suddenly it uses 9 GB of RAM. On Mon, Jun 29, 2020 at 1:45 PM Ryan W <rya...@gmail.com> wrote: > > > It figures it would happen again a couple hours after I suggested the > issue > > might be resolved. Just now, Solr stopped running. I cleared the cache > in > > my app a couple times around the time that it happened, so perhaps that > was > > somehow too taxing for the server. However, I've never allocated so much > > RAM to a website before, so it's odd that I'm getting these failures. My > > colleagues were astonished when I said people on the solr-user list were > > telling me I might need 32GB just for solr. > > > > I manage another project that uses Drupal + Solr, and we have a total of > > 8GB of RAM on that server and Solr never, ever stops. I've been managing > > that site for years and never seen a Solr outage. On that project, > > Drupal + Solr is OK with 8GB, but somehow this other project needs 64 GB > or > > more? > > > > "The thing that’s unsettling about this is that assuming you were hitting > > OOMs, and were running the OOM-killer script, you _should_ have had very > > clear evidence that that was the cause." > > > > How do I know if I'm running the OOM-killer script? > > > > Thank you. > > > > On Mon, Jun 29, 2020 at 12:12 PM Erick Erickson <erickerick...@gmail.com > > > > wrote: > > > > > The thing that’s unsettling about this is that assuming you were > hitting > > > OOMs, > > > and were running the OOM-killer script, you _should_ have had very > clear > > > evidence that that was the cause. > > > > > > If you were not running the killer script, the apologies for not asking > > > about that > > > in the first place. Java’s performance is unpredictable when OOMs > happen, > > > which is the point of the killer script: at least Solr stops rather > than > > do > > > something inexplicable. > > > > > > Best, > > > Erick > > > > > > > On Jun 29, 2020, at 11:52 AM, David Hastings < > > > hastings.recurs...@gmail.com> wrote: > > > > > > > > sometimes just throwing money/ram/ssd at the problem is just the best > > > > answer. > > > > > > > > On Mon, Jun 29, 2020 at 11:38 AM Ryan W <rya...@gmail.com> wrote: > > > > > > > >> Thanks everyone. Just to give an update on this issue, I bumped the > > RAM > > > >> available to Solr up to 16GB a couple weeks ago, and haven’t had any > > > >> problem since. > > > >> > > > >> > > > >> On Tue, Jun 16, 2020 at 1:00 PM David Hastings < > > > >> hastings.recurs...@gmail.com> > > > >> wrote: > > > >> > > > >>> me personally, around 290gb. as much as we could shove into them > > > >>> > > > >>> On Tue, Jun 16, 2020 at 12:44 PM Erick Erickson < > > > erickerick...@gmail.com > > > >>> > > > >>> wrote: > > > >>> > > > >>>> How much physical RAM? A rule of thumb is that you should allocate > > no > > > >>> more > > > >>>> than 25-50 percent of the total physical RAM to Solr. That's > > > >> cumulative, > > > >>>> i.e. the sum of the heap allocations across all your JVMs should > be > > > >> below > > > >>>> that percentage. See Uwe Schindler's mmapdirectiry blog... > > > >>>> > > > >>>> Shot in the dark... > > > >>>> > > > >>>> On Tue, Jun 16, 2020, 11:51 David Hastings < > > > >> hastings.recurs...@gmail.com > > > >>>> > > > >>>> wrote: > > > >>>> > > > >>>>> To add to this, i generally have solr start with this: > > > >>>>> -Xms31000m-Xmx31000m > > > >>>>> > > > >>>>> and the only other thing that runs on them are maria db gallera > > > >> cluster > > > >>>>> nodes that are not in use (aside from replication) > > > >>>>> > > > >>>>> the 31gb is not an accident either, you dont want 32gb. > > > >>>>> > > > >>>>> > > > >>>>> On Tue, Jun 16, 2020 at 11:26 AM Shawn Heisey < > apa...@elyograg.org > > > > > > >>>> wrote: > > > >>>>> > > > >>>>>> On 6/11/2020 11:52 AM, Ryan W wrote: > > > >>>>>>>> I will check "dmesg" first, to find out any hardware error > > > >>> message. > > > >>>>>> > > > >>>>>> <snip> > > > >>>>>> > > > >>>>>>> [1521232.781801] Out of memory: Kill process 117529 (httpd) > > > >> score 9 > > > >>>> or > > > >>>>>>> sacrifice child > > > >>>>>>> [1521232.782908] Killed process 117529 (httpd), UID 48, > > > >>>>>> total-vm:675824kB, > > > >>>>>>> anon-rss:181844kB, file-rss:0kB, shmem-rss:0kB > > > >>>>>>> > > > >>>>>>> Is this a relevant "Out of memory" message? Does this suggest > an > > > >>> OOM > > > >>>>>>> situation is the culprit? > > > >>>>>> > > > >>>>>> Because this was in the "dmesg" output, it indicates that it is > > the > > > >>>>>> operating system killing programs because the *system* doesn't > > have > > > >>> any > > > >>>>>> memory left. It wasn't Java that did this, and it wasn't Solr > > that > > > >>> was > > > >>>>>> killed. It very well could have been Solr that was killed at > > > >> another > > > >>>>>> time, though. > > > >>>>>> > > > >>>>>> The process that it killed this time is named httpd ... which is > > > >> most > > > >>>>>> likely the Apache webserver. Because the UID is 48, this is > > > >> probably > > > >>>> an > > > >>>>>> OS derived from Redhat, where the "apache" user has UID and GID > 48 > > > >> by > > > >>>>>> default. Apache with its default config can be VERY memory > hungry > > > >>> when > > > >>>>>> it gets busy. > > > >>>>>> > > > >>>>>>> -XX:InitialHeapSize=536870912 -XX:MaxHeapSize=536870912 > > > >>>>>> > > > >>>>>> This says that you started Solr with the default 512MB heap. > > Which > > > >>> is > > > >>>>>> VERY VERY small. The default is small so that Solr will start > on > > > >>>>>> virtually any hardware. Almost every user must increase the > heap > > > >>> size. > > > >>>>>> And because the OS is killing processes, it is likely that the > > > >> system > > > >>>>>> does not have enough memory installed for what you have running > on > > > >>> it. > > > >>>>>> > > > >>>>>> It is generally not a good idea to share the server hardware > > > >> between > > > >>>>>> Solr and other software, unless the system has a lot of spare > > > >>>> resources, > > > >>>>>> memory in particular. > > > >>>>>> > > > >>>>>> Thanks, > > > >>>>>> Shawn > > > >>>>>> > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > > > >
