Hey all, Update to this thread in case others have a similar issue. It IS the setting below it appears. This is listed as a "Potential" tuning setting. It is basically required for any real workload. I saw this after pulling out some of last hairs I have left on my head (in messages, which I was not even focused on, Doh!).
Feb 15 08:38:33 puppetserver: OpenJDK 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled. Feb 15 08:38:33 puppetserver: OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize= Feb 15 08:38:33 puppetserver: CodeCache: size=245760Kb used=207748Kb max_used=242118Kb free=38011Kb Feb 15 08:38:33 puppetserver: bounds [0x00007f4394be4000, 0x00007f43a3be4000, 0x00007f43a3be4000] Feb 15 08:38:33 puppetserver: total_blobs=29412 nmethods=28684 adapters=635 Feb 15 08:38:33 puppetserver: compilation: disabled (not enough contiguous free space left) Boom goes the dynamite. This explains a great many things. I have just adjusted the settings and now need to wait and see if it is better. I will drop a final post to finish this thread if it works out. Mike On Tuesday, February 12, 2019 at 7:24:21 AM UTC-6, Mike Sharpton wrote: > > I just found this as well. > > If you're working outside of lab environment, increase ReservedCodeCache > to 512m under normal load. If you're working with 6-12 JRuby instances > (or a max-requests-per-instance value significantly less than 100k), run > with a ReservedCodeCache of 1G. Twelve or more JRuby instances in a > single server might require 2G or more. > > We need to look into this setting. > > On Monday, February 11, 2019 at 7:42:01 AM UTC-6, Mike Sharpton wrote: >> >> Hey all, >> >> We have recently upgraded our environment from Puppetserver 4.2.2 to >> Puppetserver 6.0.2. We are running a mix of Puppet 4 and Puppet 6 agents >> until we can get them all upgraded to 6. We have around 6000 nodes, and we >> had 4 Puppetservers, but we added two more due to capacity issues with >> Puppet 6. The load is MUCH higher with Puppet 6. To the question, I am >> seeing longer and longer agent run times after about two days of the >> services running. The only error in the logs that seems to have any >> relation to this is this string. >> >> 2019-02-11T04:32:28.409-06:00 ERROR [qtp1148783071-4075] [p.r.core] >> Internal Server Error: java.io.IOException: >> java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms >> >> >> After I restart the puppetserver service, this goes away for about two >> days. I think Puppetserver is dying a slow death under this load (load >> average of around 5-6). We are running Puppetserver on vm's that are >> 10X8GB and using 6 Jruby workers per Puppetserver and a 4GB heap. I have >> not seen any OOM exceptions and the process never crashes. Has anyone else >> seen anything like this? I did some Googling and didn't find a ton of >> relevant stuff. Perhaps we need to upgrade to the latest version to see if >> this helps? Even more capacity? Seems silly. Thanks in advance! >> >> Mike >> > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/f2aa8d37-609f-42ab-b04d-c497abfc699a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
