Ok, so yea, it was a good samaritan updating all the puppet agents via satellite. No big deal on the clients, but that does break the server in a very annoying way.
Thank you for the assistance! On Thursday, November 9, 2023 at 12:23:05 PM UTC-8 James Millsap wrote: > Migrating/upgrading this whole stack is on my list. > > A newer version of the puppet agent was installed on this server right > before everything started failing. Main difference appears to be hiera > 3.12 vs the 3.4 that was part of the 5.x agent. No idea how that happened > yet, as no one was logged in and no remote execution was logged. I may > just restore the whole VM and call it a day. > > Just in case, here is the output of --trace > > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/rest.rb:36:in > > `rescue in find' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/rest.rb:7:in > > `find' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:223:in > > `find' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:725:in > `block in retrieve_new_catalog' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:567:in `block > in thinmark' > /opt/puppetlabs/puppet/lib/ruby/2.7.0/benchmark.rb:308:in `realtime' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:566:in > `thinmark' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:724:in > `retrieve_new_catalog' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:82:in > `retrieve_catalog' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:267:in > `prepare_and_retrieve_catalog' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:428:in > `run_internal' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:339:in > `run' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:83:in `block > (6 levels) in run' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in > `override' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:82:in `block > (5 levels) in run' > /opt/puppetlabs/puppet/lib/ruby/2.7.0/timeout.rb:95:in `block in timeout' > /opt/puppetlabs/puppet/lib/ruby/2.7.0/timeout.rb:105:in `timeout' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:81:in `block > (4 levels) in run' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent/locker.rb:21:in > `lock' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:71:in `block > (3 levels) in run' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:162:in > `with_client' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:67:in `block > (2 levels) in run' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:127:in > `run_in_fork' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:66:in `block > in run' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:172:in > `controlled_run' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:47:in `run' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:433:in > > `onetime' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:393:in > > `block in run_command' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in > `override' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:390:in > > `run_command' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:421:in > `block in run' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:756:in > `exit_on_fail' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:421:in > `run' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in > > `run' > /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in > `execute' > /opt/puppetlabs/puppet/bin/puppet:5:in `<main>' > > Warning: Not using cache on failed catalog > Error: Could not retrieve catalog; skipping run > > On Thursday, November 9, 2023 at 8:25:05 AM UTC-8 Martin Alfke wrote: > >> >> On 8. Nov 2023, at 20:17, James Millsap <[email protected]> wrote: >> >> Thank you very much for the reply! We are running these versions... >> >> puppetserver version: 5.3.11 >> ruby 2.0.0p648 >> psych (2.0.0) >> hiera 3.12.0 >> >> >> That is super outdated Puppet version. >> Can you please check the puppetserver log file. >> There should be a ruby stack trace which might give you a better idea on >> what is broken. >> Did you update any module recently? >> >> >> On Wednesday, November 8, 2023 at 11:01:16 AM UTC-8 ben.ford wrote: >> >>> Hey James! That error sounds like you have an out-of-date Psych YAML >>> gem. >>> >>> What versions of Puppet, Ruby, Hiera, and Psych are you using? >>> >>> >>> >>> On Nov 8, 2023, at 8:54 AM, James Millsap <[email protected]> wrote: >>> >>> We have a fairly mature puppet instance along with foreman for >>> classification. For some reason, yesterday, everything started failing >>> with this error: >>> >>> Info: Using environment 'production' >>> Info: Retrieving pluginfacts >>> Info: Retrieving plugin >>> Info: Loading facts >>> Error: Could not retrieve catalog from remote server: Error 500 on >>> SERVER: Server Error: Evaluation Error: Error while evaluating a Function >>> Call, undefined method `safe_load' for Puppet::Util::Yaml:Module (file: >>> /etc/puppetlabs/code/environments/production/manifests/site.pp, line: 3, >>> column: 1) on node hostname.company.com >>> Warning: Not using cache on failed catalog >>> Error: Could not retrieve catalog; skipping run >>> >>> Line 3 of the site.pp is just this: >>> hiera_include('classes', '') >>> >>> Which has not changed in many years. There is either a bug that >>> appeared from nowhere, or a syntax error buried somewhere. I have been >>> going through our repository and rolling back every yaml change, but so far >>> nothing has worked. If anyone has a clever way to get puppet to give me >>> some more information so I can find this, I would be eternally grateful. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/puppet-users/cb051ca4-92a5-43c7-a7d0-9e8942b0af73n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/puppet-users/cb051ca4-92a5-43c7-a7d0-9e8942b0af73n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> >>> *CAUTION:* This email originated from outside of the organization. Do >>> not click on links or open attachments unless you recognize the sender and >>> know the content is safe. >>> >>> >>> >>> This e-mail may contain information that is privileged or confidential. >>> If you are not the intended recipient, please delete the e-mail and any >>> attachments and notify us immediately. >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/puppet-users/bf1b9d65-aa35-4f09-a445-6e0bce2ede12n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/puppet-users/bf1b9d65-aa35-4f09-a445-6e0bce2ede12n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> >> -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/41abf246-fe52-47f3-9f7e-26344a9191f6n%40googlegroups.com.
