Justin, yes it's happening in all environments which leads me to believe it's related to an old copy in /opt/puppetlabs/puppet/cache/lib/puppet/type. Still trying to wrap my head around why one domain installation is fine and the other domain installation is not.
Here is the contents of that directory which works: [root@myserverlab type]# pwd /opt/puppetlabs/puppet/cache/lib/puppet/type [root@myserverlab type]# ls -al total 24 drwxr-xr-x 2 root root 77 Jul 16 16:04 . drwxr-xr-x 6 root root 61 Feb 3 2020 .. -rw-r--r-- 1 root root 1706 Jul 16 16:04 anchor.rb -rw-r--r-- 1 root root 6921 Jul 16 16:04 file_line.rb -rw-r--r-- 1 root root 1863 May 1 2017 httparch.rb -rw-r--r-- 1 root root 6957 Jul 13 17:04 httpfile.rb [root@myserverlab type]# Here is the contents of the directory that doesn't work: [root@myserverprod type]# pwd /opt/puppetlabs/puppet/cache/lib/puppet/type [root@myserverprod type]# ls -al total 24 drwxr-xr-x 2 root root 77 Sep 30 2018 . drwxr-xr-x 6 root root 61 Apr 24 2017 .. -rw-r--r-- 1 root root 1752 Sep 30 2018 anchor.rb -rw-r--r-- 1 root root 7113 Sep 30 2018 file_line.rb -rw-r--r-- 1 root root 1863 May 15 2017 httparch.rb -rw-r--r-- 1 root root 6357 Apr 24 2017 httpfile.rb [root@myserverprod type]# You can clearly see the date and size difference of httpfile.rb. I quadruple checked the puppet module directory on the prod server and the code does have the quick_check parm. For some reason it is just not refreshing the server cache. Both domains have a value of 0 for environment_timeout for each environment. On Friday, August 28, 2020 at 2:32:05 PM UTC-4 Justin Stoller wrote: > On Fri, Aug 28, 2020 at 10:14 AM [email protected] <[email protected]> > wrote: > >> Great info but I think I might have found the issue. >> >> So we don't use r10k to deploy code we use a different tool. But what I >> found is on the puppet server (master) the httpfile.rb in >> /opt/puppetlabs/puppet/cache/lib/puppet/type is the older version. >> > > I think puppet/cache is read by the agent not the server. I would expect > that to cause problems on applying a catalog from the server, not result in > a failed compilation. But barring a .resource_tyeps directory existing in > an environment it must be an incorrect version of the httpfile.rb in the > server's loadpath. > > >> I didn't find any ./resource_types directory in our environment >> directories so not sure if we are using environment isolation or not. >> > > Just to clarify it will be ".resource_types" with a leading dot and will > be hidden by default. [1] > > As part of Justin's suggestion to allow the DELETE option to be valid, I >> had to restart each of our 4 puppet servers so according to some of this >> conversation, that should have refreshed the cache right? >> > > Restarting or reloading will evict the in memory cache so if you have a > very long environment_timeout it will work as well as doing an eviction of > all your environments. It will not however remove any old files in your > .resources_types directory. You will need to run the `puppet generate > types` command with `--force` for that. > > >> >> What else is odd is the domain where the quick_check parm is work seems >> to be getting refreshed somehow in /opt/puppetlabs/puppet/cache/lib/puppet >> subdirectories (just looking at time stamps). The deploy process works the >> same in that domain along with the domain where quick_check is not working. >> > > Can you validate that the failures happen not along a "domain" but along > puppet environments. Like all the nodes that use httpfile in production > have this failure but those in staging don't have this issue? If you have > some succeeding and some failing I would expect this to be the environment > to be the condition causing the different behavior. > > >> Since the /opt/puppetlabs/bin/puppet generate types --environment >> production --force operates by environment, could this possible break the >> environment as well? These are production boxes I need to run this on and >> want to make sure I don't break anything. Also using the environment parm >> will this then setup environment isolation and do i have to manually manage >> that each time code is deployed to that environment? >> > > The environment param to `puppet generate types` specifies which > environment to act on, without it the command will only act on the > environment specified in the puppet.conf for the "main" section (The "main" > or "user" sections are almost always unmanaged and adopt the default values > which would be "production" for "environment" setting). > > Running the command should be a relatively safe command, however I'm going > to advocate for anyone "doing it live" on a production box. In PE we deploy > this code and run this command in a staging area and then either lock the > server while we copy the files over or atomically manage a symlink. If you > are using environment caching as well it should be even safer because types > will only be read from disk on the first compilation that uses them and > then cached in memory after that. > > hth, > justin > > 1. > https://puppet.com/docs/puppet/6.17/environment_isolation.html#env_generate_types > > > On Tuesday, August 25, 2020 at 5:09:28 PM UTC-4 Justin Stoller wrote: >> >>> > why wouldn't puppet just do this automatically when a module changes? >>> >>> Some background. Puppet's type and provider system modifies the running >>> Puppet instance when they're _loaded_. This causes issues when you try to >>> load multiple conflicting versions of a type in different environments. To >>> work around this we have a kind of header file for your types that Puppet >>> can read w/o actually loading the type. This way Puppet Server can load >>> multiple versions of a type (as long as those different versions are in >>> different environments) and check that each environment uses the type >>> correctly for that version. >>> >>> The command Dirk gave you, loads those types safely in a separate >>> process and then serializes their parameter information into a format for >>> Puppet to later read that doesn't corrupt its global state. It places this >>> information in the ".resource_types" directory at the root of your >>> environment (like "/etc/puppetlabs/code/environments/production") >>> >>> Also, in order to speed up Puppet Server catalog compilation, we attempt >>> to cache information like type parameters. >>> >>> In PE, if you use our built in code management facilities, we generate >>> this type information on every commit (if needed), distribute it to your >>> compilers, and then evict the environment cache so that any new information >>> will be read. >>> >>> In FOSS, r10k has a config setting to generate this info when it deploys >>> an environment [1]. >>> >>> >>> Now, the error you're ultimately getting involves Puppet Server thinking >>> that you're using the httpfile class wrong. It looks like the "quick_check" >>> field was added in the latest version. So really the first question would >>> be, are you using the latest version in this environment? >>> >>> Assuming you're doing that you probably either have the environment >>> cache containing an older version of the module (which should be resolved >>> by restarting the server or evicting the environment cache) or an old >>> .resource_types in the root of your environment that should be removed and >>> regenerated like Dirk said. Possibly you could have an older version in a >>> different environment that's being loaded first, but I don't think that'd >>> cause a problem for uncached, new parameters on a type. >>> >>> HTH, >>> Justin >>> >>> 1. >>> https://github.com/puppetlabs/r10k/blob/master/doc/dynamic-environments/configuration.mkd#generate_types >>> >>> On Tue, Aug 25, 2020 at 9:42 AM [email protected] <[email protected]> >>> wrote: >>> >>>> So a followup to the original question. >>>> >>>> As a test we created a simple module on the node which is failing when >>>> puppet agent executes. Running puppet apply, the parameter quick_check is >>>> found and the module completes successfully. So why would puppet apply >>>> work and not puppet agent? >>>> >>>> Code: >>>> >>>> class testmod() >>>> >>>> { >>>> >>>> httpfile { "ansible-2.8.0a1.tar.gz": >>>> >>>> ensure => present, >>>> >>>> path => "/u01/testmod/ansible-2.8.0a1.tar.gz", >>>> >>>> source => " >>>> https://mynexus.domain.com/nexus/repository/ae-raw-ansible-group/ansible/ansible-2.8.0a1.tar.gz >>>> >>>> <https://nexus.aetna.com/nexus/repository/ae-raw-ansible-group/ansible/ansible-2.8.0a1.tar.gz> >>>> ", >>>> >>>> quick_check => true, >>>> >>>> # hash => 'hex form SHA2 hash OR an URL to the .sha file with that >>>> hash' >>>> >>>> } >>>> >>>> } >>>> >>>> >>>> >>>> Here is my run: >>>> [root@node testmod]# puppet apply --modulepath=/home/toor --test -e >>>> "include testmod" --verbose >>>> >>>> On Tuesday, August 25, 2020 at 12:38:05 PM UTC-4 [email protected] >>>> wrote: >>>> >>>>> Dirk, why wouldn't puppet just do this automatically when a module >>>>> changes? Is there a bug somewhere? >>>>> >>>>> On Tuesday, August 25, 2020 at 2:43:03 AM UTC-4 Dirk Heinrichs wrote: >>>>> >>>>>> Am Montag, den 24.08.2020, 11:06 -0700 schrieb [email protected]: >>>>>> >>>>>> Justin, I implemented the suggestion you made however after running >>>>>> the curl command against the 2 environments having the issue and >>>>>> receiving >>>>>> the 204 response, the puppet module is still getting the 500 error. Do >>>>>> you >>>>>> or anyone else have any other suggestions? Is it possible it's related >>>>>> to >>>>>> ruby and/or java? Frankly I'm stumped. >>>>>> >>>>>> >>>>>> Didn't see this earlier, sorry. >>>>>> >>>>>> The "no parameter named 'xxx'" error can usually be resolved by >>>>>> recreating the metadata for your Puppet environment(s). This can be done >>>>>> on >>>>>> the Puppet master using the following command (for the production >>>>>> environment): >>>>>> >>>>>> /opt/puppetlabs/bin/puppet generate types --environment production >>>>>> --force >>>>>> >>>>>> I've added this command to my environment update script after running >>>>>> into this problem myself a few months ago after updating some external >>>>>> modules from the forge. >>>>>> >>>>>> See https://puppet.com/docs/puppet/5.5/environment_isolation.html for >>>>>> the details. >>>>>> >>>>>> HTH... >>>>>> >>>>>> Dirk >>>>>> >>>>>> -- >>>>>> >>>>>> *Dirk Heinrichs* >>>>>> Senior Systems Engineer, Delivery Pipeline >>>>>> OpenText ™ Discovery | Recommind >>>>>> *Phone*: +49 2226 15966 18 <+49%202226%201596618> >>>>>> *Email*: [email protected] >>>>>> *Website*: www.recommind.de >>>>>> Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach >>>>>> Vertretungsberechtigte Geschäftsführer Gordon Davies, Madhu >>>>>> Ranganathan, Christian Waida, Registergericht Amtsgericht Bonn, >>>>>> Registernummer HRB 10646 >>>>>> This e-mail may contain confidential and/or privileged information. >>>>>> If you are not the intended recipient (or have received this e-mail in >>>>>> error) please notify the sender immediately and destroy this e-mail. Any >>>>>> unauthorized copying, disclosure or distribution of the material in this >>>>>> e-mail is strictly forbidden >>>>>> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte >>>>>> Informationen. Wenn Sie nicht der richtige Adressat sind oder diese >>>>>> E-Mail >>>>>> irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und >>>>>> vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte >>>>>> Weitergabe dieser Mail sind nicht gestattet. >>>>>> >>>>> -- >>>> >>> You received this message because you are subscribed to the Google >>>> Groups "Puppet Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/puppet-users/80022945-ecdc-43ef-b857-ca2813c37919n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/puppet-users/80022945-ecdc-43ef-b857-ca2813c37919n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/puppet-users/9cbe27fa-f4a9-476b-8f88-6490bd83435an%40googlegroups.com >> >> <https://groups.google.com/d/msgid/puppet-users/9cbe27fa-f4a9-476b-8f88-6490bd83435an%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/93d1956d-afdc-4e75-b5cc-231c44f42771n%40googlegroups.com.
