So I've done some research on the puppet generate types command. I'm seeing many different results from not having issues to causing issues with puppet apply and puppet agent executions. If I was to run this command and things go wrong, how do you reverse it? Remove the .resource_types directory?
On Friday, August 28, 2020 at 2:47:26 PM UTC-4 [email protected] wrote: > Justin, yes it's happening in all environments which leads me to believe > it's related to an old copy > in /opt/puppetlabs/puppet/cache/lib/puppet/type. Still trying to wrap my > head around why one domain installation is fine and the other domain > installation is not. > > Here is the contents of that directory which works: > [root@myserverlab type]# pwd > /opt/puppetlabs/puppet/cache/lib/puppet/type > [root@myserverlab type]# ls -al > total 24 > drwxr-xr-x 2 root root 77 Jul 16 16:04 . > drwxr-xr-x 6 root root 61 Feb 3 2020 .. > -rw-r--r-- 1 root root 1706 Jul 16 16:04 anchor.rb > -rw-r--r-- 1 root root 6921 Jul 16 16:04 file_line.rb > -rw-r--r-- 1 root root 1863 May 1 2017 httparch.rb > -rw-r--r-- 1 root root 6957 Jul 13 17:04 httpfile.rb > [root@myserverlab type]# > > Here is the contents of the directory that doesn't work: > [root@myserverprod type]# pwd > /opt/puppetlabs/puppet/cache/lib/puppet/type > [root@myserverprod type]# ls -al > total 24 > drwxr-xr-x 2 root root 77 Sep 30 2018 . > drwxr-xr-x 6 root root 61 Apr 24 2017 .. > -rw-r--r-- 1 root root 1752 Sep 30 2018 anchor.rb > -rw-r--r-- 1 root root 7113 Sep 30 2018 file_line.rb > -rw-r--r-- 1 root root 1863 May 15 2017 httparch.rb > -rw-r--r-- 1 root root 6357 Apr 24 2017 httpfile.rb > [root@myserverprod type]# > > You can clearly see the date and size difference of httpfile.rb. I > quadruple checked the puppet module directory on the prod server and the > code does have the quick_check parm. For some reason it is just not > refreshing the server cache. Both domains have a value of 0 for > environment_timeout for each environment. > > On Friday, August 28, 2020 at 2:32:05 PM UTC-4 Justin Stoller wrote: > >> On Fri, Aug 28, 2020 at 10:14 AM [email protected] <[email protected]> >> wrote: >> >>> Great info but I think I might have found the issue. >>> >>> So we don't use r10k to deploy code we use a different tool. But what I >>> found is on the puppet server (master) the httpfile.rb in >>> /opt/puppetlabs/puppet/cache/lib/puppet/type is the older version. >>> >> >> I think puppet/cache is read by the agent not the server. I would expect >> that to cause problems on applying a catalog from the server, not result in >> a failed compilation. But barring a .resource_tyeps directory existing in >> an environment it must be an incorrect version of the httpfile.rb in the >> server's loadpath. >> >> >>> I didn't find any ./resource_types directory in our environment >>> directories so not sure if we are using environment isolation or not. >>> >> >> Just to clarify it will be ".resource_types" with a leading dot and will >> be hidden by default. [1] >> >> As part of Justin's suggestion to allow the DELETE option to be valid, >>> I had to restart each of our 4 puppet servers so according to some of this >>> conversation, that should have refreshed the cache right? >>> >> >> Restarting or reloading will evict the in memory cache so if you have a >> very long environment_timeout it will work as well as doing an eviction of >> all your environments. It will not however remove any old files in your >> .resources_types directory. You will need to run the `puppet generate >> types` command with `--force` for that. >> >> >>> >>> What else is odd is the domain where the quick_check parm is work seems >>> to be getting refreshed somehow in /opt/puppetlabs/puppet/cache/lib/puppet >>> subdirectories (just looking at time stamps). The deploy process works the >>> same in that domain along with the domain where quick_check is not working. >>> >> >> Can you validate that the failures happen not along a "domain" but along >> puppet environments. Like all the nodes that use httpfile in production >> have this failure but those in staging don't have this issue? If you have >> some succeeding and some failing I would expect this to be the environment >> to be the condition causing the different behavior. >> >> >>> Since the /opt/puppetlabs/bin/puppet generate types --environment >>> production --force operates by environment, could this possible break the >>> environment as well? These are production boxes I need to run this on and >>> want to make sure I don't break anything. Also using the environment parm >>> will this then setup environment isolation and do i have to manually manage >>> that each time code is deployed to that environment? >>> >> >> The environment param to `puppet generate types` specifies which >> environment to act on, without it the command will only act on the >> environment specified in the puppet.conf for the "main" section (The "main" >> or "user" sections are almost always unmanaged and adopt the default values >> which would be "production" for "environment" setting). >> >> Running the command should be a relatively safe command, however I'm >> going to advocate for anyone "doing it live" on a production box. In PE we >> deploy this code and run this command in a staging area and then either >> lock the server while we copy the files over or atomically manage a >> symlink. If you are using environment caching as well it should be even >> safer because types will only be read from disk on the first compilation >> that uses them and then cached in memory after that. >> >> hth, >> justin >> >> 1. >> https://puppet.com/docs/puppet/6.17/environment_isolation.html#env_generate_types >> >> >> On Tuesday, August 25, 2020 at 5:09:28 PM UTC-4 Justin Stoller wrote: >>> >>>> > why wouldn't puppet just do this automatically when a module changes? >>>> >>>> Some background. Puppet's type and provider system modifies the running >>>> Puppet instance when they're _loaded_. This causes issues when you try to >>>> load multiple conflicting versions of a type in different environments. To >>>> work around this we have a kind of header file for your types that Puppet >>>> can read w/o actually loading the type. This way Puppet Server can load >>>> multiple versions of a type (as long as those different versions are in >>>> different environments) and check that each environment uses the type >>>> correctly for that version. >>>> >>>> The command Dirk gave you, loads those types safely in a separate >>>> process and then serializes their parameter information into a format for >>>> Puppet to later read that doesn't corrupt its global state. It places this >>>> information in the ".resource_types" directory at the root of your >>>> environment (like "/etc/puppetlabs/code/environments/production") >>>> >>>> Also, in order to speed up Puppet Server catalog compilation, we >>>> attempt to cache information like type parameters. >>>> >>>> In PE, if you use our built in code management facilities, we generate >>>> this type information on every commit (if needed), distribute it to your >>>> compilers, and then evict the environment cache so that any new >>>> information >>>> will be read. >>>> >>>> In FOSS, r10k has a config setting to generate this info when it >>>> deploys an environment [1]. >>>> >>>> >>>> Now, the error you're ultimately getting involves Puppet Server >>>> thinking that you're using the httpfile class wrong. It looks like the >>>> "quick_check" field was added in the latest version. So really the first >>>> question would be, are you using the latest version in this environment? >>>> >>>> Assuming you're doing that you probably either have the environment >>>> cache containing an older version of the module (which should be resolved >>>> by restarting the server or evicting the environment cache) or an old >>>> .resource_types in the root of your environment that should be removed and >>>> regenerated like Dirk said. Possibly you could have an older version in a >>>> different environment that's being loaded first, but I don't think that'd >>>> cause a problem for uncached, new parameters on a type. >>>> >>>> HTH, >>>> Justin >>>> >>>> 1. >>>> https://github.com/puppetlabs/r10k/blob/master/doc/dynamic-environments/configuration.mkd#generate_types >>>> >>>> On Tue, Aug 25, 2020 at 9:42 AM [email protected] <[email protected]> >>>> wrote: >>>> >>>>> So a followup to the original question. >>>>> >>>>> As a test we created a simple module on the node which is failing when >>>>> puppet agent executes. Running puppet apply, the parameter quick_check >>>>> is >>>>> found and the module completes successfully. So why would puppet apply >>>>> work and not puppet agent? >>>>> >>>>> Code: >>>>> >>>>> class testmod() >>>>> >>>>> { >>>>> >>>>> httpfile { "ansible-2.8.0a1.tar.gz": >>>>> >>>>> ensure => present, >>>>> >>>>> path => "/u01/testmod/ansible-2.8.0a1.tar.gz", >>>>> >>>>> source => " >>>>> https://mynexus.domain.com/nexus/repository/ae-raw-ansible-group/ansible/ansible-2.8.0a1.tar.gz >>>>> >>>>> <https://nexus.aetna.com/nexus/repository/ae-raw-ansible-group/ansible/ansible-2.8.0a1.tar.gz> >>>>> ", >>>>> >>>>> quick_check => true, >>>>> >>>>> # hash => 'hex form SHA2 hash OR an URL to the .sha file with >>>>> that hash' >>>>> >>>>> } >>>>> >>>>> } >>>>> >>>>> >>>>> >>>>> Here is my run: >>>>> [root@node testmod]# puppet apply --modulepath=/home/toor --test -e >>>>> "include testmod" --verbose >>>>> >>>>> On Tuesday, August 25, 2020 at 12:38:05 PM UTC-4 [email protected] >>>>> wrote: >>>>> >>>>>> Dirk, why wouldn't puppet just do this automatically when a module >>>>>> changes? Is there a bug somewhere? >>>>>> >>>>>> On Tuesday, August 25, 2020 at 2:43:03 AM UTC-4 Dirk Heinrichs wrote: >>>>>> >>>>>>> Am Montag, den 24.08.2020, 11:06 -0700 schrieb [email protected]: >>>>>>> >>>>>>> Justin, I implemented the suggestion you made however after running >>>>>>> the curl command against the 2 environments having the issue and >>>>>>> receiving >>>>>>> the 204 response, the puppet module is still getting the 500 error. Do >>>>>>> you >>>>>>> or anyone else have any other suggestions? Is it possible it's related >>>>>>> to >>>>>>> ruby and/or java? Frankly I'm stumped. >>>>>>> >>>>>>> >>>>>>> Didn't see this earlier, sorry. >>>>>>> >>>>>>> The "no parameter named 'xxx'" error can usually be resolved by >>>>>>> recreating the metadata for your Puppet environment(s). This can be >>>>>>> done on >>>>>>> the Puppet master using the following command (for the production >>>>>>> environment): >>>>>>> >>>>>>> /opt/puppetlabs/bin/puppet generate types --environment production >>>>>>> --force >>>>>>> >>>>>>> I've added this command to my environment update script after >>>>>>> running into this problem myself a few months ago after updating some >>>>>>> external modules from the forge. >>>>>>> >>>>>>> See https://puppet.com/docs/puppet/5.5/environment_isolation.html for >>>>>>> the details. >>>>>>> >>>>>>> HTH... >>>>>>> >>>>>>> Dirk >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> *Dirk Heinrichs* >>>>>>> Senior Systems Engineer, Delivery Pipeline >>>>>>> OpenText ™ Discovery | Recommind >>>>>>> *Phone*: +49 2226 15966 18 <+49%202226%201596618> >>>>>>> *Email*: [email protected] >>>>>>> *Website*: www.recommind.de >>>>>>> Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach >>>>>>> Vertretungsberechtigte Geschäftsführer Gordon Davies, Madhu >>>>>>> Ranganathan, Christian Waida, Registergericht Amtsgericht Bonn, >>>>>>> Registernummer HRB 10646 >>>>>>> This e-mail may contain confidential and/or privileged information. >>>>>>> If you are not the intended recipient (or have received this e-mail in >>>>>>> error) please notify the sender immediately and destroy this e-mail. >>>>>>> Any >>>>>>> unauthorized copying, disclosure or distribution of the material in >>>>>>> this >>>>>>> e-mail is strictly forbidden >>>>>>> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte >>>>>>> Informationen. Wenn Sie nicht der richtige Adressat sind oder diese >>>>>>> E-Mail >>>>>>> irrtümlich erhalten haben, informieren Sie bitte sofort den Absender >>>>>>> und >>>>>>> vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte >>>>>>> Weitergabe dieser Mail sind nicht gestattet. >>>>>>> >>>>>> -- >>>>> >>>> You received this message because you are subscribed to the Google >>>>> Groups "Puppet Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/puppet-users/80022945-ecdc-43ef-b857-ca2813c37919n%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/puppet-users/80022945-ecdc-43ef-b857-ca2813c37919n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/puppet-users/9cbe27fa-f4a9-476b-8f88-6490bd83435an%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/puppet-users/9cbe27fa-f4a9-476b-8f88-6490bd83435an%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/10f72a8f-b9b3-4686-ab6d-5e39a252f339n%40googlegroups.com.
