On Tue, Sep 15, 2020 at 5:25 AM [email protected] <[email protected]> wrote:
> So I've done some research on the puppet generate types command. I'm > seeing many different results from not having issues to causing issues with > puppet apply and puppet agent executions. We fixed an issue in 6.18.0 (https://tickets.puppetlabs.com/browse/PUP-9602) that caused "puppet apply" to fail in some cases if "puppet generate types" had been run previously. > If I was to run this command and things go wrong, how do you reverse > it? Remove the .resource_types directory? > > On Friday, August 28, 2020 at 2:47:26 PM UTC-4 [email protected] wrote: > >> Justin, yes it's happening in all environments which leads me to believe >> it's related to an old copy >> in /opt/puppetlabs/puppet/cache/lib/puppet/type. Still trying to wrap my >> head around why one domain installation is fine and the other domain >> installation is not. >> >> Here is the contents of that directory which works: >> [root@myserverlab type]# pwd >> /opt/puppetlabs/puppet/cache/lib/puppet/type >> [root@myserverlab type]# ls -al >> total 24 >> drwxr-xr-x 2 root root 77 Jul 16 16:04 . >> drwxr-xr-x 6 root root 61 Feb 3 2020 .. >> -rw-r--r-- 1 root root 1706 Jul 16 16:04 anchor.rb >> -rw-r--r-- 1 root root 6921 Jul 16 16:04 file_line.rb >> -rw-r--r-- 1 root root 1863 May 1 2017 httparch.rb >> -rw-r--r-- 1 root root 6957 Jul 13 17:04 httpfile.rb >> [root@myserverlab type]# >> >> Here is the contents of the directory that doesn't work: >> [root@myserverprod type]# pwd >> /opt/puppetlabs/puppet/cache/lib/puppet/type >> [root@myserverprod type]# ls -al >> total 24 >> drwxr-xr-x 2 root root 77 Sep 30 2018 . >> drwxr-xr-x 6 root root 61 Apr 24 2017 .. >> -rw-r--r-- 1 root root 1752 Sep 30 2018 anchor.rb >> -rw-r--r-- 1 root root 7113 Sep 30 2018 file_line.rb >> -rw-r--r-- 1 root root 1863 May 15 2017 httparch.rb >> -rw-r--r-- 1 root root 6357 Apr 24 2017 httpfile.rb >> [root@myserverprod type]# >> >> You can clearly see the date and size difference of httpfile.rb. I >> quadruple checked the puppet module directory on the prod server and the >> code does have the quick_check parm. For some reason it is just not >> refreshing the server cache. Both domains have a value of 0 for >> environment_timeout for each environment. >> >> On Friday, August 28, 2020 at 2:32:05 PM UTC-4 Justin Stoller wrote: >> >>> On Fri, Aug 28, 2020 at 10:14 AM [email protected] <[email protected]> >>> wrote: >>> >>>> Great info but I think I might have found the issue. >>>> >>>> So we don't use r10k to deploy code we use a different tool. But what >>>> I found is on the puppet server (master) the httpfile.rb in >>>> /opt/puppetlabs/puppet/cache/lib/puppet/type is the older version. >>>> >>> >>> I think puppet/cache is read by the agent not the server. I would expect >>> that to cause problems on applying a catalog from the server, not result in >>> a failed compilation. But barring a .resource_tyeps directory existing in >>> an environment it must be an incorrect version of the httpfile.rb in the >>> server's loadpath. >>> >>> >>>> I didn't find any ./resource_types directory in our environment >>>> directories so not sure if we are using environment isolation or not. >>>> >>> >>> Just to clarify it will be ".resource_types" with a leading dot and will >>> be hidden by default. [1] >>> >>> As part of Justin's suggestion to allow the DELETE option to be valid, >>>> I had to restart each of our 4 puppet servers so according to some of this >>>> conversation, that should have refreshed the cache right? >>>> >>> >>> Restarting or reloading will evict the in memory cache so if you have a >>> very long environment_timeout it will work as well as doing an eviction of >>> all your environments. It will not however remove any old files in your >>> .resources_types directory. You will need to run the `puppet generate >>> types` command with `--force` for that. >>> >>> >>>> >>>> What else is odd is the domain where the quick_check parm is work seems >>>> to be getting refreshed somehow in /opt/puppetlabs/puppet/cache/lib/puppet >>>> subdirectories (just looking at time stamps). The deploy process works the >>>> same in that domain along with the domain where quick_check is not working. >>>> >>> >>> Can you validate that the failures happen not along a "domain" but along >>> puppet environments. Like all the nodes that use httpfile in production >>> have this failure but those in staging don't have this issue? If you have >>> some succeeding and some failing I would expect this to be the environment >>> to be the condition causing the different behavior. >>> >>> >>>> Since the /opt/puppetlabs/bin/puppet generate types --environment >>>> production --force operates by environment, could this possible break the >>>> environment as well? These are production boxes I need to run this on and >>>> want to make sure I don't break anything. Also using the environment parm >>>> will this then setup environment isolation and do i have to manually manage >>>> that each time code is deployed to that environment? >>>> >>> >>> The environment param to `puppet generate types` specifies which >>> environment to act on, without it the command will only act on the >>> environment specified in the puppet.conf for the "main" section (The "main" >>> or "user" sections are almost always unmanaged and adopt the default values >>> which would be "production" for "environment" setting). >>> >>> Running the command should be a relatively safe command, however I'm >>> going to advocate for anyone "doing it live" on a production box. In PE we >>> deploy this code and run this command in a staging area and then either >>> lock the server while we copy the files over or atomically manage a >>> symlink. If you are using environment caching as well it should be even >>> safer because types will only be read from disk on the first compilation >>> that uses them and then cached in memory after that. >>> >>> hth, >>> justin >>> >>> 1. >>> https://puppet.com/docs/puppet/6.17/environment_isolation.html#env_generate_types >>> >>> >>> On Tuesday, August 25, 2020 at 5:09:28 PM UTC-4 Justin Stoller wrote: >>>> >>>>> > why wouldn't puppet just do this automatically when a module changes? >>>>> >>>>> Some background. Puppet's type and provider system modifies the >>>>> running Puppet instance when they're _loaded_. This causes issues when you >>>>> try to load multiple conflicting versions of a type in different >>>>> environments. To work around this we have a kind of header file for your >>>>> types that Puppet can read w/o actually loading the type. This way Puppet >>>>> Server can load multiple versions of a type (as long as those different >>>>> versions are in different environments) and check that each environment >>>>> uses the type correctly for that version. >>>>> >>>>> The command Dirk gave you, loads those types safely in a separate >>>>> process and then serializes their parameter information into a format for >>>>> Puppet to later read that doesn't corrupt its global state. It places this >>>>> information in the ".resource_types" directory at the root of your >>>>> environment (like "/etc/puppetlabs/code/environments/production") >>>>> >>>>> Also, in order to speed up Puppet Server catalog compilation, we >>>>> attempt to cache information like type parameters. >>>>> >>>>> In PE, if you use our built in code management facilities, we generate >>>>> this type information on every commit (if needed), distribute it to your >>>>> compilers, and then evict the environment cache so that any new >>>>> information >>>>> will be read. >>>>> >>>>> In FOSS, r10k has a config setting to generate this info when it >>>>> deploys an environment [1]. >>>>> >>>>> >>>>> Now, the error you're ultimately getting involves Puppet Server >>>>> thinking that you're using the httpfile class wrong. It looks like the >>>>> "quick_check" field was added in the latest version. So really the first >>>>> question would be, are you using the latest version in this environment? >>>>> >>>>> Assuming you're doing that you probably either have the environment >>>>> cache containing an older version of the module (which should be resolved >>>>> by restarting the server or evicting the environment cache) or an old >>>>> .resource_types in the root of your environment that should be removed and >>>>> regenerated like Dirk said. Possibly you could have an older version in a >>>>> different environment that's being loaded first, but I don't think that'd >>>>> cause a problem for uncached, new parameters on a type. >>>>> >>>>> HTH, >>>>> Justin >>>>> >>>>> 1. >>>>> https://github.com/puppetlabs/r10k/blob/master/doc/dynamic-environments/configuration.mkd#generate_types >>>>> >>>>> On Tue, Aug 25, 2020 at 9:42 AM [email protected] <[email protected]> >>>>> wrote: >>>>> >>>>>> So a followup to the original question. >>>>>> >>>>>> As a test we created a simple module on the node which is failing >>>>>> when puppet agent executes. Running puppet apply, the parameter >>>>>> quick_check is found and the module completes successfully. So why would >>>>>> puppet apply work and not puppet agent? >>>>>> >>>>>> Code: >>>>>> >>>>>> class testmod() >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> { >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> httpfile >>>>>> >>>>>> { "ansible-2.8.0a1.tar.gz": >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> ensure => present, >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> path => >>>>>> >>>>>> "/u01/testmod/ansible-2.8.0a1.tar.gz", >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> source => " >>>>>> https://mynexus.domain.com/nexus/repository/ae-raw-ansible-group/ansible/ansible-2.8.0a1.tar.gz >>>>>> <https://nexus.aetna.com/nexus/repository/ae-raw-ansible-group/ansible/ansible-2.8.0a1.tar.gz> >>>>>> ", >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> quick_check => true, >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> # >>>>>> >>>>>> hash => 'hex form SHA2 hash OR an URL to the .sha file >>>>>> >>>>>> with that hash' >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> } >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> } >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Here is my run: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> [root@node testmod]# puppet apply --modulepath=/home/toor --test -e >>>>>> "include >>>>>> >>>>>> testmod" --verbose >>>>>> >>>>>> On Tuesday, August 25, 2020 at 12:38:05 PM UTC-4 [email protected] >>>>>> wrote: >>>>>> >>>>>>> Dirk, why wouldn't puppet just do this automatically when a module >>>>>>> changes? Is there a bug somewhere? >>>>>>> >>>>>>> On Tuesday, August 25, 2020 at 2:43:03 AM UTC-4 Dirk Heinrichs wrote: >>>>>>> >>>>>>>> Am Montag, den 24.08.2020, 11:06 -0700 schrieb [email protected]: >>>>>>>> >>>>>>>> Justin, I implemented the suggestion you made however after running >>>>>>>> the curl command against the 2 environments having the issue and >>>>>>>> receiving >>>>>>>> the 204 response, the puppet module is still getting the 500 error. >>>>>>>> Do you >>>>>>>> or anyone else have any other suggestions? Is it possible it's >>>>>>>> related to >>>>>>>> ruby and/or java? Frankly I'm stumped. >>>>>>>> >>>>>>>> >>>>>>>> Didn't see this earlier, sorry. >>>>>>>> >>>>>>>> The "no parameter named 'xxx'" error can usually be resolved by >>>>>>>> recreating the metadata for your Puppet environment(s). This can be >>>>>>>> done on >>>>>>>> the Puppet master using the following command (for the production >>>>>>>> environment): >>>>>>>> >>>>>>>> /opt/puppetlabs/bin/puppet generate types --environment production >>>>>>>> --force >>>>>>>> >>>>>>>> I've added this command to my environment update script after >>>>>>>> running into this problem myself a few months ago after updating some >>>>>>>> external modules from the forge. >>>>>>>> >>>>>>>> See https://puppet.com/docs/puppet/5.5/environment_isolation.html for >>>>>>>> the details. >>>>>>>> >>>>>>>> HTH... >>>>>>>> >>>>>>>> Dirk >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> *Dirk Heinrichs* >>>>>>>> Senior Systems Engineer, Delivery Pipeline >>>>>>>> OpenText ™ Discovery | Recommind >>>>>>>> *Phone*: +49 2226 15966 18 <+49%202226%201596618> >>>>>>>> *Email*: [email protected] >>>>>>>> *Website*: www.recommind.de >>>>>>>> Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach >>>>>>>> <https://www.google.com/maps/search/Von-Liebig-Stra%C3%9Fe+1,+53359+Rheinbach?entry=gmail&source=g> >>>>>>>> Vertretungsberechtigte Geschäftsführer Gordon Davies, Madhu >>>>>>>> Ranganathan, Christian Waida, Registergericht Amtsgericht Bonn, >>>>>>>> Registernummer HRB 10646 >>>>>>>> This e-mail may contain confidential and/or privileged information. >>>>>>>> If you are not the intended recipient (or have received this e-mail in >>>>>>>> error) please notify the sender immediately and destroy this e-mail. >>>>>>>> Any >>>>>>>> unauthorized copying, disclosure or distribution of the material in >>>>>>>> this >>>>>>>> e-mail is strictly forbidden >>>>>>>> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte >>>>>>>> Informationen. Wenn Sie nicht der richtige Adressat sind oder diese >>>>>>>> E-Mail >>>>>>>> irrtümlich erhalten haben, informieren Sie bitte sofort den Absender >>>>>>>> und >>>>>>>> vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte >>>>>>>> Weitergabe dieser Mail sind nicht gestattet. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>> >>>>>> >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Puppet Users" group. >>>>>> >>>>>> >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> >>>>> >>>>>> >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/puppet-users/80022945-ecdc-43ef-b857-ca2813c37919n%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/puppet-users/80022945-ecdc-43ef-b857-ca2813c37919n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> >>>> You received this message because you are subscribed to the Google >>>> Groups "Puppet Users" group. >>>> >>>> >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>> >>>> >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/puppet-users/9cbe27fa-f4a9-476b-8f88-6490bd83435an%40googlegroups.com >>>> <https://groups.google.com/d/msgid/puppet-users/9cbe27fa-f4a9-476b-8f88-6490bd83435an%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> >>>> >>> >>> > > > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > > > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/10f72a8f-b9b3-4686-ab6d-5e39a252f339n%40googlegroups.com > <https://groups.google.com/d/msgid/puppet-users/10f72a8f-b9b3-4686-ab6d-5e39a252f339n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CA%2Bu97ukQpLE4iPEfT2J5PQeoudpe-LKNTddDsHtO4Mio-UT%2Byg%40mail.gmail.com.
