Hi Rohit, I don't have great ideas about what's going on in your environment. Are you using custom built containers or the puppet namespaced containers from hub.docker.com. You could try applying this patch ( https://github.com/puppetlabs/puppetdb/commit/a1ab2f50598f12ac51acb21f256232143891dbc1) and setting PUPPETSERVER_HOSTNAME in your docker-compose.yml to puppet4.psd401.net.
In one of your earlier emails you mentioned you were using puppetserver 2.7, but it looks like in the compose file you attached you're using the puppet/puppetserver:latest, which is puppetserver 6.x, just want to make sure I know what versions of things you're running with here. The containers for puppetserver 2.7 / puppetdb 4.2 are definitely old, and with recent efforts we have made a number of changes and improvements to the containers and the compose stack ( https://github.com/puppetlabs/pupperware), but that work has been for puppetserver and puppetdb 5+. If you run `puppet agent -t` on your puppetserver container does it succeed? (docker-compose exec puppet puppet agent -t) On Wed, Nov 28, 2018 at 9:20 AM Rohit <[email protected]> wrote: > Any idea if there are other steps I can consider? If not, should I simply > rebuild the system? If I do go this route, is there a way to backup all the > Puppet configurations set for servers and services that can be reimported > in a fresh install? Would it also be suggested to go a non-Docker route due > to stability? > > > On Thursday, November 15, 2018 at 1:33:43 PM UTC-8, Morgan Rhodes wrote: > >> Hi Rohit, >> >> No, unfortunately, it's not just a change in your docker-compose.yml. >> When you're generating the certs for your puppetserver, you'll want to make >> sure you're passing the `--dns_alt_names=<altnames>`, so it would be >> something like: >> puppet cert generate puppet4.psd401.net --dns_alt_names=puppet, >> puppet.psd401.net >> >> Afterwards, you can confirm that your certificate has all of the altnames >> with `puppet cert list --all`, you should see something like: >> $ puppet cert list --all >> + "puppet4.psd401.net" (SHA256) <fingerprint> (alt names: "DNS:puppet", >> "DNS:puppet4.psd401.net") >> >> On Tue, Nov 13, 2018 at 11:23 AM Rohit <[email protected]> wrote: >> > Hello Morgan, >>> >>> Apologies for the late response here, some of our Puppet services had >>> started working but it looks like the same issue has arised and I am not >>> entirely sure why. I did check the docker-entrypoint.sh file and indeed see >>> the very exact response as you posted. However my question is for the >>> "altname" that you suggested, would I change this in the docker-compose.yml >>> file? I also realize the full docker-compose.yml did not show up in my >>> previous post but have attached it again in a separate file. >>> >>> >>> On Friday, October 19, 2018 at 4:38:12 PM UTC-7, Morgan Rhodes wrote: >>> >>>> When you look at the output of `puppet cert list all` does the >>>> certificate for your puppetmaster also include the alt name 'puppet'? >>>> (Something like 'alt names: "DNS:puppet", "DNS:testpuppet"'). If not, I'm >>>> guessing that's your problem. >>>> >>>> You mentioned in your earlier email that you were using puppetdb 4.2.0. >>>> I'm assuming you're running the puppet/puppetdb:4.2.0 container. To get the >>>> container entrypoint, I start the container manually with a custom >>>> entrypoint so I can look around, there should be a file >>>> 'docker-entrypoint.sh' in the root directory of the container. >>>> >>>> $ docker run --rm -it --entrypoint /bin/bash puppet/puppetdb:4.2.0 >>>> root@e09f677618d7:/# ls >>>> Dockerfile bin boot dev docker-entrypoint.sh etc home lib >>>> lib64 media mnt opt proc root run sbin srv sys tmp usr var >>>> root@e09f677618d7:/# cat docker-entrypoint.sh >>>> #!/bin/bash >>>> >>>> if [ ! -d "/etc/puppetlabs/puppetdb/ssl" ]; then >>>> while ! nc -z puppet 8140; do >>>> sleep 1 >>>> done >>>> set -e >>>> /opt/puppetlabs/bin/puppet agent --verbose --onetime --no-daemonize >>>> --waitforcert 120 >>>> /opt/puppetlabs/server/bin/puppetdb ssl-setup -f >>>> fi >>>> >>>> exec /opt/puppetlabs/server/bin/puppetdb "$@" >>>> root@e09f677618d7:/# >>>> >>>> The docker-entrypoint.sh script in that version of the container >>>> doesn't have any logic for a puppetserver with a non-default name, which >>>> means when it runs `puppet agent --verbose --onetime --no-daemonize >>>> --waitforcert 120` it will connect to the host named 'puppet'. From the >>>> link you have set up in your docker-compose.yml, I'm assuming your >>>> puppetserver container name is 'puppet' with the hostname ' >>>> puppet4.psd401.net'. Since the container name is 'puppet', the >>>> puppetdb container is able to resolve 'puppet' as 'puppet4....', so when it >>>> runs puppet agent -t it can connect to the host, but certificate validation >>>> will fail if puppet isn't listed as one of the valid altnames for the >>>> puppet container. >>>> >>>> On Fri, Oct 19, 2018 at 11:35 AM Rohit <[email protected]> >>>> wrote: >>>> >>> >>>>> 1. puppet_db is trying to connect our.puppet.domain, there is no >>>>> docker-entrypoint.sh script that I was able to find. >>>>> 1. For reference, this is the docker-compose.yml: >>>>> 2. >>>>> >>>>> puppetdb: >>>>> >>>>> container_name: puppet_db >>>>> >>>>> hostname: puppetdb.peninsula.wednet.edu >>>>> >>>>> dns: >>>>> >>>>> - 10.0.0.7 >>>>> >>>>> image: puppet/puppetdb:latest >>>>> >>>>> ports: >>>>> >>>>> - 8087:8080 >>>>> >>>>> - 8088:8081 >>>>> >>>>> depends_on: >>>>> >>>>> - puppet >>>>> >>>>> links: >>>>> >>>>> - puppet:puppet4.psd401.net >>>>> >>>>> - puppetdbpostgres:postgres >>>>> >>>>> volumes: >>>>> >>>>> - ./puppet-client.conf:/etc/puppetlabs/puppet/puppet.conf >>>>> >>>>> - ./puppetdb_conf:/etc/puppetlabs/puppetdb/conf.d >>>>> >>>>> - ./puppetdb_ssl:/etc/puppetlabs/puppet/ssl/ >>>>> >>>>> networks: >>>>> >>>>> puppet: >>>>> >>>>> ipv4_address: 172.19.0.4 >>>>> >>>>> restart: always >>>>> >>>>> >>>>> 2. The hostname that the puppetdb container is trying to >>>>> connect to is indeed the one listed on the certificate name on the >>>>> puppet >>>>> servers cert. >>>>> >>>>> >>>>> On Friday, October 19, 2018 at 10:09:56 AM UTC-7, Morgan Rhodes wrote: >>>>> >>>>>> A few things to verify: >>>>>> >>>>>> 1) what hostname is your puppetdb container trying to connect to >>>>>> puppetserver at? >>>>>> a) This should be in your docker-entrypoint.sh script in the >>>>>> puppetdb container. Likely either 'puppet' or '$PUPPETSERVER_HOSTNAME' >>>>>> depending on what variables you have set in your compose file and what >>>>>> version of the puppetdb container you have. >>>>>> >>>>>> 2) Is the hostname your puppetdb container is trying to connect to >>>>>> listed as one of the certificate names for your puppet server's cert? >>>>>> a) For example, in my puppetserver container when I run `puppet >>>>>> cert list --all` I see: >>>>>> >>>>>> + "testserver" (SHA256) >>>>>> F0:31:6D:1D:03:82:C0:84:0D:FA:2B:28:5B:52:CB:18:88:87:61:5F:5A:F5:7E:AB:A2:73:29:44:BC:57:D0:99 >>>>>> (alt names: "DNS:testserver", "DNS:foo") >>>>>> >>>>>> if my puppetdb container tries to connect to that host over any >>>>>> names other than 'testserver' or 'foo' I get a certificate verify failed >>>>>> error. >>>>>> >>>>>> >>>>>> On Fri, Oct 19, 2018 at 9:02 AM Rohit <[email protected]> >>>>>> wrote: >>>>>> >>>>> Hello Morgan, >>>>>>> >>>>>>> If you are refferring to the cert being in the conf/ssl/certs >>>>>>> folder, then yes, our.puppet.domain.pem is in the folder. When running >>>>>>> the >>>>>>> 'puppet cert list --all' I see three certificates (in the SHA256 >>>>>>> format): >>>>>>> >>>>>>> - computername.our.puppet.domain >>>>>>> - our.puppet.domain >>>>>>> - servername.our.puppet.domain >>>>>>> >>>>>>> If it is a DNS issue, do I have to likely change something from the >>>>>>> docker-compose side? >>>>>>> >>>>>>> >>>>>>> On Thursday, October 18, 2018 at 2:14:54 PM UTC-7, Morgan Rhodes >>>>>>> wrote: >>>>>>>> >>>>>>>> Hi Rohit, >>>>>>>> >>>>>>>> Is the hostname from `/CN=our.puppet.domain` showing up in your >>>>>>>> puppetserver's certificate? You can verify that with `puppet cert list >>>>>>>> --all` on the puppetserver container. This looks like a DNS issue. >>>>>>>> >>>>>>>> On Thursday, October 18, 2018 at 11:41:16 AM UTC-7, Rohit wrote: >>>>>>>>> >>>>>>>>> Hello, we currently have a puppet docker container setup and are >>>>>>>>> experiencing certificate issues. Basically, in our docker setup (on >>>>>>>>> our >>>>>>>>> main server) I had generated and signed new certificates, but the >>>>>>>>> puppet_db >>>>>>>>> container keeps restarting. Here are logs from the puppet_db >>>>>>>>> container: >>>>>>>>> >>>>>>>>> ‘Error: Could not retrieve catalog from remote server: >>>>>>>>> SSL_connect returned=1 errno=0 state=error: certificate verify failed: >>>>>>>>> [unable to get local issuer certificate for /CN=our.puppet.domain] >>>>>>>>> Error: Could not retrieve catalog; skipping run >>>>>>>>> Error: Could not send report: SSL_connect returned=1 errno=0 >>>>>>>>> state=error: certificate verify failed: [unable to get local issuer >>>>>>>>> certificate for /CN=our.puppet.domain]’ >>>>>>>>> >>>>>>>>> I have tried series of steps to solve this problem as it looks >>>>>>>>> like Puppet is not functioning correctly as our servers are not >>>>>>>>> properly >>>>>>>>> listening to the host server. Any idea what I can do to solve this >>>>>>>>> problem? >>>>>>>>> For reference, we are running Puppet_DB version 4.2 and Puppet Server >>>>>>>>> version 2.7.2, all of which is set up on a docker container >>>>>>>>> environment on >>>>>>>>> one server. >>>>>>>>> >>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Puppet Users" group. >>>>>>> >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>> >>>>>> >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/puppet-users/899d2bf7-ceed-4d9e-bd24-c4ba2cc93928%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/puppet-users/899d2bf7-ceed-4d9e-bd24-c4ba2cc93928%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> -- >>>>>> Morgan Rhodes >>>>>> [email protected] >>>>>> Release Engineer >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Puppet Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/puppet-users/fcf0c6da-82dd-4970-ab81-a60131b291f1%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/puppet-users/fcf0c6da-82dd-4970-ab81-a60131b291f1%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>>> Morgan Rhodes >>>> [email protected] >>>> Release Engineer >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/puppet-users/1015311c-0e2f-44f0-a096-6c5015d00d98%40googlegroups.com >>> <https://groups.google.com/d/msgid/puppet-users/1015311c-0e2f-44f0-a096-6c5015d00d98%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> Morgan Rhodes >> [email protected] >> Release Engineer >> > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/9d319859-8f13-4e6e-97c9-c2366152a4e3%40googlegroups.com > <https://groups.google.com/d/msgid/puppet-users/9d319859-8f13-4e6e-97c9-c2366152a4e3%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Morgan Rhodes [email protected] Release Engineer -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CA%2BFnDv2pbbH5pQJdgkZEYnHJd475mWX2QDX%3D_4CCtVwHohtqkg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
