Hi, I'm struggling with a simple update of PuppetDB since a couple of days, without finding the problem. I have 4 PuppetServers running Puppetserver 6.9 (puppetserver-6.9.0-1.el7.noarch). One has the CA role, the 3 others are simple masters. I have one dedicated PuppetDB server running puppetdb-6.5.0-1.
Everything is working like a charm since a couple of years. It was updated from Puppet 3, 4 and 6 without a glitch. Everything is running on CentOS 7. Now, when I want to update PuppetDb from 6.5 to 6.9, nothing works anymore. All nodes are complaining with these messages : Warning: Unable to fetch my node definition, but the agent run will continue: Warning: Error 500 on SERVER: Server Error: Could not retrieve facts for vmlabybr06.staging.rsvgnw.local: Failed to find facts from PuppetDB at vmprdpuppet41.rsvgnw.local:8140: Failed to execute '/pdb/query/v4/nodes/vmlabybr06.staging.rsvgnw.local/facts' on at least 1 of the following 'server_urls': https://vmctldeploy20.rsvgnw.local:8081 Info: Retrieving pluginfacts Info: Retrieving plugin Info: Retrieving locales Info: Loading facts Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed to execute '/pdb/cmd/v1?checksum=5da252cdae0fc1737726e9ace846d74856395703&version=5&certname=vmlabybr06.staging.rsvgnw.local&command=replace_facts&producer-timestamp=2020-04-09T13:15:44.382Z' on at least 1 of the following 'server_urls': https://vmctldeploy20.rsvgnw.local:8081 Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run In the server log I get this : 2020-04-09T15:22:45.169+02:00 WARN [qtp1002336767-143] [c.p.h.c.i.PersistentSyncHttpClient] Error executing http request javax.net.ssl.SSLException: Received fatal alert: handshake_failure at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1647) at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1615) at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1781) at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1070) at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896) at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:271) at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:316) at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:503) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) at java.lang.Thread.run(Thread.java:748) 2020-04-09T15:22:45.171+02:00 WARN [qtp1002336767-143] [puppetserver] Puppet Error connecting to vmctldeploy20.rsvgnw.local on 8081 at route /pdb/cmd/v1?checksum=0f8f2f1e474b2f551f6dc656bff34f1e43e56f6b&version=8&certname=vmlabvmt01.rsvgnw.local&command=store_report&producer-timestamp=2020-04-09T13:22:45.130Z, error message received was 'Error executing http request'. Failing over to the next PuppetDB server_url in the 'server_urls' list 2020-04-09T15:22:45.172+02:00 ERROR [qtp1002336767-143] [puppetserver] Puppet Failed to execute '/pdb/cmd/v1?checksum=0f8f2f1e474b2f551f6dc656bff34f1e43e56f6b&version=8&certname=vmlabvmt01.rsvgnw.local&command=store_report&producer-timestamp=2020-04-09T13:22:45.130Z' on at least 1 of the following 'server_urls': https://vmctldeploy20.rsvgnw.local:8081 I have checked a few things : - Updated puppetdb-termini on the puppet-master from 6.5 to 6.9 (no change) - added "verify_client_certificate = false" to /etc/puppetlabs/puppet/puppetdb.conf on the masters (no change) - added full certs list to PuppetDB server /etc/puppetlabs/puppetdb/ssl/public.pem I've read there has been a change liked to SSL in the PuppetDB 6.6 CHANGELOG. Here is what happens when I try to connect with openssl for troubleshooting, to PuppetDB 6.5 openssl s_client -host puppetdb -port 8081 -CAfile /etc/puppetlabs/puppet/ssl/certs/ca.pem CONNECTED(00000003) Can't use SSL_get_servername depth=1 CN = Puppet CA: vmctldeploy10.rsvgnw.local verify return:1 depth=0 CN = vmctldeploy20.rsvgnw.local verify return:1 140503727654720:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:ssl/record/rec_layer_s3.c:1544:SSL alert number 42 --- Certificate chain 0 s:CN = vmctldeploy20.rsvgnw.local i:CN = Puppet CA: vmctldeploy10.rsvgnw.local --- Server certificate -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- subject=CN = vmctldeploy20.rsvgnw.local issuer=CN = Puppet CA: vmctldeploy10.rsvgnw.local --- Acceptable client certificate CA names CN = Puppet CA: vmctldeploy10.rsvgnw.local Client Certificate Types: RSA sign, DSA sign, ECDSA sign Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1 Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1 Peer signing digest: SHA256 Peer signature type: RSA Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 2213 bytes and written 455 bytes Verification: OK --- The only way to go back is doing a full "revert to snaphot", as the db is migrated between 6.5 and 6.9 Any advise welcome ! Cheers Yvan -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/33e92b3d-84d0-42ce-87ee-d958b8cf78d1%40googlegroups.com.
