Thanks for the response. This is my main IPA server the rest of my small network are just linux clients.
kinit: Cannot contact any KDC for realm 'FAKE-IPA-DOMAIN.LAN' while getting initial credentials # getcert list Number of certificates and requests being tracked: 9. Request ID '20171108154417': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: SelfSign issuer: CN=sol.FAKE-IPA-DOMAIN.LAN,O=FAKE-IPA-DOMAIN.LAN subject: CN=sol.FAKE-IPA-DOMAIN.LAN,O=FAKE-IPA-DOMAIN.LAN expires: 2020-09-13 20:50:34 UTC principal name: krbtgt/[email protected] certificate template/profile: KDCs_PKINIT_Certs pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes Request ID '20181122014941': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN subject: CN=CA Audit,O=FAKE-IPA-DOMAIN.LAN expires: 2022-05-18 03:13:17 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20181122014942': status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN subject: CN=OCSP Subsystem,O=FAKE-IPA-DOMAIN.LAN expires: 2020-06-24 23:56:43 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20181122014943': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN subject: CN=CA Subsystem,O=FAKE-IPA-DOMAIN.LAN expires: 2022-05-18 03:11:57 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20181122014944': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN subject: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN expires: 2036-08-12 21:35:52 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20181122014945': status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN subject: CN=IPA RA,O=FAKE-IPA-DOMAIN.LAN expires: 2020-06-24 23:56:33 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20181122014946': status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN subject: CN=sol.FAKE-IPA-DOMAIN.LAN,O=FAKE-IPA-DOMAIN.LAN expires: 2020-06-24 23:55:43 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes Request ID '20181122014947': status: CA_UNREACHABLE ca-error: Error setting up ccache for "host" service on client using default keytab: Cannot contact any KDC for realm 'FAKE-IPA-DOMAIN.LAN'. stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-FAKE-IPA-DOMAIN-LAN',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-FAKE-IPA-DOMAIN-LAN/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-FAKE-IPA-DOMAIN-LAN',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN subject: CN=sol.FAKE-IPA-DOMAIN.LAN,O=FAKE-IPA-DOMAIN.LAN expires: 2020-07-17 16:47:45 UTC principal name: ldap/[email protected] key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv FAKE-IPA-DOMAIN-LAN track: yes auto-renew: yes Request ID '20181122014948': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN subject: CN=sol.FAKE-IPA-DOMAIN.LAN,O=FAKE-IPA-DOMAIN.LAN expires: 2022-03-16 22:14:54 UTC dns: sol.FAKE-IPA-DOMAIN.LAN principal name: HTTP/[email protected] key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes What can I do next? Thanks, -ms ________________________________ From: Florence Blanc-Renaud <[email protected]> Sent: Tuesday, June 30, 2020 1:45 AM To: FreeIPA users list <[email protected]> Cc: Mariusz Stolarczyk <[email protected]> Subject: Re: [Freeipa-users] ipa-server-upgrade failed after yum update on CentOS7 On 6/30/20 10:24 AM, Mariusz Stolarczyk via FreeIPA-users wrote: > All, > > I did a routine server updates last night on my IPA server. After the > reboot I first noticed the DNS was not resolving and the ipa.service > failed. The ipa.service failed to start so I ran the following: > > > # ipactl start > IPA version error: data needs to be upgraded (expected version > '4.6.6-11.el7.centos', current version '4.6.5-11.el7.centos.4') > Automatically running upgrade, for details see /var/log/ipaupgrade.log > Be patient, this may take a few minutes. > Automatic upgrade failed: Update complete > Upgrading the configuration of the IPA services > [Verifying that root certificate is published] > [Migrate CRL publish directory] > CRL tree already moved > [Verifying that CA proxy configuration is correct] > [Verifying that KDC configuration is using ipa-kdb backend] > [Fix DS schema file syntax] > Syntax already fixed > [Removing RA cert from DS NSS database] > RA cert already removed > [Enable sidgen and extdom plugins by default] > [Updating HTTPD service IPA configuration] > [Updating HTTPD service IPA WSGI configuration] > Nothing to do for configure_httpd_wsgi_conf > [Updating mod_nss protocol versions] > Protocol versions already updated > [Updating mod_nss cipher suite] > [Updating mod_nss enabling OCSP] > [Fixing trust flags in /etc/httpd/alias] > Trust flags already processed > [Moving HTTPD service keytab to gssproxy] > [Removing self-signed CA] > [Removing Dogtag 9 CA] > [Checking for deprecated KDC configuration files] > [Checking for deprecated backups of Samba configuration files] > [Remove FILE: prefix from 'dedicated keytab file' in Samba configuration] > [Update 'max smbd processes' in Samba configuration to prevent unlimited > SMBLoris attack amplification] > [Add missing CA DNS records] > IPA CA DNS records already processed > [Removing deprecated DNS configuration options] > [Ensuring minimal number of connections] > [Updating GSSAPI configuration in DNS] > [Updating pid-file configuration in DNS] > [Checking global forwarding policy in named.conf to avoid conflicts with > automatic empty zones] > Changes to named.conf have been made, restart named > [Upgrading CA schema] > CA schema update complete (no changes) > [Verifying that CA audit signing cert has 2 year validity] > [Update certmonger certificate renewal configuration] > Certmonger certificate renewal configuration already up-to-date > [Enable PKIX certificate path discovery and validation] > PKIX already enabled > [Authorizing RA Agent to modify profiles] > [Authorizing RA Agent to manage lightweight CAs] > [Ensuring Lightweight CAs container exists in Dogtag database] > [Adding default OCSP URI configuration] > [Ensuring CA is using LDAPProfileSubsystem] > [Migrating certificate profiles to LDAP] > IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run > command ipa-server-upgrade manually. > Unexpected error - see /var/log/ipaupgrade.log for details: > NetworkError: cannot connect to > 'https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffake-ipa-host.fake-ipa-domain.lan%3A8443%2Fca%2Frest%2Faccount%2Flogin&data=02%7C01%7C%7C474697e47e794ce1189c08d81cd1f156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291035300777903&sdata=FEc7EdbY6TKtCQlwtF39um4xgRPGVsxcMB08SpP1eRQ%3D&reserved=0': > [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) > The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for > more information > > See the upgrade log for more details and/or run > /usr/sbin/ipa-server-upgrade again > Aborting ipactl > > > The end of the /var/log/ipaupgrade.log file: > > 2020-06-29T22:43:38Z DEBUG stderr= > 2020-06-29T22:43:38Z DEBUG Loading Index file from > '/var/lib/ipa/sysrestore/sysrestore.index' > 2020-06-29T22:43:38Z DEBUG Starting external process > 2020-06-29T22:43:38Z DEBUG args=/usr/bin/certutil -d > dbm:/etc/pki/pki-tomcat/alias -L -f /etc/pki/pki-tomcat/alias/pwdfile.txt > 2020-06-29T22:43:38Z DEBUG Process finished, return code=0 > 2020-06-29T22:43:38Z DEBUG stdout= > Certificate Nickname                    >  Trust > Attributes > >  SSL,S/MIME,JAR/XPI > > caSigningCert cert-pki-ca                  >  CTu,Cu,Cu > subsystemCert cert-pki-ca                  >  u,u,u > Server-Cert cert-pki-ca                   >  u,u,u > ocspSigningCert cert-pki-ca                  > u,u,u > auditSigningCert cert-pki-ca                 > u,u,Pu > > 2020-06-29T22:43:38Z DEBUG stderr= > 2020-06-29T22:43:38Z INFO Certmonger certificate renewal configuration > already up-to-date > 2020-06-29T22:43:38Z INFO [Enable PKIX certificate path discovery and > validation] > 2020-06-29T22:43:38Z DEBUG Loading StateFile from > '/var/lib/ipa/sysupgrade/sysupgrade.state' > 2020-06-29T22:43:38Z INFO PKIX already enabled > 2020-06-29T22:43:38Z INFO [Authorizing RA Agent to modify profiles] > 2020-06-29T22:43:38Z INFO [Authorizing RA Agent to manage lightweight CAs] > 2020-06-29T22:43:38Z INFO [Ensuring Lightweight CAs container exists in > Dogtag database] > 2020-06-29T22:43:38Z DEBUG Created connection context.ldap2_140346851657552 > 2020-06-29T22:43:38Z DEBUG flushing > ldapi://%2fvar%2frun%2fslapd-FAKE-IPA-DOMAIN-LAN.socket from SchemaCache > 2020-06-29T22:43:38Z DEBUG retrieving schema for SchemaCache > url=ldapi://%2fvar%2frun%2fslapd-FAKE-IPA-DOMAIN-LAN.socket > conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7fa50c3e8e60> > 2020-06-29T22:43:39Z DEBUG Destroyed connection > context.ldap2_140346851657552 > 2020-06-29T22:43:39Z INFO [Adding default OCSP URI configuration] > 2020-06-29T22:43:39Z INFO [Ensuring CA is using LDAPProfileSubsystem] > 2020-06-29T22:43:39Z INFO [Migrating certificate profiles to LDAP] > 2020-06-29T22:43:39Z DEBUG Created connection context.ldap2_140346825804304 > 2020-06-29T22:43:39Z DEBUG flushing > ldapi://%2fvar%2frun%2fslapd-FAKE-IPA-DOMAIN-LAN.socket from SchemaCache > 2020-06-29T22:43:39Z DEBUG retrieving schema for SchemaCache > url=ldapi://%2fvar%2frun%2fslapd-FAKE-IPA-DOMAIN-LAN.socket > conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7fa50ac19b90> > 2020-06-29T22:43:39Z DEBUG Destroyed connection > context.ldap2_140346825804304 > 2020-06-29T22:43:39Z DEBUG request GET > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffake-ipa-host.fake-ipa-domain.lan%3A8443%2Fca%2Frest%2Faccount%2Flogin&data=02%7C01%7C%7C474697e47e794ce1189c08d81cd1f156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291035300787897&sdata=iaoyf6VSEUgKUjT9%2F%2Bp6EFAqL5BEXGxhSdx59V1W%2BnE%3D&reserved=0 > 2020-06-29T22:43:39Z DEBUG request body '' > 2020-06-29T22:43:39Z DEBUG httplib request failed: > Traceback (most recent call last): >  File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line > 220, in _httplib_request >   conn.request(method, path, body=request_body, headers=headers) >  File "/usr/lib64/python2.7/httplib.py", line 1056, in request >   self._send_request(method, url, body, headers) >  File "/usr/lib64/python2.7/httplib.py", line 1090, in _send_request >   self.endheaders(body) >  File "/usr/lib64/python2.7/httplib.py", line 1052, in endheaders >   self._send_output(message_body) >  File "/usr/lib64/python2.7/httplib.py", line 890, in _send_output >   self.send(msg) >  File "/usr/lib64/python2.7/httplib.py", line 852, in send >   self.connect() >  File "/usr/lib64/python2.7/httplib.py", line 1275, in connect >   server_hostname=sni_hostname) >  File "/usr/lib64/python2.7/ssl.py", line 348, in wrap_socket >   _context=self) >  File "/usr/lib64/python2.7/ssl.py", line 609, in __init__ >   self.do_handshake() >  File "/usr/lib64/python2.7/ssl.py", line 831, in do_handshake >   self._sslobj.do_handshake() > SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed > (_ssl.c:618) > 2020-06-29T22:43:39Z ERROR IPA server upgrade failed: Inspect > /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. > 2020-06-29T22:43:39Z DEBUG  File > "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in > execute >   return_value = self.run() >  File > "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", > line 54, in run >   server.upgrade() >  File > "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", > line 2166, in upgrade >   upgrade_configuration() >  File > "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", > line 2038, in upgrade_configuration >   ca_enable_ldap_profile_subsystem(ca) >  File > "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", > line 425, in ca_enable_ldap_profile_subsystem >   cainstance.migrate_profiles_to_ldap() >  File > "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line > 2027, in migrate_profiles_to_ldap >   _create_dogtag_profile(profile_id, profile_data, overwrite=False) >  File > "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line > 2033, in _create_dogtag_profile >   with api.Backend.ra_certprofile as profile_api: >  File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", > line 1311, in __enter__ >   method='GET' >  File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line > 167, in https_request >   method=method, headers=headers) >  File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line > 229, in _httplib_request >   raise NetworkError(uri=uri, error=str(e)) > > 2020-06-29T22:43:39Z DEBUG The ipa-server-upgrade command failed, > exception: NetworkError: cannot connect to > 'https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffake-ipa-host.fake-ipa-domain.lan%3A8443%2Fca%2Frest%2Faccount%2Flogin&data=02%7C01%7C%7C474697e47e794ce1189c08d81cd1f156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291035300787897&sdata=iaoyf6VSEUgKUjT9%2F%2Bp6EFAqL5BEXGxhSdx59V1W%2BnE%3D&reserved=0': > [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) > 2020-06-29T22:43:39Z ERROR Unexpected error - see > /var/log/ipaupgrade.log for details: > NetworkError: cannot connect to > 'https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffake-ipa-host.fake-ipa-domain.lan%3A8443%2Fca%2Frest%2Faccount%2Flogin&data=02%7C01%7C%7C474697e47e794ce1189c08d81cd1f156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291035300787897&sdata=iaoyf6VSEUgKUjT9%2F%2Bp6EFAqL5BEXGxhSdx59V1W%2BnE%3D&reserved=0': > [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) > 2020-06-29T22:43:39Z ERROR The ipa-server-upgrade command failed. See > /var/log/ipaupgrade.log for more information > > > What should be my next debug steps? > Hi, I would check whether any certificate expired: $ getcert list Look specifically for the "status: " and "expires: " labels. If some certs have expired, you will need to find the CA renewal master and fix this host first. To find the CA renewal master: $ kinit admin $ ipa config-show | grep "CA renewal" If you need help, please mention: - the output of "ipa server-role-find" - the output of "getcert list" on all the server nodes - are the httpd and ldap server certificates issued by IPA CA or by an external Certificate Authority? HTH, flo > Thanks in advance, > -ms > > > _______________________________________________ > FreeIPA-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Fedora Code of Conduct: > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.fedoraproject.org%2Fen-US%2Fproject%2Fcode-of-conduct%2F&data=02%7C01%7C%7C474697e47e794ce1189c08d81cd1f156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291035300787897&sdata=eZhJR06t5Pi280VE7SCAfBX6AzurzSA3e5qcbSNGHiE%3D&reserved=0 > List Guidelines: > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffedoraproject.org%2Fwiki%2FMailing_list_guidelines&data=02%7C01%7C%7C474697e47e794ce1189c08d81cd1f156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291035300787897&sdata=UvxQm1OecFaStjKLSSIMoIJ72IZgDnjv8Pmq9uPeL9s%3D&reserved=0 > List Archives: > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.fedorahosted.org%2Farchives%2Flist%2Ffreeipa-users%40lists.fedorahosted.org&data=02%7C01%7C%7C474697e47e794ce1189c08d81cd1f156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291035300787897&sdata=ItMCi20mfQLIoEorvQ20Fau0PGGFmRpgVAvbkgvAhMY%3D&reserved=0 >
_______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected]
