Hi team,
ipa-healthcheck has been a great tool for us. I run it weekly on all IPA servers via cron. This week ipa-healthcheck reported errors on all IPA servers. Take IPA server ipa2 as an example for the investigation: [root@ipa2 ~]# ipa-healthcheck --failures-only --output-type=human ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040: Request id 20190425210040 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210052: Request id 20190425210052 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210053: Request id 20190425210053 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210054: Request id 20190425210054 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040: Request id 20190425210040 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210052: Request id 20190425210052 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210053: Request id 20190425210053 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210054: Request id 20190425210054 expires in 27 days ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description does not match 2;186;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and 2;66;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O= EXAMPLE.COM expected ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210052: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210053: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210054: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210055: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210056: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205849: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205831: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210120: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.dogtag.ca.DogtagCertsConnectivityCheck: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) [root@ipa2 ~]# The list of certs: [root@ipa2 ~]# getcert list Number of certificates and requests being tracked: 9. Request ID '20190425205831': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=ipa2.example.com,O=EXAMPLE.COM expires: 2023-03-29 21:37:22 UTC dns: ipa2.example.com principal name: ldap/[email protected] key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv EXAMPLE-COM track: yes auto-renew: yes Request ID '20190425205849': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=ipa2.example.com,O=EXAMPLE.COM expires: 2023-03-29 21:37:46 UTC dns: ipa2.example.com principal name: HTTP/[email protected] key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes Request ID '20190425210040': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=IPA RA,O=EXAMPLE.COM expires: 2022-05-11 03:40:55 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20190425210052': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=CA Audit,O=EXAMPLE.COM expires: 2022-05-11 03:40:05 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190425210053': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=OCSP Subsystem,O=EXAMPLE.COM expires: 2022-05-11 03:40:25 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190425210054': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=CA Subsystem,O=EXAMPLE.COM expires: 2022-05-11 03:40:05 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190425210055': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=Certificate Authority,O=EXAMPLE.COM expires: 2038-06-28 21:19:45 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190425210056': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=ipa2.example.com,O=EXAMPLE.COM expires: 2023-03-07 22:37:22 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes Request ID '20190425210120': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=ipa2.example.com,O=EXAMPLE.COM expires: 2023-03-29 21:37:52 UTC principal name: krbtgt/[email protected] key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes [root@ipa2 ~]# There are 4 certs which expire on 2022-05-11 which match "expires in 27 days". Take 20190425210040 as an example, we have: WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040: Request id 20190425210040 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040: Request id 20190425210040 expires in 27 days ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) Request ID '20190425210040': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=IPA RA,O=EXAMPLE.COM expires: 2022-05-11 03:40:55 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes I was able to manually renew it: [root@ipa2 ~]# ipa-getcert resubmit -i '20190425210040' Resubmitting "20190425210040" to "dogtag-ipa-ca-renew-agent". [root@ipa2 ~]# After renew, it "expires: 2024-04-02 06:09:32 UTC": Request ID '20190425210040': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=IPA RA,O=EXAMPLE.COM expires: 2024-04-02 06:09:32 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes How to fix the issue reported by ipa-healthcheck? And what is this issue about? All IPA servers are at same level: CentOS Linux release 7.9.2009 (Core) ipa-*server*.x86_64 4.6.8-5.el7.centos.7 *slapi-nis*.x86_64 0.56.5-3.el7_9 *389-ds-base*.x86_64 1.3.10.2-12.el7_9 *389-ds-base*-libs.x86_64 1.3.10.2-12.el7_9 Many thanks! Kathy.
_______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected] Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
