Scott Z. via FreeIPA-users wrote: > On the failing node, the output of "getcert list" does not show any > expired certs. I have hand-copied the info info this email below (it's > interesting to note that while the other IdM servers are tracking 9 > certs, the problem server is only tracking 8):
The tomcat Server-Cert is not tracked. To confirm: # getcert list -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' Running ipa-server-upgrade should detect and fix the missing tracking. To see if it is expired: # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' rob > > Number of certificates and requests being tracked: 8 > Request ID '<###>': > status: MONITORING > stuck: no > key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' > certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' > CA: SelfSign > issuer: CN=<servername>,O=<domain> > subject: CN=<servername>,O=<domain> > expires: 2020-09-12 19:51:34 UTC > principal name: krbtgt/<domain> > certificate template/profile: KDCs_PKINIT_Certs > pre-save command: > post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert > track: yes > auto-renew: yes > > Request ID '<###>': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > cert-pki-ca',token=NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > cert-pki-ca',token=NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=Certificate Authority,O=<domain> > subject: CN=CA Audit,O=<domain> > expires: 2021-08-10 17:20:21 UTC > key usage: digitialSignature,nonRepudiation > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "auditSigningCert cert-pki-ca" > track: yes > auto-renew: yes > > Request ID '<###>': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert > cert-pki-ca',token=NSS > Certificate DB',pin set > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert > cert-pki-ca',token=NSS > Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=Certificate Authority,O=<domain> > subject: CN=OCSP Subsystem,O=<domain> > expires: 2021-08-10 17:19:42 UTC > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "subsystemCert cert-pki-ca" > track: yes > auto-renew: yes > > Request ID '<###>': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert > cert-pki-ca',token=NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert > cert-pki-ca',token=NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=Certificate Authority,O=<domain> > subject: CN=CA Subsystem,O=<domain> > expires: 2021-08-10 17:19:51 UTC > key usage: > digitialSignature,nonRepudiation,keyEnchipherment,dataEncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "subsystemCert cert-pki-ca" > track: yes > auto-renew: yes > > Request ID '<###>': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert > cert-pki-ca',token=NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert > cert-pki-ca',token=NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=Certificate Authority,O=<domain> > subject: CN=Certificate Authority,O=<domain> > expires: 2037-09-28 14:29:02 UTC > key usage: digitialSignature,nonRepudiation,keyCertSign,cRLSign > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "caSigningCert cert-pki-ca" > track: yes > auto-renew: yes > > Request ID '<###>': > status: MONITORING > stuck: no > key pair storage: type=NSSDB,location='/var/lib/ipa/ra-agent.key' > certificate: type=NSSDB,location='/var/lib/ipa/ra-agent.pem' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=Certificate Authority,O=<domain> > subject: CN=IPA RA,,O=<domain> > expires: 2021-08-10 17:20:41 UTC > key usage: > digitialSignature,nonRepudiation,keyEnchipherment,dataEncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre > post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert > track: yes > auto-renew: yes > > Request ID '<###>': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-<domain>',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-<domain>/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-<domain>',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O=<domain> > subject: CN=<server>,O=<domain> > expires: 2021-09-09 19:53:33 UTC > principal name: ldap/<serverFQDN@domain> > key usage: > digitialSignature,nonRepudiation,keyEnchipherment,dataEncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv <domain> > track: yes > auto-renew: yes > > Request ID '<###>': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-<domain>',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=Certificate Authority,O=<domain> > subject: CN=<server>,O=<domain> > expires: 2021-09-09 19:51:45 UTC > principal name: HTTP/<serverFQDN@domain> > key usage: > digitialSignature,nonRepudiation,keyEnchipherment,dataEncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/libexec/ipa/certmonger/restart_httpd > track: yes > auto-renew: yes > > Thank you so much again! > Scot > > > > ------------------------------------------------------------------------ > *From:* Florence Blanc-Renaud <[email protected]> > *Sent:* Thursday, August 6, 2020 2:46 AM > *To:* FreeIPA users list <[email protected]> > *Cc:* Scott Z. <[email protected]> > *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting >  > On 8/6/20 12:53 AM, Scott Z. via FreeIPA-users wrote: >> Thanks much for the assistance. Here is where I am with your suggestions: >> 1) Checked on the cert with "certutil -L -d /etc/pki/pki-tomcat/alias -n >> 'Server-Cert cert-pki-ca' and I see that the Validity is indeed old >> (almost a year old actually, I assume IPA only checks it when it first >> starts up so it didn't care that it was expired until the server was >> rebooted?) > > certmonger checks the certificate validity periodically (configurable in > certmonger.conf) and tries multiple times to renew soon-to-expire certs. > The system probably had an issue that was not detected and the cert > reached its expiration date. > >> >> 2) ran ipactl start --ignore-service-failures >> Â Â Â Â Â Â a. most services started, obviously pki-tomcatd >>did not >> 3) ran "kinit admin" >> Â Â Â Â Â Â a. was forced to change the password, but >>otherwise nothing happened >> 4) Ran "ipa config-show |grep -i master >> Â Â Â Â Â a. I see that the IPA CA renewal master is a >>different idm machine. >> 5) Ran "getcert list | grep -E "Request|certificate:|expires:" >> Â Â Â Â Â a.I see all certs are currently valid (none expired) >> 6) Ran the command "getcert list" on the problem server, but I cannot >> paste the output here because it's on an airgaped environment so while I >> apologize for this and realize it makes things more difficult, perhaps >> if you tell me what I should be looking for or more specifically what >> you're interested in I can pluck that out and manually include it here? >> So in summary, it is indeed an expired "Server-Cert cert-pki-ca' >> certificate on the problem server, and it can theoretically be renew by >> the Master at this time. > The interesting part is the list of expired certs on the failing node > (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed > instructions are available here: > https://access.redhat.com/solutions/3357331 How do I manually renew > Identity Management (IPA) certificates on RHEL7 after they have expired? > (Replica IPA Server) > > flo > >> Many thanks! >> Scott >> >> ------------------------------------------------------------------------ >> *From:* Florence Blanc-Renaud <[email protected]> >> *Sent:* Monday, August 3, 2020 9:34 PM >> *To:* FreeIPA users list <[email protected]> >> *Cc:* Scott Z. <[email protected]> >> *Subject:* Re: [Freeipa-users] pki-tomcatd not starting >> On 8/3/20 10:14 PM, Scott Z. via FreeIPA-users wrote: >>> Not sure I'm sending this to the right place, but here it goes. I >>> inherited a FreeIPA/Identity Manager setup in an enclave (no internet >>> access) environment that is running into problems. There are at >>> least 3 >>> different IdM servers running in the environment spread out across >>> different geographical areas. One of those areas suffered an >>> unschedule >>> power outage recently, and ever since we brought everything back up, the >>> IdM server for this region is having an issue. Please bear with me >>> as I >>> have zero formal experience, training, or real knowledge with IdM. >>> >>> Logging in to the serverv (it's a VM server, running Centos 7.5), I run >>> "ipactl status" and it shows "Directory Service: STOPPED". I then >>> run >>> "ipactl restart", and things go fine until it gets to "Starting >>> pki-tomcatd Service", where it hangs for quite some time before failing >>> to start and killing all the other services. I check the log at >>> /var/log/pki/pki-tomcat/ca/debug and I see various errors such as >>> (forgive any mistypings, I have to manually type these in as I can't >>> import or screen capure the logs and put them in this message): >>> "/java.lang.Exception: Certificate Server-Cert cert-pki-ca is invalid: >>> Invalid certificate: (-8181) Peer's Certificate has expired/" >>> And slightly further down in the same log: >>> "/Cannot reset factory: connections not all returned/" >>> "/CertificateAuthority.shutdown: failed to reset dbFactory: Cannot reset >>> LDAP connection factory because some connections are still outstanding/" >>> ... still further down" >>> "/returnConn:mNumConns now 3 Invalid class name repositorytop/" >>> >>> Assuming I have some weird certificate issue with this server in >>> particular, I try to run a few more commands: >>> "certutil -L -d /etc/httpd/alias" --> returns a Server-Cert >>> listing >>> with u,u,u as it's trust attributes, and <IDM.domain> IPA CA with CT,C,C >>> for it's attributes. Comparing to a second IdM server in this >>> environment, it seems to be missing a "Signing-Cert"? >>> >> Hi, >> PKI is using the NSSDB in /etc/pki/pki-tomcat/alias, and its server cert >> has the nickname 'Server-Cert cert-pki-ca'. You should check that this >> one is not expired with: >> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' >> | grep 'Not ' >> >> If the certificate is indeed expired, it will have to be renewed but you >> need first to find which IPA server is the CA renewal master. On your >> server, force a service start and check the CA renewal master: >> # ipactl start --ignore-service-failures >> # kinit admin >> # ipa config-show | grep "renewal master" >> Â Â IPA CA renewal master: server.domain.com >> >> You need to make sure that all the certificates are valid on the CA >> renewal master: >> (on the CA renewal master)# getcert list | grep -E >> "Request|certificate:|expires:" >> >> - if the CA renewal master is not OK, please post the output of "# >> getcert list" (without the grep) on the CA renewal master. This node >> will have to be repaired first. >> - if the CA renewal master is OK, please post the output of "# getcert >> list" (also without the grep) on the failing node. >> >> We'll be able to help based on this information. >> flo >> >>> I also did a "getcert list", and all certs it has show that they expire >>> in the future (nothing shows as bein currently expired). >>> >>> I'm confused; it seems to that it is seeing an expired cert *somewhere*, >>> but how do I track down which 'peer' the log file is talking about that >>> has an expired cert? Meanwhile none of the linux clients that >>> point to >>> this IdM server are allowing people to log in/authenticate. >>> Many thanks for any help! >>> Scott >>> >>> >>> _______________________________________________ >>> FreeIPA-users mailing list -- [email protected] >>> To unsubscribe send an email to [email protected] >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedorahosted.org/archives/list/[email protected] >>> >> >> >> _______________________________________________ >> FreeIPA-users mailing list -- [email protected] >> To unsubscribe send an email to [email protected] >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedorahosted.org/archives/list/[email protected] >> > > > _______________________________________________ > FreeIPA-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/[email protected] > _______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected]
