Hi, it looks like some of the certificates used by PKI are also expired (they are stored in /etc/pki/pki-tomcat/alias). Since you're running IPA 4.9, you can use the command ipa-cert-fix. Please read the man page with extra care, it recommends to backup certificates and keys before you proceed. You mentioned having a pair of IPA servers, do they both have expired certificates? If one of them is good, there are also other options to retrieve the renewed certificates from the good server and install them on the other one (the 3 certs ocspSigningCert, subsystemCert and auditSigningCert are shared on all the CA instances).
flo On Fri, Mar 11, 2022 at 2:36 PM Morgan Cox via FreeIPA-users < [email protected]> wrote: > > > Hi > > We have a pair of Freeipa (4.9.x) Rhel8 Freeipa servers. > > Previously we had installed a 3rd party cert for httpd + dirsrv (only) - > this expired recently. I was unable to login to ui . This issue however may > not be connected with this. It appears to be linked to Tomcat -> LDAPS > connectiopn ?? - error when trying to login was 'Login failed due to an > unknown reason' > > I could login if I changed server time to the past - but the certificates > page is broken 'Certificate operation cannot be completed: Unable to > communicate with CMS (503)' (time has been set back to normal now) > > As a result I cannot renew my httpd/dirsv cert > > Can anyone help me restore pki-tomcatd ? This may not be connected to > web/dirsv cert expiry (and just be a coincidence) > > If I try using > > # ipa-server-certinstall --http --dirsrv ireland.idm.domain.uk.key > ireland.idm.domain.uk.crt > > I get > > ----- > > Directory Manager password: > > Enter private key unlock password: > > cannot connect to 'https://london.idm.domain.uk:443/acme/directory': > [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897) > The ipa-server-certinstall command failed. > > ---- > > I can however install the cert to just the dirsv > > --- > [root@london mcox]# ipa-server-certinstall --dirsrv > london.idm.domain.uk.key london.idm.domain.uk.crt > Directory Manager password: > > Enter private key unlock password: > > > Please restart ipa services after installing certificate (ipactl restart) > --- > > However after ipactl restart -> pki-tomcatd Service: STOPPED (all other > services are working) > > The main IPA system aside from this appears to work - i.e I can login and > sudo to clients, and kinit, etc works > > As a work-around I can login to the UI if I manually copy the cert/key to > > > /var/lib/ipa/certs/httpd.crt > /var/lib/ipa/private/httpd.key > > However the pki-tomcatd service is still down - I see these errors > > - On certifcates tab : IPA Error 4301: CertificateOperationError - > Certificate operation cannot be completed: Unable to communicate with CMS > (503) > - On Certificate authorities pages I see : Some operations failed -> > details -> Failed to authenticate to CA REST API > > pki-tomcatd logs show > > ------- > Mar 11 12:54:10 london.idm.domain.uk systemd[1]: Starting PKI Tomcat > Server pki-tomcat... > Mar 11 12:54:15 london.idm.domain.uk server[509585]: Java virtual machine > used: /usr/lib/jvm/jre-1.8.0-openjdk/bin/java > Mar 11 12:54:15 london.idm.domain.uk server[509585]: classpath used: > /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/ant.jar:/usr/share/java/ant-launcher.jar:/usr/lib/jvm/java/lib/tools.jar > Mar 11 12:54:15 london.idm.domain.uk server[509585]: main class used: > org.apache.catalina.startup.Bootstrap > Mar 11 12:54:15 london.idm.domain.uk server[509585]: flags used: > -Dcom.redhat.fips=false > Mar 11 12:54:15 london.idm.domain.uk server[509585]: options used: > -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat > -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp > -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > -Djava.security.manager > -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy > Mar 11 12:54:15 london.idm.domain.uk server[509585]: arguments used: start > Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]: > pki.client: /usr/libexec/ipa/ipa-pki-wait-running:63: The subsystem in > PKIConnection.__init__() has been deprecated ( > https://www.dogtagpki.org/wiki/PKI_10.8_Python_Changes). > Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]: > ipa-pki-wait-running: Created connection > http://london.idm.domain.uk:8080/ca > Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]: > ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host=' > london.idm.domain.uk', port=8080): Max retries exceeded with url: > /ca/admin/ca/getStatus (Caused by > NewConnectionError('<urllib3.connection.HTTPConnection object at > 0x7fb6c264e668>: Failed to establish a new connection: [Errno 111] > Connection refused',)) > Mar 11 12:54:17 london.idm.domain.uk ipa-pki-wait-running[509586]: > ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host=' > london.idm.domain.uk', port=8080): Max retries exceeded with url: > /ca/admin/ca/getStatus (Caused by > NewConnectionError('<urllib3.connection.HTTPConnection object at > 0x7fb6c264ea58>: Failed to establish a new connection: [Errno 111] > Connection refused',)) > Mar 11 12:54:18 london.idm.domain.uk server[509585]: WARNING: Some of the > specified [protocols] are not supported by the SSL engine and have been > skipped: [[TLSv1, TLSv1.1]] > Mar 11 12:54:19 london.idm.domain.uk ipa-pki-wait-running[509586]: > ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host=' > london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0) > Mar 11 12:54:21 london.idm.domain.uk ipa-pki-wait-running[509586]: > ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host=' > london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0) > Mar 11 12:54:23 london.idm.domain.uk server[509585]: SEVERE: One or more > listeners failed to start. Full details will be found in the appropriate > container log file > Mar 11 12:54:23 london.idm.domain.uk server[509585]: SEVERE: Context > [/ca] startup failed due to previous errors > Mar 11 12:54:23 london.idm.domain.uk server[509585]: WARNING: The web > application [ca] appears to have started a thread named [LDAPConnThread-0 > ldaps://london.idm.domain.uk:636] but has failed to stop it. This is very > likely to create a memory leak. Stack trace of thread: > Mar 11 12:54:23 london.idm.domain.uk server[509585]: > java.net.SocketInputStream.socketRead0(Native Method) > Mar 11 12:54:23 london.idm.domain.uk server[509585]: > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > Mar 11 12:54:23 london.idm.domain.uk server[509585]: > java.net.SocketInputStream.read(SocketInputStream.java:171) > Mar 11 12:54:23 london.idm.domain.uk server[509585]: > java.net.SocketInputStream.read(SocketInputStream.java:141) > Mar 11 12:54:23 london.idm.domain.uk server[509585]: > java.net.SocketInputStream.read(SocketInputStream.java:127) > Mar 11 12:54:23 london.idm.domain.uk server[509585]: > org.mozilla.jss.ssl.SSLSocket.socketRead(Native Method) > Mar 11 12:54:23 london.idm.domain.uk server[509585]: > org.mozilla.jss.ssl.SSLSocket.read(SSLSocket.java:1505) > Mar 11 12:54:23 london.idm.domain.uk server[509585]: > org.mozilla.jss.ssl.SSLInputStream.read(SSLInputStream.java:43) > Mar 11 12:54:23 london.idm.domain.uk server[509585]: > java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > Mar 11 12:54:23 london.idm.domain.uk server[509585]: > java.io.BufferedInputStream.read(BufferedInputStream.java:265) > Mar 11 12:54:23 london.idm.domain.uk server[509585]: > netscape.ldap.ber.stream.BERElement.getElement(Unknown Source) > Mar 11 12:54:23 london.idm.domain.uk server[509585]: > netscape.ldap.LDAPConnThread.run(Unknown Source) > Mar 11 12:54:23 london.idm.domain.uk server[509585]: > java.lang.Thread.run(Thread.java:748) > Mar 11 12:54:23 london.idm.domain.uk ipa-pki-wait-running[509586]: > ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host=' > london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0) > Mar 11 12:54:24 london.idm.domain.uk server[509585]: SEVERE: One or more > listeners failed to start. Full details will be found in the appropriate > container log file > Mar 11 12:54:24 london.idm.domain.uk server[509585]: SEVERE: Context > [/acme] startup failed due to previous errors > Mar 11 12:54:24 london.idm.domain.uk server[509585]: WARNING: The web > application [acme] appears to have started a thread named [LDAPConnThread-1 > ldaps://london.idm.domain.uk:636] but has failed to stop it. This is very > likely to create a memory leak. Stack trace of thread: > Mar 11 12:54:24 london.idm.domain.uk server[509585]: > java.net.SocketInputStream.socketRead0(Native Method) > Mar 11 12:54:24 london.idm.domain.uk server[509585]: > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > Mar 11 12:54:24 london.idm.domain.uk server[509585]: > java.net.SocketInputStream.read(SocketInputStream.java:171) > Mar 11 12:54:24 london.idm.domain.uk server[509585]: > java.net.SocketInputStream.read(SocketInputStream.java:141) > Mar 11 12:54:24 london.idm.domain.uk server[509585]: > java.net.SocketInputStream.read(SocketInputStream.java:127) > Mar 11 12:54:24 london.idm.domain.uk server[509585]: > org.mozilla.jss.ssl.SSLSocket.socketRead(Native Method) > Mar 11 12:54:24 london.idm.domain.uk server[509585]: > org.mozilla.jss.ssl.SSLSocket.read(SSLSocket.java:1505) > Mar 11 12:54:24 london.idm.domain.uk server[509585]: > org.mozilla.jss.ssl.SSLInputStream.read(SSLInputStream.java:43) > Mar 11 12:54:24 london.idm.domain.uk server[509585]: > java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > Mar 11 12:54:24 london.idm.domain.uk server[509585]: > java.io.BufferedInputStream.read(BufferedInputStream.java:265) > Mar 11 12:54:24 london.idm.domain.uk server[509585]: > netscape.ldap.ber.stream.BERElement.getElement(Unknown Source) > Mar 11 12:54:24 london.idm.domain.uk server[509585]: > netscape.ldap.LDAPConnThread.run(Unknown Source) > Mar 11 12:54:24 london.idm.domain.uk server[509585]: > java.lang.Thread.run(Thread.java:748) > Mar 11 12:54:25 london.idm.domain.uk ipa-pki-wait-running[509586]: > ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error: for > url: http://london.idm.domain.uk:8080/ca/admin/ca/getStatus > Mar 11 12:54:26 london.idm.domain.uk ipa-pki-wait-running[509586]: > ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error: for > url: http://london.idm.domain.uk:8080/ca/admin/ca/getStatus > Mar 11 12:54:27 london.idm.domain.uk ipa-pki-wait-running[509586]: > ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error: for > url: http://london.idm.domain.uk:8080/ca/admin/ca/getStatus > ... > ------- > > Other logs show : (i've just added the main error - not entire java error > > /var/log/pki/pki-tomcat/acme/debug.2022-03-11.log : > > ----- > 12:34:01 [main] SEVERE: Exception sending context initialized event to > listener instance of class [org.dogtagpki.acme.server.ACMEEngine] > java.lang.RuntimeException: Unable to start ACME engine: Unable to connect > to LDAP server: Authentication failed > ----- > > /var/log/pki/pki-tomcat/ca/debug.2022-03-11.log : > > ----- > 2022-03-11 12:33:59 [main] SEVERE: Unable to start CA engine: Unable to > connect to LDAP server: Authentication failed > Unable to connect to L2022-03-11 12:33:59 [main] INFO: Shutting down CA > subsystem > .... > 2022-03-11 12:33:59 [main] SEVERE: Exception sending context destroyed > event to listener instance of class [org.dogtagpki.server.ca.CAEngine] > java.lang.NullPointerException > DAP server: Authentication failed > ----- > > I have checked this -> > > # getcert list |grep expire > expires: 2024-02-13 00:32:37 GMT > expires: unknown > expires: unknown > expires: unknown > expires: unknown > expires: 2024-01-22 00:29:51 GMT > expires: 2024-01-22 00:30:38 GMT > > > > And I have ran ipa-healthcheck > > I can see > > ---- > Expired Cert: ocsp_signing > Expired Cert: subsystem > Expired Cert: audit_signing > > Internal server error 503 Server Error: Service Unavailable for url: > http://london.idm.domain.uk:80/ca/rest/securityDomain/domainInfo > Internal server error HTTPSConnectionPool(host='london.idm.domain.uk', > port=8443): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused > by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object > at 0x7fc4e6c58198>: Failed to establish a new connection: [Errno 111] > Connection refused',)) > > --- > > Also some expired certs > > "source": "pki.server.healthcheck.certs.expiration", > "check": "CASystemCertExpiryCheck", > "result": "ERROR", > "uuid": "36c8fbed-571d-4d38-9919-53322fea4aa2", > "when": "20220311130832Z", > "duration": "0.188329", > "kw": { > "cert_id": "ocsp_signing", > "expiry_date": "Mar 01 2022", > "msg": "Certificate has ALREADY EXPIRED" > } > }, > { > "source": "pki.server.healthcheck.certs.expiration", > "check": "CASystemCertExpiryCheck", > "result": "ERROR", > "uuid": "195970e4-e2fd-4eca-aeac-f1e97e9c3b13", > "when": "20220311130832Z", > "duration": "0.360146", > "kw": { > "cert_id": "subsystem", > "expiry_date": "Mar 01 2022", > "msg": "Certificate has ALREADY EXPIRED" > } > }, > { > "source": "pki.server.healthcheck.certs.expiration", > "check": "CASystemCertExpiryCheck", > "result": "ERROR", > "uuid": "a84a9bc5-de4d-4cdc-b7fd-41b83f3a11af", > "when": "20220311130833Z", > "duration": "0.454225", > "kw": { > "cert_id": "audit_signing", > "expiry_date": "Mar 01 2022", > "msg": "Certificate has ALREADY EXPIRED" > } > > > I have attached the full output of healthcheck to : > https://pastebin.com/xfNLR0Ja (domain name changed) > > On the last ipa update there was also a issue with pki-tomcatd - i.e - I > have to remove the block 'requiredSecret=' in > /etc/pki/pki-tomcat/server.xml to fix it, this was however working for a > month or so after . > > Any help to troubleshooting this would be welcomed > > Thanks > _______________________________________________ > FreeIPA-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/[email protected] > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure >
_______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected] Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
