Cristian Le via FreeIPA-users wrote: > Ok, let me walk through some of the specific errors, and I will also > censor out some of the output since this is going to the public > mail-list as well. > > Starting from the beginning. > - I have set the date to `1 month` before certificate expired with `sudo > date` > - I ran `ipactl restart --force` and `pki-tomcatd` failed
If tomcat fails to start then you need to stop and figure out why. Any further poking at certificates won't yield anything useful without a working CA. > - I try to run `ipa-cacert-manage renew` with `--external-ca` and > `--external-cert-file`, with the CSR signed with `openssl x509 -req > -copy_extensions=copyall`, but this one failed because of `ERROR: CA > certificate chain in is incomplete: missing certificate with subject`, > even though I passed both the root and new CA What failed? > Then I have tried to manually add the new CA, but then, that did not > work. So moving on to the next layer, when I run to `sudo getcert list`, > the certificates I get: Manually add how? > ``` > Request ID '20220810070559': > status: CA_UNREACHABLE > ca-error: Error setting up ccache for "host" service on client > using default keytab: Cannot contact any KDC for requested realm. The KDC isn't running. > stuck: no > key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' > certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' > CA: IPA > issuer: CN=FreeIPA Root Certificate Authority,O=My Lab > subject: CN=ipa1.my.lab,O=My Lab > issued: 2023-09-20 14:21:40 UTC > expires: 2025-09-20 14:21:40 UTC > Request ID '20230920203122': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=FreeIPA Root Certificate Authority,O=My Lab > subject: CN=CA Audit,O=My Lab > issued: 2023-09-20 14:21:35 UTC > expires: 2025-09-09 14:21:35 UTC > Request ID '20230920203123': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS > Certificate DB',pin set > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS > Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=FreeIPA Root Certificate Authority,O=My Lab > subject: CN=OCSP Subsystem,O=My Lab > issued: 2023-09-20 14:21:33 UTC > expires: 2025-09-09 14:21:33 UTC > Request ID '20230920203125': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=FreeIPA Root Certificate Authority,O=My Lab > subject: CN=CA Subsystem,O=My Lab > issued: 2023-09-20 14:21:31 UTC > expires: 2025-09-09 14:21:31 UTC > Request ID '20230920203128': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert > cert-pki-ca',token='NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=FreeIPA Root Certificate Authority,O=My Lab > subject: CN=ipa1.my.lab,O=My Lab > issued: 2023-09-20 11:29:21 UTC > expires: 2023-12-20 11:29:21 UTC > Request ID '20230920203130': > status: MONITORING > stuck: no > key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' > certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=FreeIPA Root Certificate Authority,O=My Lab > subject: CN=IPA RA,O=My Lab > issued: 2023-09-20 14:21:36 UTC > expires: 2025-09-09 14:21:36 UTC > Request ID '20230920203131': > status: CA_UNREACHABLE > ca-error: Error setting up ccache for "host" service on client > using default keytab: Cannot contact any KDC for requested realm. > stuck: no > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-MY-LAB',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-MY-LAB/pwdfile.txt' > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-MY-LAB',nickname='Server-Cert',token='NSS > Certificate DB' > CA: IPA > issuer: CN=FreeIPA Root Certificate Authority,O=My Lab > subject: CN=ipa1.my.lab,O=My Lab > issued: 2023-09-20 14:21:39 UTC > expires: 2025-09-20 14:21:39 UTC > Request ID '20230920203134': > status: CA_UNREACHABLE > stuck: no > key pair storage: > type=FILE,location='/var/lib/ipa/private/httpd.key',pinfile='/var/lib/ipa/passwds/ipa1.my.lab-443-RSA' > certificate: type=FILE,location='/var/lib/ipa/certs/httpd.crt' > CA: IPA > issuer: CN=My Lab Root CA,O=My Lab > subject: CN=ipa1.my.lab,O=My Lab > issued: 2023-07-10 04:31:16 UTC > expires: 2023-08-09 04:31:16 UTC > Request ID '20230510042436': > status: MONITORING > ca-error: Updated certificate not available > stuck: no > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert > cert-pki-ca',token='NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=My Lab Root CA,O=My Lab > subject: CN=FreeIPA Root Certificate Authority,O=My Lab > issued: 2022-08-10 04:29:31 UTC > expires: 2023-08-10 04:29:31 UTC > ``` > Note that even the `MONITORING` ones are unusable due to the > non-overlapping date. So when I do something like `ipa-certupdate` I get: > `The ipa-certupdate command failed, exception: DatabaseError: Connect > error: error:0A000086:SSL routines::certificate verify failed (unable to > get issuer certificate)` when trying to use `ipaldap.py`. > > It is a mess, that's why I think it is better to have a way to simply > re-bootstrap all certificates. There isn't really a good way to start over. rob > > On 2023/09/22 15:57, Florence Blanc-Renaud wrote: >> Hi, >> >> On Fri, Sep 22, 2023 at 12:36 PM Cristian Le <[email protected]> wrote: >> >> Hi Florence, >> >> Thanks for the feedback, let me clarify the situation on the >> certificates: >> - External CA is still valid and it is a self-signed certificate >> that we use for other services. So we can manually sign any >> service certificates to get them back up and running >> - IPA CA is expired, let's say Aug/10 >> - I have managed to import a renewed IPA CA and ran `ipa-cert-fix` >> (and also seemed to have run `ipa-certupdate`) on a current date, >> let's say Sep/20. But not all services were recovered and now >> there is no overlap between earliest date in service certificates >> and the original IPA CA >> >> Which are the not-recovered services? Can you provide the output of >> "getcert list" at the current date? >> flo >> >> - I have run a backup, but also did some system upgrades to get >> the `ipa-cacert-manage prune` command, but when I've tried to >> recover it, I've found that the backup was not there. >> >> > you can still find the original certificates in the LDAP >> database (below ou=certificateRepository,ou=ca,o=ipaca) but it >> requires a bit of searching. You would need to restore the expired >> certificates, go back in time and force the renewal. >> >> I suspect we cannot do this all within LDAP right? If we get back >> the expired certificates, how do we restore them in each service? >> `httpd` is straightforward, and I guess `nssdb` should be doable, >> assuming the same key is used, but is there another database type >> where the certificates are located? Are all the certificates >> tracked by `getcert list`? Is it safe to assume that after running >> something like `ipa-cert-fix`, they are using the same private key? >> >> Some symptoms in the current setup: >> - When we are forward in time, `pki-tomcatd` is able to run, but >> then I can't do any `ipa-cert-fix` or `ipa-cacert-manage renew`. >> From what I've read, all of these commands (or at least >> `ipa-cacert-manage renew`) must be done backwards in time. >> - When we are backwards in time `pki-tomcatd` is unable to run, >> failing to access `:8080/ca/admin/ca/getStatus`. This then blocks >> various other services to be run. But about `ipactl restart`, only >> `pki-tomcatd` service is actually failing (and ipa service itself >> of course). >> >> I have navigated to `ou=certificateRepository,ou=ca,o=ipaca` and >> indeed there are still a bunch of certificates there in linear >> order. What are the services I should look for in there? I am >> using Apache Directory Studio and I can download the >> `userCertificate`. Should I just run `certutil -A` with those >> values with corresponding `subjectName`? >> >> BTW, I want to document this process on the website, should I make >> a PR on the github repo or is there somewhere else? >> >> Kind regards, >> Cristian >> >> On 2023/09/22 9:00, Florence Blanc-Renaud wrote: >>> Hi, >>> >>> On Thu, Sep 21, 2023 at 5:04 PM Cristian Le via FreeIPA-users >>> <[email protected]> wrote: >>> >>> I have tried my luck around with all the helpers: `pki-server >>> cert-fix`, `ipa-cacert-manage`, `ipa-certupdate`, etc. but >>> each one is failing on me for multiple reasons. >>> - `ipa-cacert-manage` Cannot update the CA with >>> `--external-cert-file` because the root ca is not detected to >>> be in the trust list >>> >>> This command is useful if you need to trust a new external CA or >>> renew IPA CA. Is your IPA CA expired? >>> >>> - `ipa-cert-fix` Was run without overlapping validity time, >>> and the certificate were re-created, so now it is not >>> recoverable, neither back in time, nor in current time >>> >>> It is recommended to do a backup before running ipa-cert-fix. If >>> you didn't, and want to try the back-in-time method, you can >>> still find the original certificates in the LDAP database (below >>> ou=certificateRepository,ou=ca,o=ipaca) but it requires a bit of >>> searching. You would need to restore the expired certificates, go >>> back in time and force the renewal. >>> >>> - `pki-tomcat` is failing >>> >>> What is the current situation? Which certs are expired (getcert >>> list)? If you start the services with "ipactl start >>> --ignore-service-failures", is pki-tomcat the only service failing? >>> flo >>> >>> It is quite a mess and I would like to ask for some guidance >>> on how one could recover manually from such dependency issues: >>> - Is it possible to do a `ipa-server-install` and keep the >>> user data? >>> >>> - If I sign all of the service's certificates manually, what >>> are all of the manual steps needed to get the services back >>> up so that the helpers can be run. >>> - I've tried to install the CA certificate in the nssdb >>> database, ldap, and /etc/ipa/ca.crt. Are there other locations? >>> - I've recreated an httpd certificate signed by the root, >>> but I can't figure how to do the same with the ones located >>> in the nssdb database, i.e. to recreate a csr with the same >>> data as one of the certificates there >>> - What is the order of services that should be updated. My >>> understanding is CA -> `certutil`'s CA -> httpd + slapd + >>> pki-tomcat (not sure where the last one is or how to edit it) >>> -> `ipa-certupdate` >>> _______________________________________________ >>> FreeIPA-users mailing list -- >>> [email protected] >>> To unsubscribe send an email to >>> [email protected] >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> >>> https://lists.fedorahosted.org/archives/list/[email protected] >>> Do not reply to spam, report it: >>> https://pagure.io/fedora-infrastructure/new_issue >>> > > _______________________________________________ > FreeIPA-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/[email protected] > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue > _______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected] Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
