Ilya Kogan wrote: > Wow ok, that was easy. `getcert list` now reports correct expiration > dates for those certificates and they're all in MONITORING. It still has > that ca-error field although it's no longer trying to renew. Is that > going to be an issue or is it just going to try again when it's time to > renew and succeed?
I don't know. I'd check the journal to see if it logged any errors post-restart. I don't believe that the ca-error is stored between restarts. You could grep in /var/lib/certmonger/requests to see I suppose. rob > > On Wed, Jul 8, 2020, 9:43 AM Florence Blanc-Renaud <[email protected] > <mailto:[email protected]>> wrote: > > On 7/6/20 7:59 PM, Ilya Kogan via FreeIPA-users wrote: > > Hi, > > > > Thanks for the help so far! I've actually run `ipa-cert-fix` on both > > nodes, it says everything is ok on both nodes. When I run it with > > verbose mode, it spits out the command it's running and the > certificate > > it got, for example: > > > > ``` > > ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d', > > 'sql:/etc/pki/pki-tomcat/alias', '-L', '-n', 'storageCert > > cert-pki-kra', '-a', '-f', > '/etc/pki/pki-tomcat/alias/pwdfile.txt'] > > ``` > > > > > > If I then take that cert and ask what `openssl x509 -text -noout` > thinks > > about it, it tells me that it's valid from 2020-06-29 to 2022-06-29. > > Strangely, though, when I ask `getcert list`, it shows that the > certificate: > > > > ``` > > certificate: > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > cert-pki-kra',token='NSS > > Certificate DB' > > ``` > > > > > > expires on 2020-06-27. It's almost as if this node's certificate has > > _already_ been renewed but certmonger (I think) doesn't know about > it, > > which might be why it's having trouble renewing it. > > > Hi, > > you may want to restart certmonger to force it re-reading the > certificate information: > # sudo systemctl restart certmonger > > flo > > > Here's what the two nodes say about replication: > > > > From node one: > > > > ``` > > ipa-two.mydomain.org <http://ipa-two.mydomain.org> > <http://ipa-two.mydomain.org> > > last update status: Error (0) Replica acquired successfully: > > Incremental update succeeded > > last update ended: 2020-07-06 17:46:17+00:00 > > ``` > > > > > > From node two: > > > > ``` > > ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> > <http://ipa-one.gaea.mythicnet.org> > > last update status: Error (0) Replica acquired successfully: > > Incremental update succeeded > > last update ended: 2020-07-06 17:46:17+00:00 > > ``` > > > > > > I suppose this might be a good time to mention that this is a > simple two > > node multi-master setup. Finally, I'm not sure if I'm doing this > > correctly, but to make absolutely sure about which node is the > renewal > > master, I ran this on both nodes: > > > > ``` > > ldapsearch -H ldap://ipa-one.gaea.mythicnet.org > <http://ipa-one.gaea.mythicnet.org> > > <http://ipa-one.gaea.mythicnet.org> -D 'cn=Directory Manager' > -W -b > > 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' > > '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn > > ldapsearch -H ldap://ipa-two.gaea.mythicnet.org > <http://ipa-two.gaea.mythicnet.org> > > <http://ipa-two.gaea.mythicnet.org> -D 'cn=Directory Manager' > -W -b > > 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' > > '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn > > ``` > > > > > > The result for both is: > > > > ``` > > dn: cn=CA,cn=ipa-one.gaea.mythicnet.org > <http://ipa-one.gaea.mythicnet.org> > > > > <http://ipa-one.gaea.mythicnet.org>,cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org > > ``` > > > > > > So it looks like the renewal master is the one having this problem. > > > > > > Ilya Kogan > > w: github.com/ikogan <http://github.com/ikogan> > <http://github.com/ikogan> e: [email protected] > <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>> > > <http://twitter.com/ilkogan> <https://www.linkedin.com/in/ilyakogan/> > > > > > > > > On Mon, Jul 6, 2020 at 1:25 PM Rob Crittenden <[email protected] > <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > > > Florence Blanc-Renaud via FreeIPA-users wrote: > > > On 7/5/20 2:23 AM, Ilya Kogan via FreeIPA-users wrote: > > >> Hi, > > >> > > >> I seem to be facing a similar issue with one of my KRAs. > My KRA > > >> certificates were, for some reason, not automatically > renewed when > > >> they expired last month. Using `ipa-cert-fix` correctly fixed > > them on > > >> _one_ host. On the other, they seem to be stuck in the > renewal state > > >> and `ipa-cert-fix` claims there's nothing to do: > > >> > > >> ``` > > >> Request ID '20191031183458': > > >> status: MONITORING > > >> ca-error: Server at > > >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit"; > replied: > > >> Missing credential: sessionID > > >> stuck: no > > >> key pair storage: > > >> > > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > > >> cert-pki-kra',token='NSS Certificate DB',pin set > > >> certificate: > > >> > > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > > >> cert-pki-kra',token='NSS Certificate DB' > > >> CA: dogtag-ipa-ca-renew-agent > > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG > <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> > > >> <http://MYDOMAIN.ORG> > > >> subject: CN=KRA Audit,O=MYDOMAIN.ORG > <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> <http://MYDOMAIN.ORG> > > >> expires: 2020-06-27 01:54:34 EDT > > >> key usage: digitalSignature,nonRepudiation > > >> pre-save command: > /usr/libexec/ipa/certmonger/stop_pkicad > > >> post-save command: > > /usr/libexec/ipa/certmonger/renew_ca_cert > > >> "auditSigningCert cert-pki-kra" > > >> track: yes > > >> auto-renew: yes > > >> Request ID '20191031183459': > > >> status: MONITORING > > >> ca-error: Server at > > >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit"; > replied: > > >> Missing credential: sessionID > > >> stuck: no > > >> key pair storage: > > >> > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert > > cert-pki-kra',token='NSS > > >> Certificate DB',pin set > > >> certificate: > > >> > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert > > cert-pki-kra',token='NSS > > >> Certificate DB' > > >> CA: dogtag-ipa-ca-renew-agent > > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG > <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> > > >> <http://MYDOMAIN.ORG> > > >> subject: CN=KRA Transport > Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> > > >> <http://MYDOMAIN.ORG> > > >> expires: 2020-06-27 01:54:30 EDT > > >> key usage: > > >> > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > >> eku: id-kp-clientAuth > > >> pre-save command: > /usr/libexec/ipa/certmonger/stop_pkicad > > >> post-save command: > > /usr/libexec/ipa/certmonger/renew_ca_cert > > >> "transportCert cert-pki-kra" > > >> track: yes > > >> auto-renew: yes > > >> Request ID '20191031183500': > > >> status: MONITORING > > >> ca-error: Server at > > >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit"; > replied: > > >> Missing credential: sessionID > > >> stuck: no > > >> key pair storage: > > >> > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > > >> cert-pki-kra',token='NSS Certificate DB',pin set > > >> certificate: > > >> > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > > >> cert-pki-kra',token='NSS Certificate DB' > > >> CA: dogtag-ipa-ca-renew-agent > > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG > <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> > > >> <http://MYDOMAIN.ORG> > > >> subject: CN=KRA Storage > Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> > > >> <http://MYDOMAIN.ORG> > > >> expires: 2020-06-27 01:54:32 EDT > > >> key usage: > > >> > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > >> eku: id-kp-clientAuth > > >> pre-save command: > /usr/libexec/ipa/certmonger/stop_pkicad > > >> post-save command: > > /usr/libexec/ipa/certmonger/renew_ca_cert > > >> "storageCert cert-pki-kra" > > >> track: yes > > >> auto-renew: yes > > >> ``` > > >> > > >> Here are the sequence of events that seem to have led to this: > > >> > > >> 1. Install FreeIPA Master many years ago and continue to > upgrade it > > >> from time to time. > > >> 2. Install FreeIPA Replica a few years after and continue > to upgrade > > >> it from time to time. > > >> 3. Allow the certificates to expire on both nodes. > > >> 4. Attempt to patch the replica via `yum upgrade` on the > second > > node. > > >> 5. Notice after reboot that `pki-tomcatd` is having > trouble and > > >> discover certificate issues. > > >> 5. Issue `ipa-cert-fix`, reboot again, and notice that > things are > > >> working. Try and create a key in the vault. > > >> 6. Attempt to patch the master via `yum upgrade` on the > first node. > > >> 7. Notice after reboot that everything seems to be ok. Try and > > create > > >> a key in the vault. > > >> 8. Notice a few days later that renewal seems to be broken > on the > > >> first node. > > >> > > >> At this point `ipa-cert-fix` just shows that everything is > fine. > > If I > > >> run it with -v, and then check the "storageCert cert-pki-kra" > > >> certificate with `openssl x509 -text -in`, I'm shown: > > > > > > Hi, > > > just double-checking, but did you run ipa-cert-fix on the > replica > > that > > > was repaired in step 5? If that's the case, it's normal that > > > ipa-cert-fix does not see any issue as it's running only > locally and > > > does not attempt to repair remote nodes. > > > > > > You will need to login to the node with expired certs and run > > > ipa-cert-fix there. > > > > I'd also look to see which one is the renewal master. That is > the one > > that should renew the cert. I'm too curious why the renewal > raised an > > error (as if it actually tried to renew) rather than either go > into > > CA_WORKING or pick up the updated cert. > > > > I'd also make sure that replication is working. On each master: > > > > # ipa-csreplica-manage list -v `hostname` > > > > rob > > > > > > > > HTH, > > > flo > > > > > >> > > >> Validity > > >> Not Before: Jun 29 00:52:33 2020 GMT > > >> Not After : Jun 19 00:52:33 2022 GMT > > >> > > >> On the second known, `getcert list` shows correct > expirations for > > >> those certificates: > > >> > > >> Request ID '20191206005909': > > >> status: MONITORING > > >> stuck: no > > >> key pair storage: > > >> > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > > >> cert-pki-kra',token='NSS Certificate DB',pin set > > >> certificate: > > >> > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > > >> cert-pki-kra',token='NSS Certificate DB' > > >> CA: dogtag-ipa-ca-renew-agent > > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG > <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> > > >> <http://MYDOMAIN.ORG> > > >> subject: CN=KRA Storage > Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> > > >> <http://MYDOMAIN.ORG> > > >> expires: 2022-06-18 20:52:33 EDT > > >> key usage: > > >> > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > >> eku: id-kp-clientAuth > > >> pre-save command: > /usr/libexec/ipa/certmonger/stop_pkicad > > >> post-save command: > > /usr/libexec/ipa/certmonger/renew_ca_cert > > >> "storageCert cert-pki-kra" > > >> track: yes > > >> auto-renew: yes > > >> > > >> It seems like _something_, perhaps `ipa-cert-fix` somehow > renewed > > >> these certificates but...outside of certmonger? Is this > some other > > >> version of > https://bugzilla.redhat.com/show_bug.cgi?id=1788907? The > > >> certificates are not in CA_WORKING though, they're in > MONITORING. > > >> > > >> What can I do to get myself out of this state as it seems like > > I'm in > > >> a "this could explode at any moment" situation? > > >> > > >> This is on Fedora 30 with IP version: > > >> > > >> Last metadata expiration check: 0:23:05 ago on Sat 04 Jul 2020 > > >> 07:59:16 PM EDT. > > >> Installed Packages > > >> Name : certmonger > > >> Version : 0.79.9 > > >> Release : 1.fc30 > > >> Architecture : x86_64 > > >> Size : 3.4 M > > >> Source : certmonger-0.79.9-1.fc30.src.rpm > > >> Repository : @System > > >> From repo : updates > > >> > > >> .. snip .. > > >> > > >> Name : freeipa-server > > >> Version : 4.8.3 > > >> Release : 1.fc30 > > >> Architecture : x86_64 > > >> Size : 1.3 M > > >> Source : freeipa-4.8.3-1.fc30.src.rpm > > >> Repository : @System > > >> From repo : updates > > >> > > >> .. snip .. > > >> > > >> Thanks! > > >> > > >> > > >> Ilya Kogan > > >> w: github.com/ikogan <http://github.com/ikogan> > <http://github.com/ikogan> > > <http://github.com/ikogan> e: > > >> [email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>> > > <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>> > > >> <http://twitter.com/ilkogan> > > <https://www.linkedin.com/in/ilyakogan/> > > >> > > >> > > >> _______________________________________________ > > >> FreeIPA-users mailing list -- > > [email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>> > > >> To unsubscribe send an email to > > >> [email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>> > > >> Fedora Code of Conduct: > > >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > >> List Guidelines: > > https://fedoraproject.org/wiki/Mailing_list_guidelines > > >> List Archives: > > >> > > > > https://lists.fedorahosted.org/archives/list/[email protected] > > >> > > >> > > > _______________________________________________ > > > FreeIPA-users mailing list -- > > [email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>> > > > To unsubscribe send an email to > > [email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>> > > > Fedora Code of Conduct: > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > > List Guidelines: > > https://fedoraproject.org/wiki/Mailing_list_guidelines > > > List Archives: > > > > > > > https://lists.fedorahosted.org/archives/list/[email protected] > > > > > > > > > _______________________________________________ > > FreeIPA-users mailing list -- [email protected] > <mailto:[email protected]> > > To unsubscribe send an email to > [email protected] > <mailto:[email protected]> > > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > > https://lists.fedorahosted.org/archives/list/[email protected] > > > _______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected]
