Ilya Kogan via FreeIPA-users wrote: > Thanks for that info, I don't see any suspicious errors in startup that > I haven't seen before. Just the following: > > - Token named "NSS Generic Crypto Services", not "NSS Certificate DB", > skipping. > - Error opening "/etc/httpd/alias/pwdfile.txt": No such file or directory. > > I don't think either of these are really an issue but I could be wrong.
You're right. > > Grepping the request files does indeed show those ca-error values > though. They don't really bother me if they won't cause issues. It seems > like it's just the last error it got from the CA, which just won't be > updated until it tries to request something next time. That's probably true as well. The error won't clear until certmonger tries the request again. rob > > On Wed, Jul 8, 2020, 2:41 PM Rob Crittenden <[email protected] > <mailto:[email protected]>> wrote: > > Ilya Kogan wrote: > > Wow ok, that was easy. `getcert list` now reports correct expiration > > dates for those certificates and they're all in MONITORING. It > still has > > that ca-error field although it's no longer trying to renew. Is that > > going to be an issue or is it just going to try again when it's > time to > > renew and succeed? > > I don't know. I'd check the journal to see if it logged any errors > post-restart. I don't believe that the ca-error is stored between > restarts. You could grep in /var/lib/certmonger/requests to see I > suppose. > > rob > > > > > On Wed, Jul 8, 2020, 9:43 AM Florence Blanc-Renaud <[email protected] > <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > > > On 7/6/20 7:59 PM, Ilya Kogan via FreeIPA-users wrote: > > > Hi, > > > > > > Thanks for the help so far! I've actually run `ipa-cert-fix` > on both > > > nodes, it says everything is ok on both nodes. When I run it > with > > > verbose mode, it spits out the command it's running and the > > certificate > > > it got, for example: > > > > > > ``` > > > ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d', > > > 'sql:/etc/pki/pki-tomcat/alias', '-L', '-n', 'storageCert > > > cert-pki-kra', '-a', '-f', > > '/etc/pki/pki-tomcat/alias/pwdfile.txt'] > > > ``` > > > > > > > > > If I then take that cert and ask what `openssl x509 -text > -noout` > > thinks > > > about it, it tells me that it's valid from 2020-06-29 to > 2022-06-29. > > > Strangely, though, when I ask `getcert list`, it shows that the > > certificate: > > > > > > ``` > > > certificate: > > > > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > > cert-pki-kra',token='NSS > > > Certificate DB' > > > ``` > > > > > > > > > expires on 2020-06-27. It's almost as if this node's > certificate has > > > _already_ been renewed but certmonger (I think) doesn't know > about > > it, > > > which might be why it's having trouble renewing it. > > > > > Hi, > > > > you may want to restart certmonger to force it re-reading the > > certificate information: > > # sudo systemctl restart certmonger > > > > flo > > > > > Here's what the two nodes say about replication: > > > > > > From node one: > > > > > > ``` > > > ipa-two.mydomain.org <http://ipa-two.mydomain.org> > <http://ipa-two.mydomain.org> > > <http://ipa-two.mydomain.org> > > > last update status: Error (0) Replica acquired > successfully: > > > Incremental update succeeded > > > last update ended: 2020-07-06 17:46:17+00:00 > > > ``` > > > > > > > > > From node two: > > > > > > ``` > > > ipa-one.gaea.mythicnet.org > <http://ipa-one.gaea.mythicnet.org> <http://ipa-one.gaea.mythicnet.org> > > <http://ipa-one.gaea.mythicnet.org> > > > last update status: Error (0) Replica acquired > successfully: > > > Incremental update succeeded > > > last update ended: 2020-07-06 17:46:17+00:00 > > > ``` > > > > > > > > > I suppose this might be a good time to mention that this is a > > simple two > > > node multi-master setup. Finally, I'm not sure if I'm doing this > > > correctly, but to make absolutely sure about which node is the > > renewal > > > master, I ran this on both nodes: > > > > > > ``` > > > ldapsearch -H ldap://ipa-one.gaea.mythicnet.org > <http://ipa-one.gaea.mythicnet.org> > > <http://ipa-one.gaea.mythicnet.org> > > > <http://ipa-one.gaea.mythicnet.org> -D 'cn=Directory > Manager' > > -W -b > > > 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' > > > '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn > > > ldapsearch -H ldap://ipa-two.gaea.mythicnet.org > <http://ipa-two.gaea.mythicnet.org> > > <http://ipa-two.gaea.mythicnet.org> > > > <http://ipa-two.gaea.mythicnet.org> -D 'cn=Directory > Manager' > > -W -b > > > 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' > > > '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn > > > ``` > > > > > > > > > The result for both is: > > > > > > ``` > > > dn: cn=CA,cn=ipa-one.gaea.mythicnet.org > <http://ipa-one.gaea.mythicnet.org> > > <http://ipa-one.gaea.mythicnet.org> > > > > > > > <http://ipa-one.gaea.mythicnet.org>,cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org > > > ``` > > > > > > > > > So it looks like the renewal master is the one having this > problem. > > > > > > > > > Ilya Kogan > > > w: github.com/ikogan <http://github.com/ikogan> > <http://github.com/ikogan> > > <http://github.com/ikogan> e: [email protected] > <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>> > > > <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>> > > > <http://twitter.com/ilkogan> > <https://www.linkedin.com/in/ilyakogan/> > > > > > > > > > > > > On Mon, Jul 6, 2020 at 1:25 PM Rob Crittenden > <[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>> > > > <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>>> wrote: > > > > > > Florence Blanc-Renaud via FreeIPA-users wrote: > > > > On 7/5/20 2:23 AM, Ilya Kogan via FreeIPA-users wrote: > > > >> Hi, > > > >> > > > >> I seem to be facing a similar issue with one of my KRAs. > > My KRA > > > >> certificates were, for some reason, not automatically > > renewed when > > > >> they expired last month. Using `ipa-cert-fix` > correctly fixed > > > them on > > > >> _one_ host. On the other, they seem to be stuck in the > > renewal state > > > >> and `ipa-cert-fix` claims there's nothing to do: > > > >> > > > >> ``` > > > >> Request ID '20191031183458': > > > >> status: MONITORING > > > >> ca-error: Server at > > > >> > "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit"; > > replied: > > > >> Missing credential: sessionID > > > >> stuck: no > > > >> key pair storage: > > > >> > > > > > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > > > >> cert-pki-kra',token='NSS Certificate DB',pin set > > > >> certificate: > > > >> > > > > > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > > > >> cert-pki-kra',token='NSS Certificate DB' > > > >> CA: dogtag-ipa-ca-renew-agent > > > >> issuer: CN=Certificate > Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> > > > <http://MYDOMAIN.ORG> > > > >> <http://MYDOMAIN.ORG> > > > >> subject: CN=KRA Audit,O=MYDOMAIN.ORG > <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> > > > <http://MYDOMAIN.ORG> <http://MYDOMAIN.ORG> > > > >> expires: 2020-06-27 01:54:34 EDT > > > >> key usage: digitalSignature,nonRepudiation > > > >> pre-save command: > > /usr/libexec/ipa/certmonger/stop_pkicad > > > >> post-save command: > > > /usr/libexec/ipa/certmonger/renew_ca_cert > > > >> "auditSigningCert cert-pki-kra" > > > >> track: yes > > > >> auto-renew: yes > > > >> Request ID '20191031183459': > > > >> status: MONITORING > > > >> ca-error: Server at > > > >> > "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit"; > > replied: > > > >> Missing credential: sessionID > > > >> stuck: no > > > >> key pair storage: > > > >> > > > > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert > > > cert-pki-kra',token='NSS > > > >> Certificate DB',pin set > > > >> certificate: > > > >> > > > > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert > > > cert-pki-kra',token='NSS > > > >> Certificate DB' > > > >> CA: dogtag-ipa-ca-renew-agent > > > >> issuer: CN=Certificate > Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> > > > <http://MYDOMAIN.ORG> > > > >> <http://MYDOMAIN.ORG> > > > >> subject: CN=KRA Transport > > Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > > > <http://MYDOMAIN.ORG> > > > >> <http://MYDOMAIN.ORG> > > > >> expires: 2020-06-27 01:54:30 EDT > > > >> key usage: > > > >> > > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > > >> eku: id-kp-clientAuth > > > >> pre-save command: > > /usr/libexec/ipa/certmonger/stop_pkicad > > > >> post-save command: > > > /usr/libexec/ipa/certmonger/renew_ca_cert > > > >> "transportCert cert-pki-kra" > > > >> track: yes > > > >> auto-renew: yes > > > >> Request ID '20191031183500': > > > >> status: MONITORING > > > >> ca-error: Server at > > > >> > "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit"; > > replied: > > > >> Missing credential: sessionID > > > >> stuck: no > > > >> key pair storage: > > > >> > > > > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > > > >> cert-pki-kra',token='NSS Certificate DB',pin set > > > >> certificate: > > > >> > > > > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > > > >> cert-pki-kra',token='NSS Certificate DB' > > > >> CA: dogtag-ipa-ca-renew-agent > > > >> issuer: CN=Certificate > Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> > > > <http://MYDOMAIN.ORG> > > > >> <http://MYDOMAIN.ORG> > > > >> subject: CN=KRA Storage > > Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > > > <http://MYDOMAIN.ORG> > > > >> <http://MYDOMAIN.ORG> > > > >> expires: 2020-06-27 01:54:32 EDT > > > >> key usage: > > > >> > > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > > >> eku: id-kp-clientAuth > > > >> pre-save command: > > /usr/libexec/ipa/certmonger/stop_pkicad > > > >> post-save command: > > > /usr/libexec/ipa/certmonger/renew_ca_cert > > > >> "storageCert cert-pki-kra" > > > >> track: yes > > > >> auto-renew: yes > > > >> ``` > > > >> > > > >> Here are the sequence of events that seem to have > led to this: > > > >> > > > >> 1. Install FreeIPA Master many years ago and continue to > > upgrade it > > > >> from time to time. > > > >> 2. Install FreeIPA Replica a few years after and > continue > > to upgrade > > > >> it from time to time. > > > >> 3. Allow the certificates to expire on both nodes. > > > >> 4. Attempt to patch the replica via `yum upgrade` on the > > second > > > node. > > > >> 5. Notice after reboot that `pki-tomcatd` is having > > trouble and > > > >> discover certificate issues. > > > >> 5. Issue `ipa-cert-fix`, reboot again, and notice that > > things are > > > >> working. Try and create a key in the vault. > > > >> 6. Attempt to patch the master via `yum upgrade` on the > > first node. > > > >> 7. Notice after reboot that everything seems to be > ok. Try and > > > create > > > >> a key in the vault. > > > >> 8. Notice a few days later that renewal seems to be > broken > > on the > > > >> first node. > > > >> > > > >> At this point `ipa-cert-fix` just shows that > everything is > > fine. > > > If I > > > >> run it with -v, and then check the "storageCert > cert-pki-kra" > > > >> certificate with `openssl x509 -text -in`, I'm shown: > > > > > > > > Hi, > > > > just double-checking, but did you run ipa-cert-fix on the > > replica > > > that > > > > was repaired in step 5? If that's the case, it's > normal that > > > > ipa-cert-fix does not see any issue as it's running only > > locally and > > > > does not attempt to repair remote nodes. > > > > > > > > You will need to login to the node with expired certs > and run > > > > ipa-cert-fix there. > > > > > > I'd also look to see which one is the renewal master. > That is > > the one > > > that should renew the cert. I'm too curious why the renewal > > raised an > > > error (as if it actually tried to renew) rather than > either go > > into > > > CA_WORKING or pick up the updated cert. > > > > > > I'd also make sure that replication is working. On each > master: > > > > > > # ipa-csreplica-manage list -v `hostname` > > > > > > rob > > > > > > > > > > > HTH, > > > > flo > > > > > > > >> > > > >> Validity > > > >> Not Before: Jun 29 00:52:33 2020 GMT > > > >> Not After : Jun 19 00:52:33 2022 GMT > > > >> > > > >> On the second known, `getcert list` shows correct > > expirations for > > > >> those certificates: > > > >> > > > >> Request ID '20191206005909': > > > >> status: MONITORING > > > >> stuck: no > > > >> key pair storage: > > > >> > > > > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > > > >> cert-pki-kra',token='NSS Certificate DB',pin set > > > >> certificate: > > > >> > > > > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > > > >> cert-pki-kra',token='NSS Certificate DB' > > > >> CA: dogtag-ipa-ca-renew-agent > > > >> issuer: CN=Certificate > Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> > > > <http://MYDOMAIN.ORG> > > > >> <http://MYDOMAIN.ORG> > > > >> subject: CN=KRA Storage > > Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > > > <http://MYDOMAIN.ORG> > > > >> <http://MYDOMAIN.ORG> > > > >> expires: 2022-06-18 20:52:33 EDT > > > >> key usage: > > > >> > > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > > >> eku: id-kp-clientAuth > > > >> pre-save command: > > /usr/libexec/ipa/certmonger/stop_pkicad > > > >> post-save command: > > > /usr/libexec/ipa/certmonger/renew_ca_cert > > > >> "storageCert cert-pki-kra" > > > >> track: yes > > > >> auto-renew: yes > > > >> > > > >> It seems like _something_, perhaps `ipa-cert-fix` > somehow > > renewed > > > >> these certificates but...outside of certmonger? Is this > > some other > > > >> version of > > https://bugzilla.redhat.com/show_bug.cgi?id=1788907? The > > > >> certificates are not in CA_WORKING though, they're in > > MONITORING. > > > >> > > > >> What can I do to get myself out of this state as it > seems like > > > I'm in > > > >> a "this could explode at any moment" situation? > > > >> > > > >> This is on Fedora 30 with IP version: > > > >> > > > >> Last metadata expiration check: 0:23:05 ago on Sat > 04 Jul 2020 > > > >> 07:59:16 PM EDT. > > > >> Installed Packages > > > >> Name : certmonger > > > >> Version : 0.79.9 > > > >> Release : 1.fc30 > > > >> Architecture : x86_64 > > > >> Size : 3.4 M > > > >> Source : certmonger-0.79.9-1.fc30.src.rpm > > > >> Repository : @System > > > >> From repo : updates > > > >> > > > >> .. snip .. > > > >> > > > >> Name : freeipa-server > > > >> Version : 4.8.3 > > > >> Release : 1.fc30 > > > >> Architecture : x86_64 > > > >> Size : 1.3 M > > > >> Source : freeipa-4.8.3-1.fc30.src.rpm > > > >> Repository : @System > > > >> From repo : updates > > > >> > > > >> .. snip .. > > > >> > > > >> Thanks! > > > >> > > > >> > > > >> Ilya Kogan > > > >> w: github.com/ikogan <http://github.com/ikogan> > <http://github.com/ikogan> > > <http://github.com/ikogan> > > > <http://github.com/ikogan> e: > > > >> [email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>> > > <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>> > > > <mailto:[email protected] > <mailto:[email protected]> <mailto:[email protected] > <mailto:[email protected]>> > > <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>>> > > > >> <http://twitter.com/ilkogan> > > > <https://www.linkedin.com/in/ilyakogan/> > > > >> > > > >> > > > >> _______________________________________________ > > > >> FreeIPA-users mailing list -- > > > [email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>> > > > <mailto:[email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>>> > > > >> To unsubscribe send an email to > > > >> [email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>> > > > <mailto:[email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>>> > > > >> Fedora Code of Conduct: > > > >> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > > >> List Guidelines: > > > https://fedoraproject.org/wiki/Mailing_list_guidelines > > > >> List Archives: > > > >> > > > > > > > https://lists.fedorahosted.org/archives/list/[email protected] > > > >> > > > >> > > > > _______________________________________________ > > > > FreeIPA-users mailing list -- > > > [email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>> > > > <mailto:[email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>>> > > > > To unsubscribe send an email to > > > [email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>> > > > <mailto:[email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>>> > > > > Fedora Code of Conduct: > > > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > > > List Guidelines: > > > https://fedoraproject.org/wiki/Mailing_list_guidelines > > > > List Archives: > > > > > > > > > > > https://lists.fedorahosted.org/archives/list/[email protected] > > > > > > > > > > > > > _______________________________________________ > > > FreeIPA-users mailing list -- > [email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>> > > > To unsubscribe send an email to > > [email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>> > > > Fedora Code of Conduct: > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > > List Guidelines: > > https://fedoraproject.org/wiki/Mailing_list_guidelines > > > List Archives: > > > > https://lists.fedorahosted.org/archives/list/[email protected] > > > > > > > > _______________________________________________ > FreeIPA-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/[email protected] > _______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected]
