Hi, I think I spotted the issue: https://pagure.io/freeipa/issue/9381
It was fixed on the ipa-4-10 branch but never released in ipa 4.10. Since your ipa1 host has freeipa-server-4.10.1-1.fc37 it doesn't have the patch. Check if you have a drop-in file /etc/systemd/system/[email protected]/ipa.conf If not, create one with the following content: # cat /etc/systemd/system/[email protected]/ipa.conf [Service] Environment=LC_ALL=C.UTF-8 ExecStartPost=/usr/libexec/ipa/ipa-pki-wait-running then launch systemctl --system daemon-reload and ipa-server-upgrade. HTH, flo On Thu, Feb 20, 2025 at 5:29 PM Boris <[email protected]> wrote: > Hey, > > *pki-server subsystem-show ca *and *curl --cert > /var/lib/ipa/ra-agent.pem --key /var/lib/ipa/ra-agent.key > https://`hostname`:8443/ca/rest/account/login* > gave the expected results. > > My thought was that there is no ca available during the update, and thats > why I wanted to add the 2nd host as CA. > > I feel a bit nervous about posting both logs, because it feels hard to > clean them up from some information. > I gave my best. You can find both logs here: > https://blktrace.kervyn.de/debug.2025-02-20.log.gz > https://blktrace.kervyn.de/ipaupgrade.log.gz > > > > Am Do., 20. Feb. 2025 um 16:53 Uhr schrieb Florence Blanc-Renaud < > [email protected]>: > >> Hi, >> >> On Wed, Feb 19, 2025 at 11:57 AM Boris <[email protected]> wrote: >> >>> Hi flo, >>> >>> `ipa cert-show 1` works on both IPA hosts and returns correct data, from >>> what I can tell. >>> >> That's strange because cert-show is also authenticating to the CA REST >> API. >> >> >>> `ipa config-show` gies the following: >>> IPA masters: ipa1.redacted, ipa2.redacted >>> IPA master capable of PKINIT: ipa2.redacted >>> IPA CA servers: ipa1.redacted >>> IPA CA renewal master: ipa1.redacted >>> IPA DNS servers: ipa1.redacted, ipa2.redacted >>> >>> regarding the named crashes: I think the problem might be related to >>> ldap. The last time the named daemons were in a restart/crash loop I >>> restarted the ipa2 host which immediately resolved the problem. >>> ipa1: >>> bind-9.18.19-1.fc37.x86_64 >>> bind-dyndb-ldap-11.10-17.fc37.x86_64 >>> >>> ipa2: >>> bind-9.16.28-1.fc35.x86_64 >>> bind-dyndb-ldap-11.9-12.fc35.x86_64 >>> >>> for the coredump I would need your guidance what to do, because I am not >>> that firm with named debugging. >>> >>> Here are the last couple of line from the /var/log/ipaupgrade.log file. >>> The update seems to go through, but it fails when it needs to authenticate >>> with the CA REST API >>> >>> 2025-01-21T20:24:14Z DEBUG stderr= >>> 2025-01-21T20:24:14Z DEBUG Starting external process >>> 2025-01-21T20:24:14Z DEBUG args=['/bin/systemctl', 'start', >>> 'certmonger.service'] >>> 2025-01-21T20:24:15Z DEBUG Process finished, return code=0 >>> 2025-01-21T20:24:15Z DEBUG stdout= >>> 2025-01-21T20:24:15Z DEBUG stderr= >>> 2025-01-21T20:24:15Z DEBUG Starting external process >>> 2025-01-21T20:24:15Z DEBUG args=['/bin/systemctl', 'is-active', >>> 'certmonger.service'] >>> 2025-01-21T20:24:15Z DEBUG Process finished, return code=0 >>> 2025-01-21T20:24:15Z DEBUG stdout=active >>> >>> 2025-01-21T20:24:15Z DEBUG stderr= >>> 2025-01-21T20:24:15Z DEBUG Start of certmonger.service complete >>> 2025-01-21T20:24:15Z DEBUG Starting external process >>> 2025-01-21T20:24:15Z DEBUG args=['pki-server', 'subsystem-show', 'kra'] >>> 2025-01-21T20:24:15Z DEBUG Process finished, return code=1 >>> 2025-01-21T20:24:15Z DEBUG stdout= >>> 2025-01-21T20:24:15Z DEBUG stderr=ERROR: ERROR: No kra subsystem in >>> instance pki-tomcat. >>> >>> 2025-01-21T20:24:15Z INFO [Update certmonger certificate renewal >>> configuration] >>> 2025-01-21T20:24:15Z DEBUG Loading Index file from >>> '/var/lib/ipa/sysrestore/sysrestore.index' >>> 2025-01-21T20:24:15Z DEBUG Starting external process >>> 2025-01-21T20:24:15Z DEBUG args=['/usr/bin/certutil', '-d', >>> 'sql:/etc/dirsrv/slapd-redacted/', '-L', '-n', 'Server-Cert', '-a', '-f', >>> '/etc/dirsrv/slapd-redacted/pwdfile.txt'] >>> 2025-01-21T20:24:15Z DEBUG Process finished, return code=0 >>> 2025-01-21T20:24:15Z DEBUG stdout=-----BEGIN CERTIFICATE----- >>> redacted >>> -----END CERTIFICATE----- >>> >>> 2025-01-21T20:24:15Z DEBUG stderr= >>> 2025-01-21T20:24:15Z DEBUG Loading Index file from >>> '/var/lib/ipa/sysrestore/sysrestore.index' >>> 2025-01-21T20:24:15Z DEBUG Starting external process >>> 2025-01-21T20:24:15Z DEBUG args=['/usr/bin/certutil', '-d', >>> 'sql:/etc/pki/pki-tomcat/alias', '-L', '-f', >>> '/etc/pki/pki-tomcat/alias/pwdfile.txt'] >>> 2025-01-21T20:24:15Z DEBUG Process finished, return code=0 >>> 2025-01-21T20:24:15Z DEBUG stdout= >>> Certificate Nickname Trust >>> Attributes >>> >>> SSL,S/MIME,JAR/XPI >>> >>> caSigningCert cert-pki-ca CTu,Cu,Cu >>> caSigningCert cert-pki-ca 6148bb27-6bd6-4a0a-b607-6ba538a6c401 u,u,u >>> ocspSigningCert cert-pki-ca u,u,u >>> subsystemCert cert-pki-ca u,u,u >>> auditSigningCert cert-pki-ca u,u,Pu >>> Server-Cert cert-pki-ca u,u,u >>> >>> 2025-01-21T20:24:15Z DEBUG stderr= >>> 2025-01-21T20:24:15Z INFO Certmonger certificate renewal configuration >>> already up-to-date >>> 2025-01-21T20:24:15Z INFO [Enable PKIX certificate path discovery and >>> validation] >>> 2025-01-21T20:24:15Z DEBUG Loading StateFile from >>> '/var/lib/ipa/sysupgrade/sysupgrade.state' >>> 2025-01-21T20:24:15Z INFO PKIX already enabled >>> 2025-01-21T20:24:15Z INFO [Authorizing RA Agent to modify profiles] >>> 2025-01-21T20:24:15Z INFO [Authorizing RA Agent to manage lightweight >>> CAs] >>> 2025-01-21T20:24:15Z INFO [Ensuring Lightweight CAs container exists in >>> Dogtag database] >>> 2025-01-21T20:24:15Z INFO [Adding default OCSP URI configuration] >>> 2025-01-21T20:24:15Z INFO [Disabling cert publishing] >>> 2025-01-21T20:24:15Z INFO [Ensuring CA is using LDAPProfileSubsystem] >>> 2025-01-21T20:24:15Z INFO [Migrating certificate profiles to LDAP] >>> 2025-01-21T20:24:15Z DEBUG Profile 'AdminCert' is already in LDAP and >>> enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'DomainController' is already in LDAP >>> and enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'ECAdminCert' is already in LDAP and >>> enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'acmeServerCert' is already in LDAP >>> and enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caAdminCert' is already in LDAP and >>> enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caAgentFileSigning' is already in >>> LDAP and enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caAgentServerCert' is already in >>> LDAP and enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caAuditSigningCert' is already in >>> LDAP and enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caCACert' is already in LDAP and >>> enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caCMCECUserCert' is already in LDAP >>> and enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caCMCECserverCert' is already in >>> LDAP and enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caCMCECsubsystemCert' is already in >>> LDAP and enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caCMCUserCert' is already in LDAP >>> and enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caCrossSignedCACert' is already in >>> LDAP and enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caDirBasedDualCert' is already in >>> LDAP and enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caDirPinUserCert' is already in LDAP >>> and enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caDirUserCert' is already in LDAP >>> and enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caDirUserRenewal' is already in LDAP >>> and enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caDualCert' is already in LDAP and >>> enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caDualRAuserCert' is already in LDAP >>> and enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caECAdminCert' is already in LDAP >>> and enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caECAgentServerCert' is already in >>> LDAP and enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caECDirPinUserCert' is already in >>> LDAP and enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caECDirUserCert' is already in LDAP >>> and enabled; skipping >>> 2025-01-21T20:24:15Z DEBUG Profile 'caECDualCert' is already in LDAP and >>> enabled; skipping >>> 2025-01-21T20:24:15Z INFO Migrating profile 'caECFullCMCSharedTokenCert' >>> 2025-01-21T20:24:15Z DEBUG request GET >>> https://ipa1.redacted:8443/ca/rest/account/login >>> 2025-01-21T20:24:15Z DEBUG request body '' >>> 2025-01-21T20:24:16Z DEBUG response status 404 >>> 2025-01-21T20:24:16Z DEBUG response headers Content-Type: >>> text/html;charset=utf-8 >>> Content-Language: en >>> Content-Length: 784 >>> Date: Tue, 21 Jan 2025 20:24:16 GMT >>> >>> >>> 2025-01-21T20:24:16Z DEBUG response body (decoded): b'<!doctype >>> html><html lang="en"><head><title>HTTP Status 404 \xe2\x80\x93 Not >>> Found</title><style type="text/css">body >>> {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b >>> {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 >>> {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} >>> .line >>> {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP >>> Status 404 \xe2\x80\x93 Not Found</h1><hr class="line" /><p><b>Type</b> >>> Status Report</p><p><b>Message</b> The requested resource >>> [/ca/rest/account/login] is not >>> available</p><p><b>Description</b> The origin server did not find a current >>> representation for the target resource or is not willing to disclose that >>> one exists.</p><hr class="line" /><h3>Apache >>> Tomcat/9.0.82</h3></body></html>' >>> 2025-01-21T20:24:16Z ERROR IPA server upgrade failed: Inspect >>> /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. >>> 2025-01-21T20:24:16Z DEBUG File >>> "/usr/lib/python3.11/site-packages/ipapython/admintool.py", line 180, in >>> execute >>> return_value = self.run() >>> ^^^^^^^^^^ >>> File >>> "/usr/lib/python3.11/site-packages/ipaserver/install/ipa_server_upgrade.py", >>> line 54, in run >>> server.upgrade() >>> File >>> "/usr/lib/python3.11/site-packages/ipaserver/install/server/upgrade.py", >>> line 2061, in upgrade >>> upgrade_configuration() >>> File >>> "/usr/lib/python3.11/site-packages/ipaserver/install/server/upgrade.py", >>> line 1914, in upgrade_configuration >>> ca_enable_ldap_profile_subsystem(ca) >>> File >>> "/usr/lib/python3.11/site-packages/ipaserver/install/server/upgrade.py", >>> line 458, in ca_enable_ldap_profile_subsystem >>> cainstance.migrate_profiles_to_ldap() >>> File >>> "/usr/lib/python3.11/site-packages/ipaserver/install/cainstance.py", line >>> 2155, in migrate_profiles_to_ldap >>> _create_dogtag_profile(profile_id, profile_data, overwrite=False) >>> File >>> "/usr/lib/python3.11/site-packages/ipaserver/install/cainstance.py", line >>> 2209, in _create_dogtag_profile >>> with api.Backend.ra_certprofile as profile_api: >>> File "/usr/lib/python3.11/site-packages/ipaserver/plugins/dogtag.py", >>> line 1211, in __enter__ >>> raise errors.RemoteRetrieveError(reason=_('Failed to authenticate to >>> CA REST API')) >>> >>> Can you check if the CA subsystem is enabled? >> # *pki-server subsystem-show ca* >> Subsystem ID: ca >> Instance ID: pki-tomcat >> Enabled: True >> >> If yes, try to authenticate to the rest API with curl: >> # *curl --cert /var/lib/ipa/ra-agent.pem --key >> /var/lib/ipa/ra-agent.key https://`hostname`:8443/ca/rest/account/login* >> {"id":"ipara","FullName":"ipara","Roles":["Certificate Manager >> Agents","Enterprise ACME Administrators","Registration Manager >> Agents","Security Domain Administrators"],"Attributes":{"Attribute":[]}} >> >> If the above commands are working, retry the upgrade with >> # *ipa-server-upgrade* >> and send us the full /var/log/ipaupgrade.log and >> /var/log/pki/pki-tomcat/ca/debug.$DATE.log. >> >> flo >> >> 2025-01-21T20:24:16Z DEBUG The ipa-server-upgrade command failed, >>> exception: RemoteRetrieveError: Failed to authenticate to CA REST API >>> 2025-01-21T20:24:16Z ERROR Unexpected error - see >>> /var/log/ipaupgrade.log for details: >>> RemoteRetrieveError: Failed to authenticate to CA REST API >>> 2025-01-21T20:24:16Z ERROR The ipa-server-upgrade command failed. See >>> /var/log/ipaupgrade.log for more information >>> >>> Am Mi., 19. Feb. 2025 um 10:56 Uhr schrieb Florence Blanc-Renaud < >>> [email protected]>: >>> >>>> Hi, >>>> >>>> in a previous message you mentioned that the directory manager password >>>> is lost. You can follow this article to reset the DM password: >>>> https://access.redhat.com/solutions/203473 >>>> >>>> Named crashes could be related to multiple issues: >>>> - inconsistent versions between bind and bind-dyndb-ldap. Which >>>> versions do you have? >>>> - an insufficient number of threads >>>> - an issue when reloading the zones >>>> If you can gather a coredump and install the debug packages it could >>>> help identify if you're hitting a known issue. >>>> >>>> You mentioned that ipa1 needs to be started with --force, can you tell >>>> which service is failing and provide the logs? There should be also more >>>> information in /var/log/ipaupgrade.log. >>>> >>>> In order to check the CA state, a useful command is 'ipa cert-show 1' >>>> as it communicates with the CA to gather the certificate details. If this >>>> command is failing (likely with "Failed to Authenticate to CA rest API") >>>> you need to understand where the config is broken. >>>> Start by checking which system is the CA renewal master: >>>> ipa config-show >>>> >>>> The CA renewal master will be your priority. >>>> >>>> flo >>>> >>>> On Wed, Feb 19, 2025 at 10:25 AM Boris via FreeIPA-users < >>>> [email protected]> wrote: >>>> >>>>> I think the CA is working, but I don't know for sure and how to verify >>>>> it. At least there are no expired certs on both ipa hosts >>>>> >>>>> [root@ipa1 ~]# getcert list | grep expires >>>>> expires: 2025-11-29 13:19:40 CET >>>>> expires: 2025-04-15 16:27:34 CEST >>>>> expires: 2025-04-15 16:26:44 CEST >>>>> expires: 2025-04-15 16:27:14 CEST >>>>> expires: 2037-08-19 16:11:12 CEST >>>>> expires: 2025-04-15 16:27:54 CEST >>>>> expires: 2025-04-15 16:27:04 CEST >>>>> expires: 2040-02-12 12:46:50 CET >>>>> expires: 2025-05-29 16:12:51 CEST >>>>> expires: 2026-01-26 13:48:23 CET >>>>> >>>>> [root@ipa2 ~]# getcert list | grep expires >>>>> expires: 2027-02-16 10:42:29 CET >>>>> expires: 2027-02-16 10:42:51 CET >>>>> expires: 2025-04-15 16:27:04 CEST >>>>> expires: 2027-02-16 10:43:26 CET >>>>> >>>>> The healthcheck showed some "group is not correct" and "files are to >>>>> permissive" which I resolved. >>>>> Now I have these to checks which do not tell me anything >>>>> "msg": "certmonger tracking request {key} found and is not >>>>> expected on an IPA master." >>>>> "msg": "No KDC workers defined in {sysconfig}" >>>>> >>>>> Am Di., 18. Feb. 2025 um 15:22 Uhr schrieb Rob Crittenden via >>>>> FreeIPA-users <[email protected]>: >>>>> >>>>>> Boris wrote: >>>>>> > Hi Rob, >>>>>> > >>>>>> > I have two hosts: ipa1 and ipa2 >>>>>> > >>>>>> > ipa1: >>>>>> > Fedora 37 >>>>>> > freeipa-server-4.10.1-1.fc37.x86_64 >>>>>> > Managed suffixes: domain, ca >>>>>> > running with ipactl start --force because the update is not working >>>>>> (The >>>>>> > ipa-server-upgrade command failed, exception: RemoteRetrieveError: >>>>>> > Failed to authenticate to CA REST API). >>>>>> > I tried to upgrade, but the upgrade did not go through. >>>>>> >>>>>> Your existing CA is having issues. I'd start by checking that your CA >>>>>> certificates are still valid: getcert list | grep expires >>>>>> >>>>>> You might also try installing the freeipa-healthcheck package and >>>>>> running ipa-healthcheck. Expect a lot of errors since it won't be able >>>>>> to connect to the CA but it will also check the validity dates, etc. >>>>>> >>>>>> > ipa2: >>>>>> > Fedora 35 >>>>>> > freeipa-server-4.9.11-1.fc35.x86_64 >>>>>> > Managed suffixes: domain >>>>>> > >>>>>> > So my thought process was: if it can not authenticate against the CA >>>>>> > REST API, I need to add the CA capability to ipa2 >>>>>> >>>>>> You need to authenticate to the CA to create a clone of it. You can't >>>>>> install another CA until you get your existing one working. >>>>>> >>>>>> rob >>>>>> >>>>>> >>>>> -- >>>>> _______________________________________________ >>>>> FreeIPA-users mailing list -- [email protected] >>>>> To unsubscribe send an email to >>>>> [email protected] >>>>> Fedora Code of Conduct: >>>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>>> List Guidelines: >>>>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>>>> List Archives: >>>>> https://lists.fedorahosted.org/archives/list/[email protected] >>>>> Do not reply to spam, report it: >>>>> https://pagure.io/fedora-infrastructure/new_issue >>>>> >>>> >>> >>> -- >>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im >>> groüen Saal. >>> >> > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > groüen Saal. >
-- _______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected] Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
