[Freeipa-users] Re: ipa-server-upgrade failed after yum update on CentOS7

Wed, 01 Jul 2020 04:42:53 -0700 

Hi,
as you have installed 4.6.5-11, the command ipa-cert-fix is available and should ease fixing the expired certs. The topology looks simple enough (a single master), so no need to worry about which server to fix first. 

More info available in [1] and in ipa-cert-fix man page.

HTH,
flo


[1]  
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/cert-renewal#renewing-expired-system-certificate-when-idm-is-offline


On 7/1/20 6:01 AM, Mariusz Stolarczyk via FreeIPA-users wrote:

The kinit command wouldn't work so it prevented the other commands. One 
of my issues is that the IPA server tries to update itself:


# ipactl start

IPA version error: data needs to be upgraded (expected version 
'4.6.6-11.el7.centos', current version '4.6.5-11.el7.centos.4')

Automatically running upgrade, for details see /var/log/ipaupgrade.log


This seemed to get me past that:

# ipactl start --skip-version-check --ignore-service-failure
Skipping version check
Existing service file detected!
Assuming stale, cleaning and proceeding
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting httpd Service
Failed to start httpd Service
Forced start, ignoring httpd Service, continuing normal operation
Starting ipa-custodia Service
Starting ntpd Service
Starting pki-tomcatd Service
Failed to start pki-tomcatd Service
Forced start, ignoring pki-tomcatd Service, continuing normal operation
Starting smb Service
Starting winbind Service
Starting ipa-otpd Service
Starting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful



However I found some instructions to rollback the system clock to get 
certmonger to renewal the expired certs. Now the httpd.service starts 
but not the pki-tomcatd.



# ipactl start --skip-version-check --ignore-service-failure
Skipping version check
Existing service file detected!
Assuming stale, cleaning and proceeding
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting httpd Service
Starting ipa-custodia Service
Starting ntpd Service
Starting pki-tomcatd Service
Failed to start pki-tomcatd Service
Forced start, ignoring pki-tomcatd Service, continuing normal operation
Starting smb Service
Starting winbind Service
Starting ipa-otpd Service
Starting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful


Now I was able to get the outputs:

# ipa config-show | grep "CA renewal"
   IPA CA renewal master: FAKE-HOST.FAKE-IPA-DOMAIN.lan


# ipa server-role-find
----------------------
6 server roles matched
----------------------
   Server name: FAKE-HOST.FAKE-IPA-DOMAIN.lan
   Role name: CA server
   Role status: enabled

   Server name: FAKE-HOST.FAKE-IPA-DOMAIN.lan
   Role name: DNS server
   Role status: enabled

   Server name: FAKE-HOST.FAKE-IPA-DOMAIN.lan
   Role name: NTP server
   Role status: enabled

   Server name: FAKE-HOST.FAKE-IPA-DOMAIN.lan
   Role name: AD trust agent
   Role status: enabled

   Server name: FAKE-HOST.FAKE-IPA-DOMAIN.lan
   Role name: KRA server
   Role status: absent

   Server name: FAKE-HOST.FAKE-IPA-DOMAIN.lan
   Role name: AD trust controller
   Role status: enabled
----------------------------
Number of entries returned 6
----------------------------


# getcert list
Number of certificates and requests being tracked: 9.
Request ID '20171108154417':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
CA: SelfSign
issuer: CN=FAKE-HOST.FAKE-IPA-DOMAIN.lan,O=FAKE-IPA-DOMAIN.LAN
subject: CN=FAKE-HOST.FAKE-IPA-DOMAIN.lan,O=FAKE-IPA-DOMAIN.LAN
expires: 2020-09-13 20:50:34 UTC
principal name: krbtgt/[email protected]
certificate template/profile: KDCs_PKINIT_Certs
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
track: yes
auto-renew: yes
Request ID '20181122014941':
status: MONITORING
stuck: no

key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB'

CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN
subject: CN=CA Audit,O=FAKE-IPA-DOMAIN.LAN
expires: 2022-05-18 03:13:17 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad

post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert 
"auditSigningCert cert-pki-ca"

track: yes
auto-renew: yes
Request ID '20181122014942':
status: MONITORING
stuck: no

key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS 
Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS 
Certificate DB'

CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN
subject: CN=OCSP Subsystem,O=FAKE-IPA-DOMAIN.LAN
expires: 2020-06-24 23:56:43 UTC
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad

post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert 
"ocspSigningCert cert-pki-ca"

track: yes
auto-renew: yes
Request ID '20181122014943':
status: MONITORING
stuck: no

key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert 
cert-pki-ca',token='NSS Certificate DB'

CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN
subject: CN=CA Subsystem,O=FAKE-IPA-DOMAIN.LAN
expires: 2022-05-18 03:11:57 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad

post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert 
"subsystemCert cert-pki-ca"

track: yes
auto-renew: yes
Request ID '20181122014944':
status: MONITORING
stuck: no

key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB'

CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN
subject: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN
expires: 2036-08-12 21:35:52 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad

post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert 
"caSigningCert cert-pki-ca"

track: yes
auto-renew: yes
Request ID '20181122014945':
status: CA_UNREACHABLE

ca-error: Error 60 connecting to 
https://FAKE-HOST.FAKE-IPA-DOMAIN.lan:8443/ca/agent/ca/profileReview: 
Peer certificate cannot be authenticated with given CA certificates.

stuck: no
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN
subject: CN=IPA RA,O=FAKE-IPA-DOMAIN.LAN
expires: 2020-06-24 23:56:33 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20181122014946':
status: CA_UNREACHABLE

ca-error: Error 60 connecting to 
https://FAKE-HOST.FAKE-IPA-DOMAIN.lan:8443/ca/agent/ca/profileReview: 
Peer certificate cannot be authenticated with given CA certificates.

stuck: no

key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB'

CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN
subject: CN=FAKE-HOST.FAKE-IPA-DOMAIN.lan,O=FAKE-IPA-DOMAIN.LAN
expires: 2020-06-24 23:55:43 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad

post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert 
"Server-Cert cert-pki-ca"

track: yes
auto-renew: yes
Request ID '20181122014947':
status: CA_UNREACHABLE

ca-error: Server at https://FAKE-HOST.FAKE-IPA-DOMAIN.lan/ipa/xml failed 
request, will retry: -504 (libcurl failed to execute the HTTP POST 
transaction, explaining:  Failed connect to 
FAKE-HOST.FAKE-IPA-DOMAIN.lan:443; Connection refused).

stuck: no

key pair storage: 
type=NSSDB,location='/etc/dirsrv/slapd-FAKE-IPA-DOMAIN-LAN',nickname='Server-Cert',token='NSS 
Certificate DB',pinfile='/etc/dirsrv/slapd-FAKE-IPA-DOMAIN-LAN/pwdfile.txt'
certificate: 
type=NSSDB,location='/etc/dirsrv/slapd-FAKE-IPA-DOMAIN-LAN',nickname='Server-Cert',token='NSS 
Certificate DB'

CA: IPA
issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN
subject: CN=FAKE-HOST.FAKE-IPA-DOMAIN.lan,O=FAKE-IPA-DOMAIN.LAN
expires: 2020-07-17 16:47:45 UTC
principal name: ldap/[email protected]
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:

post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv 
FAKE-IPA-DOMAIN-LAN

track: yes
auto-renew: yes
Request ID '20181122014948':
status: MONITORING
stuck: no

key pair storage: 
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: 
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
Certificate DB'

CA: IPA
issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN
subject: CN=FAKE-HOST.FAKE-IPA-DOMAIN.lan,O=FAKE-IPA-DOMAIN.LAN
expires: 2022-03-16 22:14:54 UTC
dns: FAKE-HOST.FAKE-IPA-DOMAIN.lan
principal name: HTTP/[email protected]
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes


I am also able to restart pki-tomcatd service after two restart attempts:


# systemctl restart [email protected]
# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: STOPPED
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful

# systemctl restart [email protected]
# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful

# systemctl status [email protected]
● [email protected] - PKI Tomcat Server pki-tomcat

    Loaded: loaded (/lib/systemd/system/[email protected]; enabled; 
vendor preset: disabled)

    Active: active (running) since Tue 2020-06-30 20:55:41 PDT; 20s ago

   Process: 9567 ExecStop=/usr/libexec/tomcat/server stop (code=exited, 
status=0/SUCCESS)
   Process: 9612 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, 
status=0/SUCCESS)

  Main PID: 9749 (java)

    CGroup: 
/system.slice/system-pki\x2dtomcatd.slice/[email protected]
            └─9749 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java 
-DRESTEASY_LIB=/usr/share/java/resteasy-base 
-Djava.library.path=/usr/lib64/nuxwdog-jni -classpath 
/usr/share/tomcat/bin/bo...



Jun 30 20:55:56 FAKE-IPA-HOST.FAKE-IPA-DOMAIN.lan server[9749]: Jun 30, 
2020 8:55:56 PM org.apache.catalina.loader.WebappClassLoaderBase 
clearReferencesThreads
Jun 30 20:55:56 FAKE-IPA-HOST.FAKE-IPA-DOMAIN.lan server[9749]: SEVERE: 
The web application [/ca] appears to have started a thread named 
[LDAPConnThread-0 ldaps://FAKE-IPA-HOST.FAKE-IPA-DOMAIN.lan:636] 
...emory leak.
Jun 30 20:55:56 FAKE-IPA-HOST.FAKE-IPA-DOMAIN.lan server[9749]: Jun 30, 
2020 8:55:56 PM org.apache.catalina.loader.WebappClassLoaderBase 
clearReferencesThreads
Jun 30 20:55:56 FAKE-IPA-HOST.FAKE-IPA-DOMAIN.lan server[9749]: SEVERE: 
The web application [/ca] appears to have started a thread named 
[LDAPConnThread-2 ldaps://FAKE-IPA-HOST.FAKE-IPA-DOMAIN.lan:636] 
...emory leak.
Jun 30 20:55:56 FAKE-IPA-HOST.FAKE-IPA-DOMAIN.lan server[9749]: Jun 30, 
2020 8:55:56 PM org.apache.catalina.loader.WebappClassLoaderBase 
clearReferencesThreads
Jun 30 20:55:56 FAKE-IPA-HOST.FAKE-IPA-DOMAIN.lan server[9749]: SEVERE: 
The web application [/ca] appears to have started a thread named 
[authorityMonitor] but has failed to stop it. Thi...emory leak.
Jun 30 20:55:56 FAKE-IPA-HOST.FAKE-IPA-DOMAIN.lan server[9749]: Jun 30, 
2020 8:55:56 PM org.apache.catalina.loader.WebappClassLoaderBase 
clearReferencesThreads
Jun 30 20:55:56 FAKE-IPA-HOST.FAKE-IPA-DOMAIN.lan server[9749]: SEVERE: 
The web application [/ca] appears to have started a thread named 
[LDAPConnThread-3 ldaps://FAKE-IPA-HOST.FAKE-IPA-DOMAIN.lan:636] 
...emory leak.
Jun 30 20:55:56 FAKE-IPA-HOST.FAKE-IPA-DOMAIN.lan server[9749]: Jun 30, 
2020 8:55:56 PM org.apache.catalina.loader.WebappClassLoaderBase 
clearReferencesThreads
Jun 30 20:55:56 FAKE-IPA-HOST.FAKE-IPA-DOMAIN.lan server[9749]: SEVERE: 
The web application [/ca] appears to have started a thread named 
[profileChangeMonitor] but has failed to stop it....emory leak.

Hint: Some lines were ellipsized, use -l to show in full.


Not sure what to do next.

Thanks,
-ms

------------------------------------------------------------------------
*From:* Rob Crittenden <[email protected]>
*Sent:* Tuesday, June 30, 2020 8:20 PM

*To:* FreeIPA users list <[email protected]>; 
Florence Blanc-Renaud <[email protected]>

*Cc:* Mariusz Stolarczyk <[email protected]>

*Subject:* Re: [Freeipa-users] Re: ipa-server-upgrade failed after yum 
update on CentOS7

Mariusz Stolarczyk via FreeIPA-users wrote:

Thanks for the response.

This is my main IPA server the rest of my small network are just linux
clients.


kinit: Cannot contact any KDC for realm 'FAKE-IPA-DOMAIN.LAN' while
getting initial credentials


The other information that Flo requested is needed as well.

Three of your certificates expired on June 24 and to create a plan to
fix it we need the other info.

rob




# getcert list
Number of certificates and requests being tracked: 9.
Request ID '20171108154417':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
CA: SelfSign
issuer: CN=sol.FAKE-IPA-DOMAIN.LAN,O=FAKE-IPA-DOMAIN.LAN
subject: CN=sol.FAKE-IPA-DOMAIN.LAN,O=FAKE-IPA-DOMAIN.LAN
expires: 2020-09-13 20:50:34 UTC
principal name: krbtgt/[email protected]
certificate template/profile: KDCs_PKINIT_Certs
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
track: yes
auto-renew: yes
Request ID '20181122014941':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN
subject: CN=CA Audit,O=FAKE-IPA-DOMAIN.LAN
expires: 2022-05-18 03:13:17 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20181122014942':
status: CA_UNREACHABLE
ca-error: Internal error
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS
Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS
Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN
subject: CN=OCSP Subsystem,O=FAKE-IPA-DOMAIN.LAN
expires: 2020-06-24 23:56:43 UTC
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20181122014943':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN
subject: CN=CA Subsystem,O=FAKE-IPA-DOMAIN.LAN
expires: 2022-05-18 03:11:57 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"subsystemCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20181122014944':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN
subject: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN
expires: 2036-08-12 21:35:52 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"caSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20181122014945':
status: CA_UNREACHABLE
ca-error: Internal error
stuck: no
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN
subject: CN=IPA RA,O=FAKE-IPA-DOMAIN.LAN
expires: 2020-06-24 23:56:33 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20181122014946':
status: CA_UNREACHABLE
ca-error: Internal error
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN
subject: CN=sol.FAKE-IPA-DOMAIN.LAN,O=FAKE-IPA-DOMAIN.LAN
expires: 2020-06-24 23:55:43 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"Server-Cert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20181122014947':
status: CA_UNREACHABLE
ca-error: Error setting up ccache for "host" service on client using
default keytab: Cannot contact any KDC for realm 'FAKE-IPA-DOMAIN.LAN'.
stuck: no
key pair storage:
type=NSSDB,location='/etc/dirsrv/slapd-FAKE-IPA-DOMAIN-LAN',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/dirsrv/slapd-FAKE-IPA-DOMAIN-LAN/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/dirsrv/slapd-FAKE-IPA-DOMAIN-LAN',nickname='Server-Cert',token='NSS
Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN
subject: CN=sol.FAKE-IPA-DOMAIN.LAN,O=FAKE-IPA-DOMAIN.LAN
expires: 2020-07-17 16:47:45 UTC
principal name: ldap/[email protected]
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv
FAKE-IPA-DOMAIN-LAN
track: yes
auto-renew: yes
Request ID '20181122014948':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=FAKE-IPA-DOMAIN.LAN
subject: CN=sol.FAKE-IPA-DOMAIN.LAN,O=FAKE-IPA-DOMAIN.LAN
expires: 2022-03-16 22:14:54 UTC
dns: sol.FAKE-IPA-DOMAIN.LAN
principal name: HTTP/[email protected]
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes


What can I do next?

Thanks,
-ms



------------------------------------------------------------------------
*From:* Florence Blanc-Renaud <[email protected]>
*Sent:* Tuesday, June 30, 2020 1:45 AM
*To:* FreeIPA users list <[email protected]>
*Cc:* Mariusz Stolarczyk <[email protected]>
*Subject:* Re: [Freeipa-users] ipa-server-upgrade failed after yum
update on CentOS7

 
On 6/30/20 10:24 AM, Mariusz Stolarczyk via FreeIPA-users wrote:

All,


I did a routine server updates last night on my IPA server. After the 
reboot I first noticed the DNS was not resolving and the ipa.service 
failed. The ipa.service failed to start so I ran the following:



# ipactl start

IPA version error: data needs to be upgraded (expected version 
'4.6.6-11.el7.centos', current version '4.6.5-11.el7.centos.4')

Automatically running upgrade, for details see /var/log/ipaupgrade.log
Be patient, this may take a few minutes.
Automatic upgrade failed: Update complete
Upgrading the configuration of the IPA services
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
[Verifying that KDC configuration is using ipa-kdb backend]
[Fix DS schema file syntax]
Syntax already fixed
[Removing RA cert from DS NSS database]
RA cert already removed
[Enable sidgen and extdom plugins by default]
[Updating HTTPD service IPA configuration]
[Updating HTTPD service IPA WSGI configuration]
Nothing to do for configure_httpd_wsgi_conf
[Updating mod_nss protocol versions]
Protocol versions already updated
[Updating mod_nss cipher suite]
[Updating mod_nss enabling OCSP]
[Fixing trust flags in /etc/httpd/alias]
Trust flags already processed
[Moving HTTPD service keytab to gssproxy]
[Removing self-signed CA]
[Removing Dogtag 9 CA]
[Checking for deprecated KDC configuration files]
[Checking for deprecated backups of Samba configuration files]
[Remove FILE: prefix from 'dedicated keytab file' in Samba configuration]

[Update 'max smbd processes' in Samba configuration to prevent unlimited 
SMBLoris attack amplification]

[Add missing CA DNS records]
IPA CA DNS records already processed
[Removing deprecated DNS configuration options]
[Ensuring minimal number of connections]
[Updating GSSAPI configuration in DNS]
[Updating pid-file configuration in DNS]

[Checking global forwarding policy in named.conf to avoid conflicts with 
automatic empty zones]

Changes to named.conf have been made, restart named
[Upgrading CA schema]
CA schema update complete (no changes)
[Verifying that CA audit signing cert has 2 year validity]
[Update certmonger certificate renewal configuration]
Certmonger certificate renewal configuration already up-to-date
[Enable PKIX certificate path discovery and validation]
PKIX already enabled
[Authorizing RA Agent to modify profiles]
[Authorizing RA Agent to manage lightweight CAs]
[Ensuring Lightweight CAs container exists in Dogtag database]
[Adding default OCSP URI configuration]
[Ensuring CA is using LDAPProfileSubsystem]
[Migrating certificate profiles to LDAP]

IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run 
command ipa-server-upgrade manually.

Unexpected error - see /var/log/ipaupgrade.log for details:

NetworkError: cannot connect to 
'https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffake-ipa-host.fake-ipa-domain.lan%3A8443%2Fca%2Frest%2Faccount%2Flogin&amp;data=02%7C01%7C%7C242bade21ef34145aace08d81d6db1fb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291704255860555&amp;sdata=32luW1pJ194Ni%2BtlneG1RSiYXydVwsg6rEgf%2BGUEMqo%3D&amp;reserved=0':



[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)

The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for 
more information



See the upgrade log for more details and/or run 
/usr/sbin/ipa-server-upgrade again

Aborting ipactl


The end of the /var/log/ipaupgrade.log file:

2020-06-29T22:43:38Z DEBUG stderr=

2020-06-29T22:43:38Z DEBUG Loading Index file from 
'/var/lib/ipa/sysrestore/sysrestore.index'

2020-06-29T22:43:38Z DEBUG Starting external process

2020-06-29T22:43:38Z DEBUG args=/usr/bin/certutil -d 
dbm:/etc/pki/pki-tomcat/alias -L -f /etc/pki/pki-tomcat/alias/pwdfile.txt

2020-06-29T22:43:38Z DEBUG Process finished, return code=0
2020-06-29T22:43:38Z DEBUG stdout=
Certificate Nickname                                        
 Trust
Attributes

                                                             
   SSL,S/MIME,JAR/XPI


caSigningCert cert-pki-ca                                    
CTu,Cu,Cu
subsystemCert cert-pki-ca                                    
u,u,u
Server-Cert cert-pki-ca                                      
u,u,u
ocspSigningCert cert-pki-ca                                  
u,u,u
auditSigningCert cert-pki-ca                                 
u,u,Pu

2020-06-29T22:43:38Z DEBUG stderr=

2020-06-29T22:43:38Z INFO Certmonger certificate renewal configuration 
already up-to-date
2020-06-29T22:43:38Z INFO [Enable PKIX certificate path discovery and 
validation]
2020-06-29T22:43:38Z DEBUG Loading StateFile from 
'/var/lib/ipa/sysupgrade/sysupgrade.state'

2020-06-29T22:43:38Z INFO PKIX already enabled
2020-06-29T22:43:38Z INFO [Authorizing RA Agent to modify profiles]
2020-06-29T22:43:38Z INFO [Authorizing RA Agent to manage lightweight CAs]

2020-06-29T22:43:38Z INFO [Ensuring Lightweight CAs container exists in 
Dogtag database]

2020-06-29T22:43:38Z DEBUG Created connection context.ldap2_140346851657552

2020-06-29T22:43:38Z DEBUG flushing 
ldapi://%2fvar%2frun%2fslapd-FAKE-IPA-DOMAIN-LAN.socket from SchemaCache
2020-06-29T22:43:38Z DEBUG retrieving schema for SchemaCache 
url=ldapi://%2fvar%2frun%2fslapd-FAKE-IPA-DOMAIN-LAN.socket 
conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7fa50c3e8e60>
2020-06-29T22:43:39Z DEBUG Destroyed connection 
context.ldap2_140346851657552

2020-06-29T22:43:39Z INFO [Adding default OCSP URI configuration]
2020-06-29T22:43:39Z INFO [Ensuring CA is using LDAPProfileSubsystem]
2020-06-29T22:43:39Z INFO [Migrating certificate profiles to LDAP]
2020-06-29T22:43:39Z DEBUG Created connection context.ldap2_140346825804304

2020-06-29T22:43:39Z DEBUG flushing 
ldapi://%2fvar%2frun%2fslapd-FAKE-IPA-DOMAIN-LAN.socket from SchemaCache
2020-06-29T22:43:39Z DEBUG retrieving schema for SchemaCache 
url=ldapi://%2fvar%2frun%2fslapd-FAKE-IPA-DOMAIN-LAN.socket 
conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7fa50ac19b90>
2020-06-29T22:43:39Z DEBUG Destroyed connection 
context.ldap2_140346825804304
2020-06-29T22:43:39Z DEBUG request GET 
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffake-ipa-host.fake-ipa-domain.lan%3A8443%2Fca%2Frest%2Faccount%2Flogin&amp;data=02%7C01%7C%7C242bade21ef34145aace08d81d6db1fb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291704255860555&amp;sdata=32luW1pJ194Ni%2BtlneG1RSiYXydVwsg6rEgf%2BGUEMqo%3D&amp;reserved=0

2020-06-29T22:43:39Z DEBUG request body ''
2020-06-29T22:43:39Z DEBUG httplib request failed:
Traceback (most recent call last):

    File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 
220, in _httplib_request

      conn.request(method, path, body=request_body, headers=headers)
    File "/usr/lib64/python2.7/httplib.py", line 1056, in request
      self._send_request(method, url, body, headers)
    File "/usr/lib64/python2.7/httplib.py", line 1090, in _send_request
      self.endheaders(body)
    File "/usr/lib64/python2.7/httplib.py", line 1052, in endheaders
      self._send_output(message_body)
    File "/usr/lib64/python2.7/httplib.py", line 890, in _send_output
      self.send(msg)
    File "/usr/lib64/python2.7/httplib.py", line 852, in send
      self.connect()
    File "/usr/lib64/python2.7/httplib.py", line 1275, in connect
      server_hostname=sni_hostname)
    File "/usr/lib64/python2.7/ssl.py", line 348, in wrap_socket
      _context=self)
    File "/usr/lib64/python2.7/ssl.py", line 609, in __init__
      self.do_handshake()
    File "/usr/lib64/python2.7/ssl.py", line 831, in do_handshake
      self._sslobj.do_handshake()

SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed 
(_ssl.c:618)
2020-06-29T22:43:39Z ERROR IPA server upgrade failed: Inspect 
/var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2020-06-29T22:43:39Z DEBUG   File 
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in 
execute

      return_value = self.run()

    File 
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", 
line 54, in run

      server.upgrade()

    File 
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", 
line 2166, in upgrade

      upgrade_configuration()

    File 
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", 
line 2038, in upgrade_configuration

      ca_enable_ldap_profile_subsystem(ca)

    File 
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", 
line 425, in ca_enable_ldap_profile_subsystem

      cainstance.migrate_profiles_to_ldap()

    File 
"/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 
2027, in migrate_profiles_to_ldap

      _create_dogtag_profile(profile_id, profile_data, overwrite=False)

    File 
"/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 
2033, in _create_dogtag_profile

      with api.Backend.ra_certprofile as profile_api:

    File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", 
line 1311, in __enter__

      method='GET'

    File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 
167, in https_request

      method=method, headers=headers)

    File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 
229, in _httplib_request

    ��  raise NetworkError(uri=uri, error=str(e))


2020-06-29T22:43:39Z DEBUG The ipa-server-upgrade command failed, 
exception: NetworkError: cannot connect to 
'https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffake-ipa-host.fake-ipa-domain.lan%3A8443%2Fca%2Frest%2Faccount%2Flogin&amp;data=02%7C01%7C%7C242bade21ef34145aace08d81d6db1fb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291704255870550&amp;sdata=W5T3upKLXylBikieXtFskvwjCUhJLLMK95PbwYrbO6g%3D&amp;reserved=0':



[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)

2020-06-29T22:43:39Z ERROR Unexpected error - see 
/var/log/ipaupgrade.log for details:
NetworkError: cannot connect to 
'https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffake-ipa-host.fake-ipa-domain.lan%3A8443%2Fca%2Frest%2Faccount%2Flogin&amp;data=02%7C01%7C%7C242bade21ef34145aace08d81d6db1fb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291704255870550&amp;sdata=W5T3upKLXylBikieXtFskvwjCUhJLLMK95PbwYrbO6g%3D&amp;reserved=0':



[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)

2020-06-29T22:43:39Z ERROR The ipa-server-upgrade command failed. See 
/var/log/ipaupgrade.log for more information



What should be my next debug steps?


Hi,

I would check whether any certificate expired:
$ getcert list

Look specifically for the "status: " and "expires: " labels. If some
certs have expired, you will need to find the CA renewal master and fix
this host first. To find the CA renewal master:
$ kinit admin
$ ipa config-show | grep "CA renewal"

If you need help, please mention:
- the output of "ipa server-role-find"
- the output of "getcert list" on all the server nodes
- are the httpd and ldap server certificates issued by IPA CA or by an
external Certificate Authority?

HTH,
flo


Thanks in advance,
-ms


_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: 
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.fedoraproject.org%2Fen-US%2Fproject%2Fcode-of-conduct%2F&amp;data=02%7C01%7C%7C242bade21ef34145aace08d81d6db1fb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291704255870550&amp;sdata=MNcJ2jAogARLqR4Unx31sSuFHACB79q7uyLCmpt5smw%3D&amp;reserved=0
List Guidelines: 
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffedoraproject.org%2Fwiki%2FMailing_list_guidelines&amp;data=02%7C01%7C%7C242bade21ef34145aace08d81d6db1fb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291704255870550&amp;sdata=WkmJlLiBmYSwav2Bh3v4HHZPqk1HbWJ5%2B8XOhWtcnrY%3D&amp;reserved=0
List Archives: 
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.fedorahosted.org%2Farchives%2Flist%2Ffreeipa-users%40lists.fedorahosted.org&amp;data=02%7C01%7C%7C242bade21ef34145aace08d81d6db1fb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291704255880545&amp;sdata=mlubUT8kK89BRnfk1uHEpKfUpcqrqX0piPkq1lT6hiM%3D&amp;reserved=0




_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: 
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.fedoraproject.org%2Fen-US%2Fproject%2Fcode-of-conduct%2F&amp;data=02%7C01%7C%7C242bade21ef34145aace08d81d6db1fb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291704255880545&amp;sdata=zEWpmZpQj4kTJrjryj99PBwZePSa1JXeJZ66lTyCyS4%3D&amp;reserved=0
List Guidelines: 
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffedoraproject.org%2Fwiki%2FMailing_list_guidelines&amp;data=02%7C01%7C%7C242bade21ef34145aace08d81d6db1fb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291704255880545&amp;sdata=QFfIP8u8X0iNZfbniQCYPphHVkbyNqtm8pPTO1ESZtw%3D&amp;reserved=0
List Archives: 
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.fedorahosted.org%2Farchives%2Flist%2Ffreeipa-users%40lists.fedorahosted.org&amp;data=02%7C01%7C%7C242bade21ef34145aace08d81d6db1fb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291704255880545&amp;sdata=mlubUT8kK89BRnfk1uHEpKfUpcqrqX0piPkq1lT6hiM%3D&amp;reserved=0




_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/[email protected]


_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/[email protected]






















					Reply via email to