On 2/8/21 4:11 PM, SRM via FreeIPA-users wrote:
I see some one else opened another thread with similar issue, but the error
messages are different so I'm going ahead & seeking help on a new thread.
I've inherited a FreeIPA installation from somebody used among 5 physical servers
with one FreeIPA server (everything CA etc on it) while other 4 physical servers
act as clients. Being someone very new at LDAP & FreeIPA, I tried to
troublshoot by googling.
System / Server Info:
OS - CentOS 7.6, Installed IPA packages version - 4.6.4, Self-Signed CA
Here are the issues & what steps I've taken so far.
1) Before certificates were expired the pki-tomcatd service was failing & I see
the following message in /var/log/pki/pki-tomcat/ca/debug:
Error: netscape.ldap.LDAPException: Authentication failed (48)
After some googling I've found this link
(https://access.redhat.com/solutions/3081821) which asks to check if certificate
blob & serial number in pkiuser matches to the 'subsystemCert cert-pki-ca' in
our case it does so there was nothing to do but we still get that error.
2) Certificates have expired - Now the certificates have expired, they were not
auto-renewed, was it because above (pki-tomcatd service failure) not sure.
2a) For this I've tried to move back the date & tried to renew them
through ipa-certupdate, the output says sucessfull but the certificates are not
getting renewed. Here is the output of one such output(renamed domain to ourorg.com
for privacy).
ipa-certupdate is not a tool for renewing expired certificates, please
refer to its man page or
https://floblanc.wordpress.com/2017/12/05/demystifying-the-certificate-authority-component-in-freeipa/
if you want to understand the various certificate-related tools in IPA.
ipapython.admintool: DEBUG: Not logging to a file
ipalib.plugable: DEBUG: importing all plugin modules in
ipaclient.remote_plugins.schema$5131ac65...
ipalib.plugable: DEBUG: importing plugin module
ipaclient.remote_plugins.schema$5131ac65.plugins
ipalib.plugable: DEBUG: importing all plugin modules in
ipaclient.plugins...
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.automember
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.automount
ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.ca
ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.cert
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.certmap
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.certprofile
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.csrgen
ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.dns
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.hbacrule
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.hbactest
ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.host
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.idrange
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.internal
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.location
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.migration
ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.misc
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.otptoken
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.otptoken_yubikey
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.passwd
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.permission
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.rpcclient
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.server
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.service
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.sudorule
ipalib.plugable: DEBUG: importing plugin module
ipaclient.plugins.topology
ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.trust
ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.user
ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.vault
ipalib.rpc: INFO: trying https://login1.ourorg.com/ipa/json
ipalib.backend: DEBUG: Created connection
context.rpcclient_139790894262416
ipalib.install.kinit: DEBUG: Initializing principal
host/[email protected] using keytab /etc/krb5.keytab
ipalib.install.kinit: DEBUG: using ccache /tmp/tmp-O7QeRu/ccache
ipalib.install.kinit: DEBUG: Attempt 1/1: success
ipalib.frontend: DEBUG: raw: ca_is_enabled(version=u'2.107')
ipalib.frontend: DEBUG: ca_is_enabled(version=u'2.107')
ipalib.rpc: INFO: [try 1]: Forwarding 'ca_is_enabled/1' to json server
'https://login1.ourorg.com/ipa/json'
ipalib.rpc: DEBUG: New HTTP connection (login1.ourorg.com)
ipalib.rpc: DEBUG: received Set-Cookie (<type
'list'>)'['ipa_session=MagBearerToken=3bDSVwqoHDuM1MRVLGVRKY2DhplAszGxcdGLUBtRRZTLVV3vj8%2bNHrexIE9KX2JdrFkcYUtCfGkQmUVoYuCUj4DRqwJBoe9Z7i3J14DadLtOVCi2fNwxNR8irDD%2fG2bn4T7ULiLR6b7k1dpS%2bXWo
iJGHOknn5EYLzi0wEOz88PauUZ7Qh1HioKfddyQhOLl1kQ6LnAsu%2fm2cACveJ8JSe2Mfmqruu8a%2fbQAIXPmRwXnC5oGN8cIk0omO4KuFQaRHWmjSNiLyG1%2bdyPiyWlxKBw%3d%3d;path=/ipa;httponly;secure;']'
ipalib.rpc: DEBUG: storing cookie
'ipa_session=MagBearerToken=3bDSVwqoHDuM1MRVLGVRKY2DhplAszGxcdGLUBtRRZTLVV3vj8%2bNHrexIE9KX2JdrFkcYUtCfGkQmUVoYuCUj4DRqwJBoe9Z7i3J14DadLtOVCi2fNwxNR8irDD%2fG2bn4T7ULiLR6b7k1dpS%2bXWoiJGHOknn5EYLzi0wEOz88P
auUZ7Qh1HioKfddyQhOLl1kQ6LnAsu%2fm2cACveJ8JSe2Mfmqruu8a%2fbQAIXPmRwXnC5oGN8cIk0omO4KuFQaRHWmjSNiLyG1%2bdyPiyWlxKBw%3d%3d;'
for principal None
ipapython.ipaldap: DEBUG: retrieving schema for SchemaCache
url=ldap://login1.ourorg.com:389 conn=<ldap.ldapobject.SimpleLDAPObject instance
at 0x7f239a5242d8>
ipalib.frontend: DEBUG: raw: ca_find(None, version=u'2.230')
ipalib.frontend: DEBUG: ca_find(None, version=u'2.230')
ipalib.rpc: INFO: [try 1]: Forwarding 'ca_find/1' to json server
'https://login1.ourorg.com/ipa/json'
ipalib.rpc: DEBUG: HTTP connection keep-alive (login1.ourorg.com)
ipalib.rpc: DEBUG: received Set-Cookie (<type
'list'>)'['ipa_session=MagBearerToken=kmtXWE4j%2buLPMXwC6RCOBvqfLCIBziy9XiM7f%2fep%2b7FYBiSPmVPwjf6USK94djhkQ6k0Rleh9KhokFWNf1AWxcH5SyVe5V6QZYLIIGzt%2fF%2f1mHl3uKOLocAauyCAz%2bVxm2FUG%2fR8ORi5
YghKrOidtRk%2bQvERwvHJKOJ8jjikvPzlWcj1x8CjO1b6ricWSigD3%2bl1UbPEYTOMKxNSL0JEW8Q0ghkPt1bryt9aEuWZVRBU%2f%2fAYnQN6WgYkrvgyBBeYXuceYPKQFtpxUmnl2js%2bDg%3d%3d;path=/ipa;httponly;secure;']'
ipalib.rpc: DEBUG: storing cookie
'ipa_session=MagBearerToken=kmtXWE4j%2buLPMXwC6RCOBvqfLCIBziy9XiM7f%2fep%2b7FYBiSPmVPwjf6USK94djhkQ6k0Rleh9KhokFWNf1AWxcH5SyVe5V6QZYLIIGzt%2fF%2f1mHl3uKOLocAauyCAz%2bVxm2FUG%2fR8ORi5YghKrOidtRk%2bQvERwvHJ
KOJ8jjikvPzlWcj1x8CjO1b6ricWSigD3%2bl1UbPEYTOMKxNSL0JEW8Q0ghkPt1bryt9aEuWZVRBU%2f%2fAYnQN6WgYkrvgyBBeYXuceYPKQFtpxUmnl2js%2bDg%3d%3d;'
for principal None
ipalib.install.sysrestore: DEBUG: Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d
dbm:/etc/dirsrv/slapd-ourorg-COM -A -n ourorg.COM IPA CA -t CT,C,C -a -f
/etc/dirsrv/slapd-ourorg-COM/pwdfile.txt
ipapython.ipautil: DEBUG: Process finished, return code=0
ipapython.ipautil: DEBUG: stdout=
ipapython.ipautil: DEBUG: stderr=
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d
dbm:/etc/dirsrv/slapd-ourorg-COM -A -n ourorg.COM IPA CA -t CT,C,C -a -f
/etc/dirsrv/slapd-ourorg-COM/pwdfile.txt
ipapython.ipautil: DEBUG: Process finished, return code=0
ipapython.ipautil: DEBUG: stdout=
ipapython.ipautil: DEBUG: stderr=
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/bin/systemctl is-active
[email protected]
ipapython.ipautil: DEBUG: Process finished, return code=0
ipapython.ipautil: DEBUG: stdout=active
ipapython.ipautil: DEBUG: stderr=
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/bin/systemctl --system daemon-reload
ipapython.ipautil: DEBUG: Process finished, return code=0
ipapython.ipautil: DEBUG: stdout=
ipapython.ipautil: DEBUG: stderr=
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/bin/systemctl restart
[email protected]
ipapython.ipautil: DEBUG: Process finished, return code=0
ipapython.ipautil: DEBUG: stdout=
ipapython.ipautil: DEBUG: stderr=
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/bin/systemctl is-active
[email protected]
ipapython.ipautil: DEBUG: Process finished, return code=0
ipapython.ipautil: DEBUG: stdout=active
ipapython.ipautil: DEBUG: stderr=
ipapython.ipautil: DEBUG: wait_for_open_ports: localhost [389] timeout
300
ipapython.ipautil: DEBUG: waiting for port: 389
ipapython.ipautil: DEBUG: SUCCESS: port: 389
ipaplatform.base.services: DEBUG: Restart of [email protected]
complete
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d
dbm:/etc/httpd/alias -A -n ourorg.COM IPA CA -t CT,C,C -a -f
/etc/httpd/alias/pwdfile.txt
ipapython.ipautil: DEBUG: Process finished, return code=0
ipapython.ipautil: DEBUG: stdout=
ipapython.ipautil: DEBUG: stderr=
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d
dbm:/etc/httpd/alias -A -n ourorg.COM IPA CA -t CT,C,C -a -f
/etc/httpd/alias/pwdfile.txt
ipapython.ipautil: DEBUG: Process finished, return code=0
ipapython.ipautil: DEBUG: stdout=
ipapython.ipautil: DEBUG: stderr=
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/bin/systemctl is-active httpd.service
ipapython.ipautil: DEBUG: Process finished, return code=0
ipapython.ipautil: DEBUG: stdout=active
ipapython.ipautil: DEBUG: stderr=
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/bin/systemctl restart httpd.service
ipapython.ipautil: DEBUG: Process finished, return code=0
ipapython.ipautil: DEBUG: stdout=
ipapython.ipautil: DEBUG: stderr=
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/bin/systemctl is-active httpd.service
ipapython.ipautil: DEBUG: Process finished, return code=0
ipapython.ipautil: DEBUG: stdout=active
ipapython.ipautil: DEBUG: stderr=
ipaplatform.base.services: DEBUG: Restart of httpd.service complete
ipaclient.install.ipa_certupdate: DEBUG: resubmitting certmonger
request '20190129222612'
ipalib.install.certmonger: DEBUG: certmonger request is in state
dbus.String(u'GENERATING_CSR', variant_level=1)
ipalib.install.certmonger: DEBUG: certmonger request is in state
dbus.String(u'POST_SAVED_CERT', variant_level=1)
ipalib.install.certmonger: DEBUG: certmonger request is in state
dbus.String(u'POST_SAVED_CERT', variant_level=1)
ipalib.install.certmonger: DEBUG: certmonger request is in state
dbus.String(u'POST_SAVED_CERT', variant_level=1)
ipalib.install.certmonger: DEBUG: certmonger request is in state
dbus.String(u'POST_SAVED_CERT', variant_level=1)
ipalib.install.certmonger: DEBUG: certmonger request is in state
dbus.String(u'POST_SAVED_CERT', variant_level=1)
ipalib.install.certmonger: DEBUG: certmonger request is in state
dbus.String(u'POST_SAVED_CERT', variant_level=1)
ipalib.install.certmonger: DEBUG: certmonger request is in state
dbus.String(u'POST_SAVED_CERT', variant_level=1)
ipalib.install.certmonger: DEBUG: certmonger request is in state
dbus.String(u'MONITORING', variant_level=1)
ipaclient.install.ipa_certupdate: DEBUG: modifying certmonger request
'20190129222612'
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb
-L -n IPA CA -a -f /etc/ipa/nssdb/pwdfile.txt
ipapython.ipautil: DEBUG: Process finished, return code=255
ipapython.ipautil: DEBUG: stdout=
ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: IPA CA
: PR_FILE_NOT_FOUND_ERROR: File not found
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb
-L -n External CA cert -a -f /etc/ipa/nssdb/pwdfile.txt
ipapython.ipautil: DEBUG: Process finished, return code=255
ipapython.ipautil: DEBUG: stdout=
ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert:
External CA cert
: PR_FILE_NOT_FOUND_ERROR: File not found
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb
-A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/ipa/nssdb/pwdfile.txt
ipapython.ipautil: DEBUG: Process finished, return code=0
ipapython.ipautil: DEBUG: stdout=
ipapython.ipautil: DEBUG: stderr=
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb
-A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/ipa/nssdb/pwdfile.txt
ipapython.ipautil: DEBUG: Process finished, return code=0
ipapython.ipautil: DEBUG: stdout=
ipapython.ipautil: DEBUG: stderr=
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/usr/bin/update-ca-trust
ipapython.ipautil: DEBUG: Process finished, return code=0
ipapython.ipautil: DEBUG: stdout=
ipapython.ipautil: DEBUG: stderr=
ipaplatform.redhat.tasks: INFO: Systemwide CA database updated.
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/usr/bin/update-ca-trust
ipapython.ipautil: DEBUG: Process finished, return code=0
ipapython.ipautil: DEBUG: stdout=
ipapython.ipautil: DEBUG: stderr=
ipaplatform.redhat.tasks: INFO: Systemwide CA database updated.
ipalib.backend: DEBUG: Destroyed connection
context.rpcclient_139790894262416
ipapython.admintool: INFO: The ipa-certupdate command was successful
In above output there are two occasions where it is mentioned
"ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: IPA CA :
PR_FILE_NOT_FOUND_ERROR: File not found" not sure if these are relevant, if so how
to debug
2b) I've also used "ipa-cacert-manage renew" following this link
https://www.freeipa.org/page/V4/CA_certificate_renewal. Not sure if this was necessary or
if doing this caused any more issues.
This tool renews IPA CA, not the other certificates, but it's highly
unlikely that the CA cert was expired.
Since the deployment has only one IPA server, you need to fix this
server. Please provide the output of "getcert list", it will show the
expiration dates for all the certificates tracked by certmonger. You
will need to change the system date to a date where all the certificates
were still valid, start the services (but not ntp/chrony) and let
certmonger renew the certs, then move back the date to the current date.
flo
3) Since certificates have expired kerberos broke, can't do "kinit admin" any longer.
Can't change passwords / create users & of course can't access webui. For any of these
actions need to move the date back. For now 'sudo' works (without having to move the date back)
& general logins work, but not sure how long they continue to work before completely break?.
4) This is a production installation with hardly any time to take down FreeIPA
let alone physical server. Is there any way to recover from this situation?.
5) If it can't be recovered can we setup another FreeIPA server installation with
the same realm / domain(need to procure another system /server) with a new CA &
etc from scratch and make all the current 5 physical servers (including current
broken FreeIPA server) as clients to the new FreeIPA installation with the same
domain / realm?.
_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/[email protected]
_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/[email protected]
