This gave me a huge return! Appears to be a long list of all the servers and applications whose users authenticate to the IPA servers.
ldapsearch -x -h 172.16.100.161 -D "cn=directory manager" -W -b "dc=itmodev,dc=gov" '(objectclass=krbprincipal)' # search result search: 2 result: 0 Success # numResponses: 142 # numEntries: 141 -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Ludwig Krispenz Sent: Tuesday, November 10, 2015 11:37 AM To: [email protected] Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos authentication error) what do you get if you search for "objectclass=krbprincipal" ? On 11/10/2015 05:27 PM, Rich Megginson wrote: > On 11/10/2015 09:16 AM, Gronde, Christopher (Contractor) wrote: >> Neither came back with anything >> >> # ldapsearch -x -h 172.16.100.161 -D "cn=directory manager" -W -b >> "dc=itmodev,dc=gov" '(uid=ldap/comipa01.itmodev.gov)' >> Enter LDAP Password: >> # extended LDIF >> # >> # LDAPv3 >> # base <dc=itmodev,dc=gov> with scope subtree # filter: >> (uid=ldap/comipa01.itmodev.gov) # requesting: ALL # >> >> # search result >> search: 2 >> result: 0 Success >> >> # numResponses: 1 >> [root@comipa02 ~]# ldapsearch -x -h 172.16.100.161 -D "cn=directory >> manager" -W -b "dc=itmodev,dc=gov" '(uid=ldap/*.gov)' uid Enter LDAP >> Password: >> # extended LDIF >> # >> # LDAPv3 >> # base <dc=itmodev,dc=gov> with scope subtree # filter: >> (uid=ldap/*.gov) # requesting: uid # >> >> # search result >> search: 2 >> result: 0 Success >> >> # numResponses: 1 > > That means this server has no LDAP service principals? I'm not sure > how to recover IPA from this scenario. > >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of Rich Megginson >> Sent: Tuesday, November 10, 2015 11:04 AM >> To: [email protected] >> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos >> authentication error) >> >> On 11/10/2015 08:18 AM, Gronde, Christopher (Contractor) wrote: >>> Thank you! I should have caught that... >>> >>> I changed the log level and then restarted dirsrv and attempted to >>> start krb5kdc and got the following... >> <snip> >> >> [10/Nov/2015:10:12:02 -0500] conn=5 fd=64 slot=64 connection from >> 172.16.100.208 to 172.16.100.161 >> [10/Nov/2015:10:12:02 -0500] conn=5 op=0 BIND dn="" method=sasl >> version=3 mech=GSSAPI >> [10/Nov/2015:10:12:03 -0500] conn=5 op=0 RESULT err=14 tag=97 >> nentries=0 etime=1, SASL bind in progress >> [10/Nov/2015:10:12:03 -0500] conn=5 op=1 BIND dn="" method=sasl >> version=3 mech=GSSAPI >> [10/Nov/2015:10:12:03 -0500] conn=5 op=1 RESULT err=14 tag=97 >> nentries=0 etime=0, SASL bind in progress >> [10/Nov/2015:10:12:03 -0500] conn=5 op=2 BIND dn="" method=sasl >> version=3 mech=GSSAPI >> [10/Nov/2015:10:12:03 -0500] conn=Internal op=-1 SRCH >> base="dc=itmodev,dc=gov" scope=2 >> filter="(uid=ldap/comipa01.itmodev.gov)" attrs=ALL >> [10/Nov/2015:10:12:03 -0500] conn=Internal op=-1 RESULT err=0 tag=48 >> nentries=0 etime=0 >> [10/Nov/2015:10:12:03 -0500] conn=5 op=2 RESULT err=49 tag=97 >> nentries=0 >> etime=0 >> [10/Nov/2015:10:12:03 -0500] conn=5 op=3 UNBIND >> [10/Nov/2015:10:12:03 -0500] conn=5 op=3 fd=64 closed - U1 >> >> <snip> >> >> This is the SASL bind. It thinks the principal in the Kerberos >> credential is "ldap/comipa01.itmodev.gov", and the SASL map tells the >> code to look for something with uid=ldap/comipa01.itmodev.gov under >> dc=itmodev,dc=gov. However, this entry is not found: RESULT err=0 >> tag=48 nentries=0. nentries=0 means no entries matched the search >> criteria. >> >> You can do the search yourself with ldapsearch: >> >> ldapsearch -x -h 172.16.100.161 -D "cn=directory manager" -W -b >> "dc=itmodev,dc=gov" '(uid=ldap/comipa01.itmodev.gov)' >> >> If you want to find out if there is some other ldap principal, do a >> search like this: >> >> ldapsearch -x -h 172.16.100.161 -D "cn=directory manager" -W -b >> "dc=itmodev,dc=gov" '(uid=ldap/*.gov)' uid >> >>>> Ran into an error trying to set that >>>> >>>> # ldapmodify -a -D "cn=directory manager" -W Enter LDAP Password: >>>> dn: cn=config >>>> changetype: modify >>>> replace: nsslapd-acesslog-level >>>> : 260 >>>> >>>> modifying entry "cn=config" >>>> ldap_modify: Server is unwilling to perform (53) >>>> additional info: Unknown attribute >>>> nsslapd-acesslog-level will be ignored >>>> >>>> [root@comipa02 ~]# ldapmodify -a -D "cn=config" -W Enter LDAP >>>> Password: >>>> ldap_bind: Inappropriate authentication (48) >>>> >>>> -----Original Message----- >>>> From: Ludwig Krispenz [mailto:[email protected]] >>>> Sent: Tuesday, November 10, 2015 9:48 AM >>>> To: Gronde, Christopher (Contractor) >>>> <[email protected]> >>>> Cc: [email protected] >>>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos >>>> authentication error) >>>> >>>> >>>> On 11/10/2015 03:32 PM, Gronde, Christopher (Contractor) wrote: >>>>> How do I change that log setting? Is that done in LDAP? Using >>>>> ldapmodify? >>>> yes, >>>> ldapmodify ... >>>> dn: cn=config >>>> changetype: modify >>>> replace: nsslapd-acesslog-level >>>> nsslapd-acesslog-level: 260 >>>>> -----Original Message----- >>>>> From: [email protected] >>>>> [mailto:[email protected]] On Behalf Of Ludwig >>>>> Krispenz >>>>> Sent: Tuesday, November 10, 2015 9:03 AM >>>>> To: [email protected] >>>>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos >>>>> authentication error) >>>>> >>>>> >>>>> On 11/10/2015 02:40 PM, Alexander Bokovoy wrote: >>>>>> On Tue, 10 Nov 2015, Gronde, Christopher (Contractor) wrote: >>>>>>> Where can I verify or change the credentials it is trying to use? >>>>>>> Is it my LDAP password? >>>>>> No, according to your logs, it is your LDAP master trying to >>>>>> replicate (push changes) to your LDAP replica: >>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 fd=64 slot=64 connection >>>>>>>> from <MASTER_IP> to <REPLICA_IP> >>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=0 BIND dn="" method=sasl >>>>>>>> version=3 mech=GSSAPI >>>>> err=49 could also be a result if the entry which is mapped from >>>>> the principal is not found in the directory. A bit more info could >>>>> be gained by enabling logging of internal searches. >>>>> Set nsslapd-acesslog-level: 260 >>>>> >>>>> and then look what internal searches are done during the gssapi >>>>> authentication >>>>>> If that is true, it would be ldap/<master> Kerberos principal >>>>>> talking to ldap/<replica> Kerberos principal. If that fails, it >>>>>> means master and replica KDCs have different understanding of >>>>>> both ldap/<master> and ldap/<replica> keys which most likely >>>>>> means keys were rotated on master and weren't propagated to replica. >>>>>> >>>>>> How to solve it? One possibility is to set master's hostname as >>>>>> KDC address in krb5.conf on replica, forcing LDAP server on >>>>>> replica to use master's KDC. I'm absolutely not sure this will >>>>>> actually work but at least it allows to see if we are indeed >>>>>> dealing with inconsistent state of service principals' keys. >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Alexander Bokovoy [mailto:[email protected]] >>>>>>> Sent: Tuesday, November 10, 2015 8:18 AM >>>>>>> To: Gronde, Christopher (Contractor) >>>>>>> <[email protected]> >>>>>>> Cc: Rob Crittenden <[email protected]>; >>>>>>> [email protected] >>>>>>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos >>>>>>> authentication error) >>>>>>> >>>>>>> On Tue, 10 Nov 2015, Gronde, Christopher (Contractor) wrote: >>>>>>>> When I tried to start the service again I got no response from >>>>>>>> tail of the log, but this is a repeating entry I see in the >>>>>>>> access log >>>>>>>> >>>>>>>> [09/Nov/2015:15:01:04 -0500] conn=1 fd=64 slot=64 connection >>>>>>>> from >>>>>>>> 127.0.0.1 to 127.0.0.1 >>>>>>>> [09/Nov/2015:15:01:04 -0500] conn=1 op=-1 fd=64 closed - B1 >>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 fd=64 slot=64 connection >>>>>>>> from <MASTER_IP> to <REPLICA_IP> >>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=0 BIND dn="" method=sasl >>>>>>>> version=3 mech=GSSAPI >>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=0 RESULT err=14 tag=97 >>>>>>>> nentries=0 etime=0, SASL bind in progress >>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=1 BIND dn="" method=sasl >>>>>>>> version=3 mech=GSSAPI >>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=1 RESULT err=14 tag=97 >>>>>>>> nentries=0 etime=0, SASL bind in progress >>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=2 BIND dn="" method=sasl >>>>>>>> version=3 mech=GSSAPI >>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=2 RESULT err=49 tag=97 >>>>>>>> nentries=0 etime=0 >>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=3 UNBIND >>>>>>>> [09/Nov/2015:15:02:01 -0500] conn=2 op=3 fd=64 closed - U1 >>>>>>>> >>>>>>>> Does anyone know what err=14 or err=49 are? >>>>>>> err=14 means SASL bind in progress -- i.e. multi-round >>>>>>> processing is ongoing. This is normal for SASL GSSAPI. >>>>>>> >>>>>>> err=49 is wrong password or username, i.e. credentials were >>>>>>> incorrect. >>>>>>> It may also mean that LDAP server side was unable to process >>>>>>> Kerberos negotiation due to not having a current Kerberos ticket >>>>>>> for own service >>>>>>> (LDAP) and trying to request it from the Kerberos KDC but >>>>>>> Kerberos KDC is down. >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Rob Crittenden [mailto:[email protected]] >>>>>>>> Sent: Monday, November 09, 2015 3:26 PM >>>>>>>> To: Gronde, Christopher (Contractor) >>>>>>>> <[email protected]>; Alexander Bokovoy >>>>>>>> <[email protected]> >>>>>>>> Cc: [email protected] >>>>>>>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos >>>>>>>> authentication error) >>>>>>>> >>>>>>>> Gronde, Christopher (Contractor) wrote: >>>>>>>>> Nothing bad came back and there is definitely data in the tree. >>>>>>>> Ok, I guess I'd try to start the kdc again and then watch the >>>>>>>> 389-ds access log (buffered) to: >>>>>>>> >>>>>>>> 1. See if it is binding at all >>>>>>>> 2. See what the search is and what, if any, results were >>>>>>>> returned >>>>>>>> >>>>>>>> This would be in /var/log/dirsrv/slapd-YOUR_REALM/access >>>>>>>> >>>>>>>> rob >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Rob Crittenden [mailto:[email protected]] >>>>>>>>> Sent: Monday, November 09, 2015 11:46 AM >>>>>>>>> To: Gronde, Christopher (Contractor) >>>>>>>>> <[email protected]>; Alexander Bokovoy >>>>>>>>> <[email protected]> >>>>>>>>> Cc: [email protected] >>>>>>>>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos >>>>>>>>> authentication error) >>>>>>>>> >>>>>>>>> Gronde, Christopher (Contractor) wrote: >>>>>>>>>> I restarted dirsrv and attempted to start krb5kdc and this is >>>>>>>>>> what the error log shows >>>>>>>>>> >>>>>>>>>> # tail /var/log/dirsrv/slapd-ITMODEV-GOV/errors >>>>>>>>>> [09/Nov/2015:11:01:02 -0500] - WARNING: userRoot: entry cache >>>>>>>>>> size 10485760B is less than db size 28016640B; We recommend >>>>>>>>>> to increase the entry cache size nsslapd-cachememsize. >>>>>>>>>> [09/Nov/2015:11:01:02 -0500] - slapd started. Listening on >>>>>>>>>> All Interfaces port 389 for LDAP requests >>>>>>>>>> [09/Nov/2015:11:06:04 -0500] - slapd shutting down - >>>>>>>>>> signaling operation threads >>>>>>>>>> [09/Nov/2015:11:06:04 -0500] - slapd shutting down - closing >>>>>>>>>> down internal subsystems and plugins >>>>>>>>>> [09/Nov/2015:11:06:04 -0500] - Waiting for 4 database threads >>>>>>>>>> to stop >>>>>>>>>> [09/Nov/2015:11:06:04 -0500] - All database threads now >>>>>>>>>> stopped >>>>>>>>>> [09/Nov/2015:11:06:04 -0500] - slapd stopped. >>>>>>>>>> [09/Nov/2015:11:14:20 -0500] - 389-Directory/1.2.11.15 >>>>>>>>>> B2015.247.1737 starting up >>>>>>>>>> [09/Nov/2015:11:14:20 -0500] - WARNING: userRoot: entry cache >>>>>>>>>> size 10485760B is less than db size 28016640B; We recommend >>>>>>>>>> to increase the entry cache size nsslapd-cachememsize. >>>>>>>>>> [09/Nov/2015:11:14:20 -0500] - slapd started. Listening on >>>>>>>>>> All Interfaces port 389 for LDAP requests >>>>>>>>> Ok, that's good. >>>>>>>>> >>>>>>>>> I'd do something like this to see what is in the db >>>>>>>>> (substitute example.com with your domain): >>>>>>>>> >>>>>>>>> $ ldapsearch -x -D 'cn=Directory Manager' -W -s one -b >>>>>>>>> cn=kerberos,dc=example,dc=com >>>>>>>>> >>>>>>>>> (don't post the output as it would include the kerberos master >>>>>>>>> key). >>>>>>>>> >>>>>>>>> If that returns nothing that's bad. >>>>>>>>> >>>>>>>>> If it succeeds I'd broaden the search base a bit to see what >>>>>>>>> data you do >>>>>>>>> have: >>>>>>>>> >>>>>>>>> $ ldapsearch -x -D 'cn=Directory Manager' -W -b >>>>>>>>> cn=groups,cn=accounts,dc=example,dc=com >>>>>>>>> >>>>>>>>> I picked groups because usually groups << users in numbers. >>>>>>>>> This is just to see if you have data in the tree. >>>>>>>>> >>>>>>>>> Let us know if either or both turns up nothing. >>>>>>>>> >>>>>>>>> rob >>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Alexander Bokovoy [mailto:[email protected]] >>>>>>>>>> Sent: Monday, November 09, 2015 10:51 AM >>>>>>>>>> To: Gronde, Christopher (Contractor) >>>>>>>>>> <[email protected]> >>>>>>>>>> Cc: [email protected] >>>>>>>>>> Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos >>>>>>>>>> authentication error) >>>>>>>>>> >>>>>>>>>> On Mon, 09 Nov 2015, Gronde, Christopher (Contractor) wrote: >>>>>>>>>>> Hello all! >>>>>>>>>>> >>>>>>>>>>> On my replica IPA server after fixing a cert issue that had >>>>>>>>>>> been going on for sometime, I have all my certs figured out >>>>>>>>>>> but the krb5kdc service will not start. >>>>>>>>>>> >>>>>>>>>>> # service krb5kdc start >>>>>>>>>>> Starting Kerberos 5 KDC: krb5kdc: cannot initialize realm >>>>>>>>>>> ITMODEV.GOV - see log file for details >>>>>>>>>>> [FAILED] >>>>>>>>>>> >>>>>>>>>>> # cat /var/log/krb5kdc.log >>>>>>>>>>> krb5kdc: Server error - while fetching master key K/M for >>>>>>>>>>> realm ITMODEV.GOV >>>>>>>>>>> krb5kdc: Server error - while fetching master key K/M for >>>>>>>>>>> realm ITMODEV.GOV >>>>>>>>>>> krb5kdc: Server error - while fetching master key K/M for >>>>>>>>>>> realm ITMODEV.GOV >>>>>>>>>>> >>>>>>>>>>> I found this article online: >>>>>>>>>>> http://research.imb.uq.edu.au/~l.rathbone/ldap/kerberos.shtm >>>>>>>>>>> l >>>>>>>>>>> >>>>>>>>>>> Which stated it might be because The slave KDC does not have >>>>>>>>>>> a stash file (.k5.EXAMPLE.COM). You need to create one. >>>>>>>>>>> Tried the command >>>>>>>>>>> listed: >>>>>>>>>>> >>>>>>>>>>> # kdb5_util stash >>>>>>>>>>> kdb5_util: Server error while retrieving master entry >>>>>>>>>>> >>>>>>>>>>> No further information found on the proceeding error above >>>>>>>>>>> for the kdb5_util command. >>>>>>>>>>> >>>>>>>>>>> Any thoughts? >>>>>>>>>> First: don't use instructions which are not related to IPA, >>>>>>>>>> please. >>>>>>>>>> >>>>>>>>>> FreeIPA has its own LDAP driver for KDC and instructions for >>>>>>>>>> anything else do not apply here at all. >>>>>>>>>> >>>>>>>>>> If you see 'Server error - while fetching master key ..' it >>>>>>>>>> means KDC LDAP driver was unable to contact LDAP server. Does >>>>>>>>>> LDAP server work on the replica? What is in its error log >>>>>>>>>> (/var/log/dirsrv/slapd-ITMODEV-GOV/errors)? >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> / Alexander Bokovoy >>>>>>>>>> >>>>>>>>>> >>>>>>> -- >>>>>>> / Alexander Bokovoy >>>>>>> >>>>> -- >>>>> Manage your subscription for the Freeipa-users mailing list: >>>>> https://www.redhat.com/mailman/listinfo/freeipa-users >>>>> Go to http://freeipa.org for more info on the project >>>>> >>> >> -- >> Manage your subscription for the Freeipa-users mailing list: >> https://www.redhat.com/mailman/listinfo/freeipa-users >> Go to http://freeipa.org for more info on the project >> > -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
