We have a FreeIPA domain running IPA server 4.1.4 on CentOS 7. We have no per zone forwarding enabled, only a single global forwarder. This seems to work fine, but then after a while (several weeks I think) will randomly stop working.
We had this issue several weeks ago on a different IPA domain (identical setup) in our production network but it was ignored because a server restart fixed it. This issue then re-surfaced in our development domain today (different network, different physical hardware, same OS and IPA versions). I received a report today from a developer that he could not ping a machine in another domain so I verified network connectivity and everything was fine. When I tried to resolve the name from the IPA dc using ping it would fail, but nslookup directly to the forward server worked fine. ipactl showed no issues, and only after I restarted the server did the lookups start working again. Console log below : Using username "myipausername". Last login: Thu Oct 1 16:36:51 2015 from 10.5.5.57 [myipausername@dc1 ~]$ sudo su - Last login: Tue Sep 29 19:03:39 UTC 2015 on pts/3 ATTEMPT FIRST PING TO UNRESOLVABLE HOST ======================================= [root@dc1 ~]# ping artifactory.externaldomain.net ping: unknown host artifactory.externaldomain.net CHECK IPA STATUS ================ [root@dc1 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING ipa_memcached Service: RUNNING httpd Service: RUNNING pki-tomcatd Service: RUNNING smb Service: RUNNING winbind Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful ATTEMPT PING OF GLOBAL FORWARDER ================================ [root@dc1 ~]# ping 10.21.0.14 PING 10.21.0.14 (10.21.0.14) 56(84) bytes of data. 64 bytes from 10.21.0.14: icmp_seq=1 ttl=64 time=0.275 ms 64 bytes from 10.21.0.14: icmp_seq=2 ttl=64 time=0.327 ms ^C --- 10.21.0.14 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1000ms rtt min/avg/max/mdev = 0.275/0.301/0.327/0.026 ms MANUAL NSLOOKUP OF DOMAIN ON GLOBAL FORWARDER FROM IPA DC ========================================================= [root@dc1 ~]# nslookup > server 10.21.0.14 Default server: 10.21.0.14 Address: 10.21.0.14#53 > artifactory.externaldomain.net Server: 10.21.0.14 Address: 10.21.0.14#53 Non-authoritative answer: artifactory.externaldomain.net canonical name = van-artifactory1.externaldomain.net. Name: van-artifactory1.externaldomain.net Address: 10.20.10.14 RE-ATTEMPT PING SINCE WE KNOW THAT NAME RESOLUTION (at least via nslookup IS WORKING FROM THIS MACHINE ====================================================================================================== > ^C[root@dc1 ~]# ping artifactory.externaldomain.net ping: unknown host artifactory.externaldomain.net [root@dc1 ~]# ping van-artifactory1.externaldomain.net ping: unknown host van-artifactory1.externaldomain.net RESTART IPA SERVICES ==================== [root@dc1 ~]# ipactl restart Restarting Directory Service Restarting krb5kdc Service Restarting kadmin Service Restarting named Service Restarting ipa_memcached Service Restarting httpd Service Restarting pki-tomcatd Service Restarting smb Service Restarting winbind Service Restarting ipa-otpd Service Restarting ipa-dnskeysyncd Service ipa: INFO: The ipactl command was successful [root@dc1 ~]# ipa dnsconfig-show ipa: ERROR: did not receive Kerberos credentials [root@dc1 ~]# kinit myipausername Password for [email protected]: OUTPUT GLOBAL FORWARDER CONFIG FOR TROUBLESHOOTING ================================================== [root@dc1 ~]# ipa dnsconfig-show Global forwarders: 10.21.0.14 Allow PTR sync: TRUE PING NOW WORKS BECAUSE IPA SERVICES WERE RESTARTED ================================================== [root@dc1 ~]# ping artifactory.externaldomain.net PING van-artifactory1.externaldomain.net (10.20.10.14) 56(84) bytes of data. 64 bytes from 10.20.10.14: icmp_seq=1 ttl=60 time=3.00 ms 64 bytes from 10.20.10.14: icmp_seq=2 ttl=60 time=1.42 ms 64 bytes from 10.20.10.14: icmp_seq=3 ttl=60 time=2.39 ms ^C --- van-artifactory1.externaldomain.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.420/2.274/3.004/0.653 ms [root@dc1 ~]# Here are some strange enties from my /var/log/messages relating to errors from today : Oct 1 20:39:31 dc1 named-pkcs11[15066]: checkhints: unable to get root NS rrset from cache: not found Oct 1 20:39:17 dc1 named-pkcs11[15066]: error (network unreachable) resolving 'pmdb1.ipadomain.net/A/IN': 2001:500:2f::f#53 Oct 1 20:39:17 dc1 named-pkcs11[15066]: error (network unreachable) resolving 'pmdb1.ipadomain.net/AAAA/IN': 2001:500:2f::f#53 Looking at the log entries, it appears that there may have been a network connectivity 'blip' (maybe a switch or router was restarted) at some point and even after connectivity was restored, the global forwarding was failing because the "we can't contact our forwarder" status seemed to get stuck in memory. -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
