Sorry about this post. I sent this email to the list 3 times over the last 48 hours and it was finally accepted after the 3rd send when I changed the subject to something totally not descriptive of my problem. Original email with original subject also finally posted today :(
> We have a FreeIPA domain running IPA server 4.1.4 on CentOS 7. > > We have no per zone forwarding enabled, only a single global forwarder. > This seems to work fine, but then after a while (several weeks I think) > will randomly stop working. > > We had this issue several weeks ago on a different IPA domain (identical > setup) in our production network but it was ignored because a server > restart fixed it. > > This issue then re-surfaced in our development domain today (different > network, different physical hardware, same OS and IPA versions). > > I received a report today from a developer that he could not ping a > machine in another domain so I verified network connectivity and > everything was fine. When I tried to resolve the name from the IPA dc > using ping it would fail, but nslookup directly to the forward server > worked fine. > > ipactl showed no issues, and only after I restarted the server did the > lookups start working again. > > Console log below : > > Using username "myipausername". > Last login: Thu Oct 1 16:36:51 2015 from 10.5.5.57 > [myipausername@dc1 ~]$ sudo su - > Last login: Tue Sep 29 19:03:39 UTC 2015 on pts/3 > > ATTEMPT FIRST PING TO UNRESOLVABLE HOST > ======================================= > [root@dc1 ~]# ping artifactory.externaldomain.net > ping: unknown host artifactory.externaldomain.net > > CHECK IPA STATUS > ================ > [root@dc1 ~]# ipactl status > Directory Service: RUNNING > krb5kdc Service: RUNNING > kadmin Service: RUNNING > named Service: RUNNING > ipa_memcached Service: RUNNING > httpd Service: RUNNING > pki-tomcatd Service: RUNNING > smb Service: RUNNING > winbind Service: RUNNING > ipa-otpd Service: RUNNING > ipa-dnskeysyncd Service: RUNNING > ipa: INFO: The ipactl command was successful > > ATTEMPT PING OF GLOBAL FORWARDER > ================================ > [root@dc1 ~]# ping 10.21.0.14 > PING 10.21.0.14 (10.21.0.14) 56(84) bytes of data. > 64 bytes from 10.21.0.14: icmp_seq=1 ttl=64 time=0.275 ms > 64 bytes from 10.21.0.14: icmp_seq=2 ttl=64 time=0.327 ms > ^C > --- 10.21.0.14 ping statistics --- > 2 packets transmitted, 2 received, 0% packet loss, time 1000ms > rtt min/avg/max/mdev = 0.275/0.301/0.327/0.026 ms > > MANUAL NSLOOKUP OF DOMAIN ON GLOBAL FORWARDER FROM IPA DC > ========================================================= > [root@dc1 ~]# nslookup >> server 10.21.0.14 > Default server: 10.21.0.14 > Address: 10.21.0.14#53 >> artifactory.externaldomain.net > Server: 10.21.0.14 > Address: 10.21.0.14#53 > > Non-authoritative answer: > artifactory.externaldomain.net canonical name = > van-artifactory1.externaldomain.net. > Name: van-artifactory1.externaldomain.net > Address: 10.20.10.14 > > RE-ATTEMPT PING SINCE WE KNOW THAT NAME RESOLUTION (at least via nslookup > IS WORKING FROM THIS MACHINE > ====================================================================================================== >> ^C[root@dc1 ~]# ping artifactory.externaldomain.net > ping: unknown host artifactory.externaldomain.net > [root@dc1 ~]# ping van-artifactory1.externaldomain.net > ping: unknown host van-artifactory1.externaldomain.net > > RESTART IPA SERVICES > ==================== > [root@dc1 ~]# ipactl restart > Restarting Directory Service > Restarting krb5kdc Service > Restarting kadmin Service > Restarting named Service > Restarting ipa_memcached Service > Restarting httpd Service > Restarting pki-tomcatd Service > Restarting smb Service > Restarting winbind Service > Restarting ipa-otpd Service > Restarting ipa-dnskeysyncd Service > ipa: INFO: The ipactl command was successful > [root@dc1 ~]# ipa dnsconfig-show > ipa: ERROR: did not receive Kerberos credentials > [root@dc1 ~]# kinit myipausername > Password for [email protected]: > > OUTPUT GLOBAL FORWARDER CONFIG FOR TROUBLESHOOTING > ================================================== > [root@dc1 ~]# ipa dnsconfig-show > Global forwarders: 10.21.0.14 > Allow PTR sync: TRUE > > PING NOW WORKS BECAUSE IPA SERVICES WERE RESTARTED > ================================================== > [root@dc1 ~]# ping artifactory.externaldomain.net > PING van-artifactory1.externaldomain.net (10.20.10.14) 56(84) bytes of > data. > 64 bytes from 10.20.10.14: icmp_seq=1 ttl=60 time=3.00 ms > 64 bytes from 10.20.10.14: icmp_seq=2 ttl=60 time=1.42 ms > 64 bytes from 10.20.10.14: icmp_seq=3 ttl=60 time=2.39 ms > ^C > --- van-artifactory1.externaldomain.net ping statistics --- > 3 packets transmitted, 3 received, 0% packet loss, time 2004ms > rtt min/avg/max/mdev = 1.420/2.274/3.004/0.653 ms > [root@dc1 ~]# > > Here are some strange enties from my /var/log/messages relating to errors > from today : > > Oct 1 20:39:31 dc1 named-pkcs11[15066]: checkhints: unable to get root NS > rrset from cache: not found > Oct 1 20:39:17 dc1 named-pkcs11[15066]: error (network unreachable) > resolving 'pmdb1.ipadomain.net/A/IN': 2001:500:2f::f#53 > Oct 1 20:39:17 dc1 named-pkcs11[15066]: error (network unreachable) > resolving 'pmdb1.ipadomain.net/AAAA/IN': 2001:500:2f::f#53 > > Looking at the log entries, it appears that there may have been a network > connectivity 'blip' (maybe a switch or router was restarted) at some point > and even after connectivity was restored, the global forwarding was > failing because the "we can't contact our forwarder" status seemed to get > stuck in memory. > > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project > -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
