** Description changed: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) - for example, (lvsip:50000 -> dnsip:53) + - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:50000 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:50000 -> dnsip:53). The - application layer then sees the response from (lvsip:53), which is + application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): - Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 - Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 + Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 + Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1681847 Title: IPVS incorrectly reverse-NATs traffic to LVS host Status in linux package in Ubuntu: Incomplete Bug description: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:50000 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:50000 -> dnsip:53). The application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1681847/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
