Hi all, Can I request some help in resolving a vlan networking issue we are encountering in the final stages of our openstack installation?
We have installed a multi host vlan network configuration on 3 hosts all running ubuntu 12.04 (openstack essex ). One of these hosts is a "public" host running the compute and network services, the other 2 hosts are on a private vlan and are running compute and network as well as all other components of the openstack installation. All physical hosts have 2 nic's in a bond (for redundancy) configured with an ip in the 10.0.0.0/24 range as a private network. The vm networks we have created are in the 192.168.0.0/16 range and the appropriate vlan tagged networks have been created on the switch. All openstack components are running fine as we can create, run and live migrate instances with no issues. All vm's can contact all physical hosts in the 10.0.0.0/24 range as well as the outside word using a proxy running on the 10.0.0.254 ip. The problem arrises when we try to communicate in between vm's running on different hosts: - name resolution is not working for vm's running on different physical hosts ( I suppose dns should work, no? ) - all packages of communication performed using the ip of the vm directly ( ping, ssh, ...) are arriving on the bridge interface of the physical host running the vm we are tying to reach, but the vm itself is not picking up or responding to the requests... The weird thing is, when we start 2 vm's on the same physical host, name resolution and networking are working fine. When we then live-migrate one of the vm's to a new physical host, the networking will continue to work for a varying amount of time after the live migration has completed! A variable amount of the packages start getting lost until we end up with no communication being possible in between the virtual machines. ( after new dhcp lease? arp table getting flushed?... ) As no errors are appearing in any of the nova logs (all on verbose...) or in the syslog (from the dnsmasq) I really have no clue as to what might be causing this issue... or is it a bug? My feeling is the per physical host vm gateway is not performing as it should and not routing the packages correctly in between physical hosts but I have no idea on how to check this other than capture the packages on the bridge interface and observe the requests not getting answered... Another option is the problem residing with the 2 physical interfaces in the network bond... but wireshark is showing all packages are arriving on the bridge interface where the vm we are trying to reach is residing so this seems unlikely? I have included the nova.conf the ifconfig and the iptables (+nat) of one of the physical hosts in this mail but can provide any other output if this might be helpful. Kind regards, Bram ################### # /etc/nova/nova.conf ################### --dhcpbridge_flagfile=/etc/nova/nova.conf --dhcpbridge=/usr/bin/nova-dhcpbridge --logdir=/var/log/nova --state_path=/var/lib/nova --lock_path=/var/lock/nova ##--force_dhcp_release ##--iscsi_helper=tgtadm --libvirt_use_virtio_for_bridges --connection_type=libvirt --root_helper=sudo nova-rootwrap --verbose --ec2_private_dns_show_ip --auth_strategy=keystone --rabbit_host=10.0.0.100 --nova_url=http://10.0.0.100:8774/v1.1/ --floating_range=999.999.999.0/24 --fixed_range=192.168.0.0/16 --routing_source_ip=10.0.0.103 --sql_connection=postgresql://clouddbadmin:[email protected]/nova --glance_api_servers=10.0.0.100:9292 --image_service=nova.image.glance.GlanceImageService --network_manager=nova.network.manager.VlanManager --vlan_interface=bond0 --public_interface=eth0 --multi-host=true ################### # ifconfig ################### bond0 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:8a inet addr:10.0.0.103 Bcast:10.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::be30:5bff:fedd:c8a/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:1400289 errors:0 dropped:67725 overruns:0 frame:0 TX packets:2414277 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1288957456 (1.2 GB) TX bytes:3217320483 (3.2 GB) br1997 Link encap:Ethernet HWaddr fa:16:3e:50:1f:3f inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::182b:5aff:feda:38f3/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:8 errors:0 dropped:0 overruns:0 frame:0 TX packets:58 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:488 (488.0 B) TX bytes:4940 (4.9 KB) br1998 Link encap:Ethernet HWaddr fa:16:3e:1e:4a:ab inet addr:192.168.0.4 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::5014:d5ff:fe05:93dd/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:4200 errors:0 dropped:15 overruns:0 frame:0 TX packets:5024 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:433834 (433.8 KB) TX bytes:20260632 (20.2 MB) eth0 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:86 inet addr:999.999.999.58 Bcast:999.999.999.255 Mask:255.255.255.0 inet6 addr: fe80::be30:5bff:fedd:c86/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:38664 errors:0 dropped:246 overruns:0 frame:0 TX packets:27311 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:5127536 (5.1 MB) TX bytes:28006322 (28.0 MB) Interrupt:36 Memory:d6000000-d6012800 eth1 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:88 inet addr:157.193.229.69 Bcast:157.193.229.255 Mask:255.255.255.0 inet6 addr: fe80::be30:5bff:fedd:c88/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:21745 errors:0 dropped:0 overruns:0 frame:0 TX packets:10 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2593490 (2.5 MB) TX bytes:1312 (1.3 KB) Interrupt:48 Memory:d8000000-d8012800 eth2 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:8a UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:322566 errors:0 dropped:2 overruns:0 frame:0 TX packets:1132927 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:171375115 (171.3 MB) TX bytes:1563837296 (1.5 GB) Interrupt:32 Memory:da000000-da012800 eth3 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:8c UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:1077723 errors:0 dropped:67478 overruns:0 frame:0 TX packets:1281350 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1117582341 (1.1 GB) TX bytes:1653483187 (1.6 GB) Interrupt:42 Memory:dc000000-dc012800 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:342519 errors:0 dropped:0 overruns:0 frame:0 TX packets:342519 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:3762417359 (3.7 GB) TX bytes:3762417359 (3.7 GB) virbr0 Link encap:Ethernet HWaddr ce:c0:87:1e:39:52 inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) vlan1997 Link encap:Ethernet HWaddr fa:16:3e:50:1f:3f inet6 addr: fe80::f816:3eff:fe50:1f3f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:9 errors:0 dropped:0 overruns:0 frame:0 TX packets:116 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:534 (534.0 B) TX bytes:7756 (7.7 KB) vlan1998 Link encap:Ethernet HWaddr fa:16:3e:1e:4a:ab inet6 addr: fe80::f816:3eff:fe1e:4aab/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:482 errors:0 dropped:0 overruns:0 frame:0 TX packets:497 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:34886 (34.8 KB) TX bytes:50938 (50.9 KB) vnet2 Link encap:Ethernet HWaddr fe:16:3e:6c:af:bc inet6 addr: fe80::fc16:3eff:fe6c:afbc/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:383 errors:0 dropped:0 overruns:0 frame:0 TX packets:280 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:84937 (84.9 KB) TX bytes:39749 (39.7 KB) ################### # sudo iptables -L ################### Chain INPUT (policy ACCEPT) target prot opt source destination nova-compute-INPUT all -- anywhere anywhere nova-network-INPUT all -- anywhere anywhere ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps Chain FORWARD (policy ACCEPT) target prot opt source destination nova-filter-top all -- anywhere anywhere nova-compute-FORWARD all -- anywhere anywhere nova-network-FORWARD all -- anywhere anywhere ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT) target prot opt source destination nova-filter-top all -- anywhere anywhere nova-compute-OUTPUT all -- anywhere anywhere nova-network-OUTPUT all -- anywhere anywhere Chain nova-compute-FORWARD (1 references) target prot opt source destination Chain nova-compute-INPUT (1 references) target prot opt source destination Chain nova-compute-OUTPUT (1 references) target prot opt source destination Chain nova-compute-inst-97 (1 references) target prot opt source destination DROP all -- anywhere anywhere state INVALID ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED nova-compute-provider all -- anywhere anywhere ACCEPT udp -- 192.168.0.4 anywhere udp spt:bootps dpt:bootpc ACCEPT all -- 192.168.0.0/24 anywhere ACCEPT icmp -- anywhere anywhere ACCEPT tcp -- anywhere anywhere tcp dpt:ssh nova-compute-sg-fallback all -- anywhere anywhere Chain nova-compute-local (1 references) target prot opt source destination nova-compute-inst-97 all -- anywhere 192.168.0.40 Chain nova-compute-provider (1 references) target prot opt source destination Chain nova-compute-sg-fallback (1 references) target prot opt source destination DROP all -- anywhere anywhere Chain nova-filter-top (2 references) target prot opt source destination nova-compute-local all -- anywhere anywhere nova-network-local all -- anywhere anywhere Chain nova-network-FORWARD (1 references) target prot opt source destination ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT udp -- anywhere 192.168.1.2 udp dpt:openvpn ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT udp -- anywhere 192.168.0.2 udp dpt:openvpn Chain nova-network-INPUT (1 references) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain Chain nova-network-OUTPUT (1 references) target prot opt source destination Chain nova-network-local (1 references) target prot opt source destination ################### # sudo iptables -L -t nat ################### Chain PREROUTING (policy ACCEPT) target prot opt source destination nova-compute-PREROUTING all -- anywhere anywhere nova-network-PREROUTING all -- anywhere anywhere Chain INPUT (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination nova-compute-OUTPUT all -- anywhere anywhere nova-network-OUTPUT all -- anywhere anywhere Chain POSTROUTING (policy ACCEPT) target prot opt source destination nova-compute-POSTROUTING all -- anywhere anywhere nova-network-POSTROUTING all -- anywhere anywhere nova-postrouting-bottom all -- anywhere anywhere Chain nova-compute-OUTPUT (1 references) target prot opt source destination Chain nova-compute-POSTROUTING (1 references) target prot opt source destination Chain nova-compute-PREROUTING (1 references) target prot opt source destination Chain nova-compute-float-snat (1 references) target prot opt source destination Chain nova-compute-snat (1 references) target prot opt source destination nova-compute-float-snat all -- anywhere anywhere Chain nova-network-OUTPUT (1 references) target prot opt source destination DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 to:192.168.1.2:1194 DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 to:192.168.0.2:1194 Chain nova-network-POSTROUTING (1 references) target prot opt source destination ACCEPT all -- 192.168.0.0/16 999.999.999.58 ACCEPT all -- 192.168.0.0/16 10.128.0.0/24 ACCEPT all -- 192.168.0.0/16 192.168.0.0/16 ! ctstate DNAT Chain nova-network-PREROUTING (1 references) target prot opt source destination DNAT tcp -- anywhere 169.254.169.254 tcp dpt:http to:999.999.999.58:8775 DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 to:192.168.1.2:1194 DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 to:192.168.0.2:1194 Chain nova-network-float-snat (1 references) target prot opt source destination Chain nova-network-snat (1 references) target prot opt source destination nova-network-float-snat all -- anywhere anywhere SNAT all -- 192.168.0.0/16 anywhere to:10.0.0.103 Chain nova-postrouting-bottom (1 references) target prot opt source destination nova-compute-snat all -- anywhere anywhere nova-network-snat all -- anywhere anywhere _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : [email protected] Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
