Ideas inline. Vish
On May 31, 2012, at 1:41 PM, Bram De Wilde wrote: > Hi all, > > Can I request some help in resolving a vlan networking issue we are > encountering in the final stages of our openstack installation? > > We have installed a multi host vlan network configuration on 3 hosts all > running ubuntu 12.04 (openstack essex ). > > One of these hosts is a "public" host running the compute and network > services, the other 2 hosts are on a private vlan and are running compute and > network as well as all other components of the openstack installation. > All physical hosts have 2 nic's in a bond (for redundancy) configured with an > ip in the 10.0.0.0/24 range as a private network. > > The vm networks we have created are in the 192.168.0.0/16 range and the > appropriate vlan tagged networks have been created on the switch. > > All openstack components are running fine as we can create, run and live > migrate instances with no issues. All vm's can contact all physical hosts in > the 10.0.0.0/24 range as well as the outside word using a proxy running on > the 10.0.0.254 ip. > > The problem arrises when we try to communicate in between vm's running on > different hosts: > - name resolution is not working for vm's running on different physical hosts > ( I suppose dns should work, no? ) This is expected in multihost mode. The copy of dnsmasq that runs on each host only knows about its own vms. You will need to set up a shared dns if you really need this to work. > - all packages of communication performed using the ip of the vm directly ( > ping, ssh, ...) are arriving on the bridge interface of the physical host > running the vm we are tying to reach, but the vm itself is not picking up or > responding to the requests... Have you set up security group rules to allow the traffic? That is the only reason I can think that packets wouldn't be getting into the vnet if it is showing up on the bridge. There is also a possiblity that bonding + bridging + vlans has some sort of an issue. > > The weird thing is, when we start 2 vm's on the same physical host, name > resolution and networking are working fine. When we then live-migrate one of > the vm's to a new physical host, the networking will continue to work for a > varying amount of time after the live migration has completed! A variable > amount of the packages start getting lost until we end up with no > communication being possible in between the virtual machines. ( after new > dhcp lease? arp table getting flushed?... ) > > As no errors are appearing in any of the nova logs (all on verbose...) or in > the syslog (from the dnsmasq) I really have no clue as to what might be > causing this issue... or is it a bug? > > My feeling is the per physical host vm gateway is not performing as it should > and not routing the packages correctly in between physical hosts but I have > no idea on how to check this other than capture the packages on the bridge > interface and observe the requests not getting answered... > Another option is the problem residing with the 2 physical interfaces in the > network bond... but wireshark is showing all packages are arriving on the > bridge interface where the vm we are trying to reach is residing so this > seems unlikely? > > I have included the nova.conf the ifconfig and the iptables (+nat) of one of > the physical hosts in this mail but can provide any other output if this > might be helpful. > > Kind regards, > Bram > > ################### > # /etc/nova/nova.conf > ################### > > --dhcpbridge_flagfile=/etc/nova/nova.conf > --dhcpbridge=/usr/bin/nova-dhcpbridge > --logdir=/var/log/nova > --state_path=/var/lib/nova > --lock_path=/var/lock/nova > ##--force_dhcp_release > ##--iscsi_helper=tgtadm > --libvirt_use_virtio_for_bridges > --connection_type=libvirt > --root_helper=sudo nova-rootwrap > --verbose > --ec2_private_dns_show_ip > --auth_strategy=keystone > --rabbit_host=10.0.0.100 > --nova_url=http://10.0.0.100:8774/v1.1/ > --floating_range=999.999.999.0/24 > --fixed_range=192.168.0.0/16 > --routing_source_ip=10.0.0.103 > --sql_connection=postgresql://clouddbadmin:[email protected]/nova > --glance_api_servers=10.0.0.100:9292 > --image_service=nova.image.glance.GlanceImageService > --network_manager=nova.network.manager.VlanManager > --vlan_interface=bond0 > --public_interface=eth0 > --multi-host=true > > ################### > # ifconfig > ################### > > bond0 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:8a > inet addr:10.0.0.103 Bcast:10.0.0.255 Mask:255.255.255.0 > inet6 addr: fe80::be30:5bff:fedd:c8a/64 Scope:Link > UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 > RX packets:1400289 errors:0 dropped:67725 overruns:0 frame:0 > TX packets:2414277 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:1288957456 (1.2 GB) TX bytes:3217320483 (3.2 GB) > > br1997 Link encap:Ethernet HWaddr fa:16:3e:50:1f:3f > inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 > inet6 addr: fe80::182b:5aff:feda:38f3/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:8 errors:0 dropped:0 overruns:0 frame:0 > TX packets:58 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:488 (488.0 B) TX bytes:4940 (4.9 KB) > > br1998 Link encap:Ethernet HWaddr fa:16:3e:1e:4a:ab > inet addr:192.168.0.4 Bcast:192.168.0.255 Mask:255.255.255.0 > inet6 addr: fe80::5014:d5ff:fe05:93dd/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:4200 errors:0 dropped:15 overruns:0 frame:0 > TX packets:5024 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:433834 (433.8 KB) TX bytes:20260632 (20.2 MB) > > eth0 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:86 > inet addr:999.999.999.58 Bcast:999.999.999.255 Mask:255.255.255.0 > inet6 addr: fe80::be30:5bff:fedd:c86/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:38664 errors:0 dropped:246 overruns:0 frame:0 > TX packets:27311 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:5127536 (5.1 MB) TX bytes:28006322 (28.0 MB) > Interrupt:36 Memory:d6000000-d6012800 > > eth1 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:88 > inet addr:157.193.229.69 Bcast:157.193.229.255 Mask:255.255.255.0 > inet6 addr: fe80::be30:5bff:fedd:c88/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:21745 errors:0 dropped:0 overruns:0 frame:0 > TX packets:10 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:2593490 (2.5 MB) TX bytes:1312 (1.3 KB) > Interrupt:48 Memory:d8000000-d8012800 > > eth2 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:8a > UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 > RX packets:322566 errors:0 dropped:2 overruns:0 frame:0 > TX packets:1132927 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:171375115 (171.3 MB) TX bytes:1563837296 (1.5 GB) > Interrupt:32 Memory:da000000-da012800 > > eth3 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:8c > UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 > RX packets:1077723 errors:0 dropped:67478 overruns:0 frame:0 > TX packets:1281350 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:1117582341 (1.1 GB) TX bytes:1653483187 (1.6 GB) > Interrupt:42 Memory:dc000000-dc012800 > > lo Link encap:Local Loopback > inet addr:127.0.0.1 Mask:255.0.0.0 > inet6 addr: ::1/128 Scope:Host > UP LOOPBACK RUNNING MTU:16436 Metric:1 > RX packets:342519 errors:0 dropped:0 overruns:0 frame:0 > TX packets:342519 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:3762417359 (3.7 GB) TX bytes:3762417359 (3.7 GB) > > virbr0 Link encap:Ethernet HWaddr ce:c0:87:1e:39:52 > inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 > UP BROADCAST MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > vlan1997 Link encap:Ethernet HWaddr fa:16:3e:50:1f:3f > inet6 addr: fe80::f816:3eff:fe50:1f3f/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:9 errors:0 dropped:0 overruns:0 frame:0 > TX packets:116 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:534 (534.0 B) TX bytes:7756 (7.7 KB) > > vlan1998 Link encap:Ethernet HWaddr fa:16:3e:1e:4a:ab > inet6 addr: fe80::f816:3eff:fe1e:4aab/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:482 errors:0 dropped:0 overruns:0 frame:0 > TX packets:497 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:34886 (34.8 KB) TX bytes:50938 (50.9 KB) > > vnet2 Link encap:Ethernet HWaddr fe:16:3e:6c:af:bc > inet6 addr: fe80::fc16:3eff:fe6c:afbc/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:383 errors:0 dropped:0 overruns:0 frame:0 > TX packets:280 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:500 > RX bytes:84937 (84.9 KB) TX bytes:39749 (39.7 KB) > > > ################### > # sudo iptables -L > ################### > > Chain INPUT (policy ACCEPT) > target prot opt source destination > nova-compute-INPUT all -- anywhere anywhere > nova-network-INPUT all -- anywhere anywhere > ACCEPT udp -- anywhere anywhere udp dpt:domain > ACCEPT tcp -- anywhere anywhere tcp dpt:domain > ACCEPT udp -- anywhere anywhere udp dpt:bootps > ACCEPT tcp -- anywhere anywhere tcp dpt:bootps > > Chain FORWARD (policy ACCEPT) > target prot opt source destination > nova-filter-top all -- anywhere anywhere > nova-compute-FORWARD all -- anywhere anywhere > nova-network-FORWARD all -- anywhere anywhere > ACCEPT all -- anywhere 192.168.122.0/24 state > RELATED,ESTABLISHED > ACCEPT all -- 192.168.122.0/24 anywhere > ACCEPT all -- anywhere anywhere > REJECT all -- anywhere anywhere reject-with > icmp-port-unreachable > REJECT all -- anywhere anywhere reject-with > icmp-port-unreachable > > Chain OUTPUT (policy ACCEPT) > target prot opt source destination > nova-filter-top all -- anywhere anywhere > nova-compute-OUTPUT all -- anywhere anywhere > nova-network-OUTPUT all -- anywhere anywhere > > Chain nova-compute-FORWARD (1 references) > target prot opt source destination > > Chain nova-compute-INPUT (1 references) > target prot opt source destination > > Chain nova-compute-OUTPUT (1 references) > target prot opt source destination > > Chain nova-compute-inst-97 (1 references) > target prot opt source destination > DROP all -- anywhere anywhere state INVALID > ACCEPT all -- anywhere anywhere state > RELATED,ESTABLISHED > nova-compute-provider all -- anywhere anywhere > ACCEPT udp -- 192.168.0.4 anywhere udp spt:bootps > dpt:bootpc > ACCEPT all -- 192.168.0.0/24 anywhere > ACCEPT icmp -- anywhere anywhere > ACCEPT tcp -- anywhere anywhere tcp dpt:ssh > nova-compute-sg-fallback all -- anywhere anywhere > > Chain nova-compute-local (1 references) > target prot opt source destination > nova-compute-inst-97 all -- anywhere 192.168.0.40 > > Chain nova-compute-provider (1 references) > target prot opt source destination > > Chain nova-compute-sg-fallback (1 references) > target prot opt source destination > DROP all -- anywhere anywhere > > Chain nova-filter-top (2 references) > target prot opt source destination > nova-compute-local all -- anywhere anywhere > nova-network-local all -- anywhere anywhere > > Chain nova-network-FORWARD (1 references) > target prot opt source destination > ACCEPT all -- anywhere anywhere > ACCEPT all -- anywhere anywhere > ACCEPT udp -- anywhere 192.168.1.2 udp dpt:openvpn > ACCEPT all -- anywhere anywhere > ACCEPT all -- anywhere anywhere > ACCEPT udp -- anywhere 192.168.0.2 udp dpt:openvpn > > Chain nova-network-INPUT (1 references) > target prot opt source destination > ACCEPT udp -- anywhere anywhere udp dpt:bootps > ACCEPT tcp -- anywhere anywhere tcp dpt:bootps > ACCEPT udp -- anywhere anywhere udp dpt:domain > ACCEPT tcp -- anywhere anywhere tcp dpt:domain > ACCEPT udp -- anywhere anywhere udp dpt:bootps > ACCEPT tcp -- anywhere anywhere tcp dpt:bootps > ACCEPT udp -- anywhere anywhere udp dpt:domain > ACCEPT tcp -- anywhere anywhere tcp dpt:domain > > Chain nova-network-OUTPUT (1 references) > target prot opt source destination > > Chain nova-network-local (1 references) > target prot opt source destination > > ################### > # sudo iptables -L -t nat > ################### > > Chain PREROUTING (policy ACCEPT) > target prot opt source destination > nova-compute-PREROUTING all -- anywhere anywhere > nova-network-PREROUTING all -- anywhere anywhere > > Chain INPUT (policy ACCEPT) > target prot opt source destination > > Chain OUTPUT (policy ACCEPT) > target prot opt source destination > nova-compute-OUTPUT all -- anywhere anywhere > nova-network-OUTPUT all -- anywhere anywhere > > Chain POSTROUTING (policy ACCEPT) > target prot opt source destination > nova-compute-POSTROUTING all -- anywhere anywhere > nova-network-POSTROUTING all -- anywhere anywhere > nova-postrouting-bottom all -- anywhere anywhere > > Chain nova-compute-OUTPUT (1 references) > target prot opt source destination > > Chain nova-compute-POSTROUTING (1 references) > target prot opt source destination > > Chain nova-compute-PREROUTING (1 references) > target prot opt source destination > > Chain nova-compute-float-snat (1 references) > target prot opt source destination > > Chain nova-compute-snat (1 references) > target prot opt source destination > nova-compute-float-snat all -- anywhere anywhere > > Chain nova-network-OUTPUT (1 references) > target prot opt source destination > DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 > to:192.168.1.2:1194 > DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 > to:192.168.0.2:1194 > > Chain nova-network-POSTROUTING (1 references) > target prot opt source destination > ACCEPT all -- 192.168.0.0/16 999.999.999.58 > ACCEPT all -- 192.168.0.0/16 10.128.0.0/24 > ACCEPT all -- 192.168.0.0/16 192.168.0.0/16 ! ctstate DNAT > > Chain nova-network-PREROUTING (1 references) > target prot opt source destination > DNAT tcp -- anywhere 169.254.169.254 tcp dpt:http > to:999.999.999.58:8775 > DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 > to:192.168.1.2:1194 > DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 > to:192.168.0.2:1194 > > Chain nova-network-float-snat (1 references) > target prot opt source destination > > Chain nova-network-snat (1 references) > target prot opt source destination > nova-network-float-snat all -- anywhere anywhere > SNAT all -- 192.168.0.0/16 anywhere to:10.0.0.103 > > Chain nova-postrouting-bottom (1 references) > target prot opt source destination > nova-compute-snat all -- anywhere anywhere > nova-network-snat all -- anywhere anywhere > _______________________________________________ > Mailing list: https://launchpad.net/~openstack > Post to : [email protected] > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : [email protected] Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
