Thanx Vish, On the name resolution: would you consider this a bug (I can file one if you would like) or a feature? Could this be fixed by changing the /usr/bin/nova-dhcpbridge script to load all mac, hostname, ip combinations for the database instead of just the physical hosts one? Or would this create other issues?
Security rules are setup correctly I guess, as all traffic to and from vm's running on the same host is not experiencing any issues. nova secgroup-list-rules default +-------------+-----------+---------+-----------+--------------+ | IP Protocol | From Port | To Port | IP Range | Source Group | +-------------+-----------+---------+-----------+--------------+ | icmp | -1 | -1 | 0.0.0.0/0 | | | tcp | 22 | 22 | 0.0.0.0/0 | | +-------------+-----------+---------+-----------+--------------+ The bonding might indeed be an issue, we are currently running a adaptive load balancing bond, thus the physical traffic can jump for one physical interface to the other at any time... I will try an disable the bonds and get back to you ass soon as I have done that. Kind regards bram On 1-jun-2012, at 09:04, Vishvananda Ishaya wrote: > Ideas inline. > > Vish > > On May 31, 2012, at 1:41 PM, Bram De Wilde wrote: > >> Hi all, >> >> Can I request some help in resolving a vlan networking issue we are >> encountering in the final stages of our openstack installation? >> >> We have installed a multi host vlan network configuration on 3 hosts all >> running ubuntu 12.04 (openstack essex ). >> >> One of these hosts is a "public" host running the compute and network >> services, the other 2 hosts are on a private vlan and are running compute >> and network as well as all other components of the openstack installation. >> All physical hosts have 2 nic's in a bond (for redundancy) configured with >> an ip in the 10.0.0.0/24 range as a private network. >> >> The vm networks we have created are in the 192.168.0.0/16 range and the >> appropriate vlan tagged networks have been created on the switch. >> >> All openstack components are running fine as we can create, run and live >> migrate instances with no issues. All vm's can contact all physical hosts in >> the 10.0.0.0/24 range as well as the outside word using a proxy running on >> the 10.0.0.254 ip. >> >> The problem arrises when we try to communicate in between vm's running on >> different hosts: >> - name resolution is not working for vm's running on different physical >> hosts ( I suppose dns should work, no? ) > > This is expected in multihost mode. The copy of dnsmasq that runs on each > host only knows about its own vms. You will need to set up a shared dns if > you really need this to work. > >> - all packages of communication performed using the ip of the vm directly ( >> ping, ssh, ...) are arriving on the bridge interface of the physical host >> running the vm we are tying to reach, but the vm itself is not picking up or >> responding to the requests... > > Have you set up security group rules to allow the traffic? That is the only > reason I can think that packets wouldn't be getting into the vnet if it is > showing up on the bridge. There is also a possiblity that bonding + bridging > + vlans has some sort of an issue. > >> >> The weird thing is, when we start 2 vm's on the same physical host, name >> resolution and networking are working fine. When we then live-migrate one of >> the vm's to a new physical host, the networking will continue to work for a >> varying amount of time after the live migration has completed! A variable >> amount of the packages start getting lost until we end up with no >> communication being possible in between the virtual machines. ( after new >> dhcp lease? arp table getting flushed?... ) >> >> As no errors are appearing in any of the nova logs (all on verbose...) or in >> the syslog (from the dnsmasq) I really have no clue as to what might be >> causing this issue... or is it a bug? >> >> My feeling is the per physical host vm gateway is not performing as it >> should and not routing the packages correctly in between physical hosts but >> I have no idea on how to check this other than capture the packages on the >> bridge interface and observe the requests not getting answered... >> Another option is the problem residing with the 2 physical interfaces in the >> network bond... but wireshark is showing all packages are arriving on the >> bridge interface where the vm we are trying to reach is residing so this >> seems unlikely? >> >> I have included the nova.conf the ifconfig and the iptables (+nat) of one of >> the physical hosts in this mail but can provide any other output if this >> might be helpful. >> >> Kind regards, >> Bram >> >> ################### >> # /etc/nova/nova.conf >> ################### >> >> --dhcpbridge_flagfile=/etc/nova/nova.conf >> --dhcpbridge=/usr/bin/nova-dhcpbridge >> --logdir=/var/log/nova >> --state_path=/var/lib/nova >> --lock_path=/var/lock/nova >> ##--force_dhcp_release >> ##--iscsi_helper=tgtadm >> --libvirt_use_virtio_for_bridges >> --connection_type=libvirt >> --root_helper=sudo nova-rootwrap >> --verbose >> --ec2_private_dns_show_ip >> --auth_strategy=keystone >> --rabbit_host=10.0.0.100 >> --nova_url=http://10.0.0.100:8774/v1.1/ >> --floating_range=999.999.999.0/24 >> --fixed_range=192.168.0.0/16 >> --routing_source_ip=10.0.0.103 >> --sql_connection=postgresql://clouddbadmin:[email protected]/nova >> --glance_api_servers=10.0.0.100:9292 >> --image_service=nova.image.glance.GlanceImageService >> --network_manager=nova.network.manager.VlanManager >> --vlan_interface=bond0 >> --public_interface=eth0 >> --multi-host=true >> >> ################### >> # ifconfig >> ################### >> >> bond0 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:8a >> inet addr:10.0.0.103 Bcast:10.0.0.255 Mask:255.255.255.0 >> inet6 addr: fe80::be30:5bff:fedd:c8a/64 Scope:Link >> UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 >> RX packets:1400289 errors:0 dropped:67725 overruns:0 frame:0 >> TX packets:2414277 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:0 >> RX bytes:1288957456 (1.2 GB) TX bytes:3217320483 (3.2 GB) >> >> br1997 Link encap:Ethernet HWaddr fa:16:3e:50:1f:3f >> inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 >> inet6 addr: fe80::182b:5aff:feda:38f3/64 Scope:Link >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:8 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:58 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:0 >> RX bytes:488 (488.0 B) TX bytes:4940 (4.9 KB) >> >> br1998 Link encap:Ethernet HWaddr fa:16:3e:1e:4a:ab >> inet addr:192.168.0.4 Bcast:192.168.0.255 Mask:255.255.255.0 >> inet6 addr: fe80::5014:d5ff:fe05:93dd/64 Scope:Link >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:4200 errors:0 dropped:15 overruns:0 frame:0 >> TX packets:5024 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:0 >> RX bytes:433834 (433.8 KB) TX bytes:20260632 (20.2 MB) >> >> eth0 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:86 >> inet addr:999.999.999.58 Bcast:999.999.999.255 Mask:255.255.255.0 >> inet6 addr: fe80::be30:5bff:fedd:c86/64 Scope:Link >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:38664 errors:0 dropped:246 overruns:0 frame:0 >> TX packets:27311 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:1000 >> RX bytes:5127536 (5.1 MB) TX bytes:28006322 (28.0 MB) >> Interrupt:36 Memory:d6000000-d6012800 >> >> eth1 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:88 >> inet addr:157.193.229.69 Bcast:157.193.229.255 Mask:255.255.255.0 >> inet6 addr: fe80::be30:5bff:fedd:c88/64 Scope:Link >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:21745 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:10 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:1000 >> RX bytes:2593490 (2.5 MB) TX bytes:1312 (1.3 KB) >> Interrupt:48 Memory:d8000000-d8012800 >> >> eth2 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:8a >> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 >> RX packets:322566 errors:0 dropped:2 overruns:0 frame:0 >> TX packets:1132927 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:1000 >> RX bytes:171375115 (171.3 MB) TX bytes:1563837296 (1.5 GB) >> Interrupt:32 Memory:da000000-da012800 >> >> eth3 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:8c >> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 >> RX packets:1077723 errors:0 dropped:67478 overruns:0 frame:0 >> TX packets:1281350 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:1000 >> RX bytes:1117582341 (1.1 GB) TX bytes:1653483187 (1.6 GB) >> Interrupt:42 Memory:dc000000-dc012800 >> >> lo Link encap:Local Loopback >> inet addr:127.0.0.1 Mask:255.0.0.0 >> inet6 addr: ::1/128 Scope:Host >> UP LOOPBACK RUNNING MTU:16436 Metric:1 >> RX packets:342519 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:342519 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:0 >> RX bytes:3762417359 (3.7 GB) TX bytes:3762417359 (3.7 GB) >> >> virbr0 Link encap:Ethernet HWaddr ce:c0:87:1e:39:52 >> inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 >> UP BROADCAST MULTICAST MTU:1500 Metric:1 >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:0 >> RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) >> >> vlan1997 Link encap:Ethernet HWaddr fa:16:3e:50:1f:3f >> inet6 addr: fe80::f816:3eff:fe50:1f3f/64 Scope:Link >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:9 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:116 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:0 >> RX bytes:534 (534.0 B) TX bytes:7756 (7.7 KB) >> >> vlan1998 Link encap:Ethernet HWaddr fa:16:3e:1e:4a:ab >> inet6 addr: fe80::f816:3eff:fe1e:4aab/64 Scope:Link >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:482 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:497 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:0 >> RX bytes:34886 (34.8 KB) TX bytes:50938 (50.9 KB) >> >> vnet2 Link encap:Ethernet HWaddr fe:16:3e:6c:af:bc >> inet6 addr: fe80::fc16:3eff:fe6c:afbc/64 Scope:Link >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:383 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:280 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:500 >> RX bytes:84937 (84.9 KB) TX bytes:39749 (39.7 KB) >> >> >> ################### >> # sudo iptables -L >> ################### >> >> Chain INPUT (policy ACCEPT) >> target prot opt source destination >> nova-compute-INPUT all -- anywhere anywhere >> nova-network-INPUT all -- anywhere anywhere >> ACCEPT udp -- anywhere anywhere udp dpt:domain >> ACCEPT tcp -- anywhere anywhere tcp dpt:domain >> ACCEPT udp -- anywhere anywhere udp dpt:bootps >> ACCEPT tcp -- anywhere anywhere tcp dpt:bootps >> >> Chain FORWARD (policy ACCEPT) >> target prot opt source destination >> nova-filter-top all -- anywhere anywhere >> nova-compute-FORWARD all -- anywhere anywhere >> nova-network-FORWARD all -- anywhere anywhere >> ACCEPT all -- anywhere 192.168.122.0/24 state >> RELATED,ESTABLISHED >> ACCEPT all -- 192.168.122.0/24 anywhere >> ACCEPT all -- anywhere anywhere >> REJECT all -- anywhere anywhere reject-with >> icmp-port-unreachable >> REJECT all -- anywhere anywhere reject-with >> icmp-port-unreachable >> >> Chain OUTPUT (policy ACCEPT) >> target prot opt source destination >> nova-filter-top all -- anywhere anywhere >> nova-compute-OUTPUT all -- anywhere anywhere >> nova-network-OUTPUT all -- anywhere anywhere >> >> Chain nova-compute-FORWARD (1 references) >> target prot opt source destination >> >> Chain nova-compute-INPUT (1 references) >> target prot opt source destination >> >> Chain nova-compute-OUTPUT (1 references) >> target prot opt source destination >> >> Chain nova-compute-inst-97 (1 references) >> target prot opt source destination >> DROP all -- anywhere anywhere state INVALID >> ACCEPT all -- anywhere anywhere state >> RELATED,ESTABLISHED >> nova-compute-provider all -- anywhere anywhere >> ACCEPT udp -- 192.168.0.4 anywhere udp spt:bootps >> dpt:bootpc >> ACCEPT all -- 192.168.0.0/24 anywhere >> ACCEPT icmp -- anywhere anywhere >> ACCEPT tcp -- anywhere anywhere tcp dpt:ssh >> nova-compute-sg-fallback all -- anywhere anywhere >> >> Chain nova-compute-local (1 references) >> target prot opt source destination >> nova-compute-inst-97 all -- anywhere 192.168.0.40 >> >> Chain nova-compute-provider (1 references) >> target prot opt source destination >> >> Chain nova-compute-sg-fallback (1 references) >> target prot opt source destination >> DROP all -- anywhere anywhere >> >> Chain nova-filter-top (2 references) >> target prot opt source destination >> nova-compute-local all -- anywhere anywhere >> nova-network-local all -- anywhere anywhere >> >> Chain nova-network-FORWARD (1 references) >> target prot opt source destination >> ACCEPT all -- anywhere anywhere >> ACCEPT all -- anywhere anywhere >> ACCEPT udp -- anywhere 192.168.1.2 udp dpt:openvpn >> ACCEPT all -- anywhere anywhere >> ACCEPT all -- anywhere anywhere >> ACCEPT udp -- anywhere 192.168.0.2 udp dpt:openvpn >> >> Chain nova-network-INPUT (1 references) >> target prot opt source destination >> ACCEPT udp -- anywhere anywhere udp dpt:bootps >> ACCEPT tcp -- anywhere anywhere tcp dpt:bootps >> ACCEPT udp -- anywhere anywhere udp dpt:domain >> ACCEPT tcp -- anywhere anywhere tcp dpt:domain >> ACCEPT udp -- anywhere anywhere udp dpt:bootps >> ACCEPT tcp -- anywhere anywhere tcp dpt:bootps >> ACCEPT udp -- anywhere anywhere udp dpt:domain >> ACCEPT tcp -- anywhere anywhere tcp dpt:domain >> >> Chain nova-network-OUTPUT (1 references) >> target prot opt source destination >> >> Chain nova-network-local (1 references) >> target prot opt source destination >> >> ################### >> # sudo iptables -L -t nat >> ################### >> >> Chain PREROUTING (policy ACCEPT) >> target prot opt source destination >> nova-compute-PREROUTING all -- anywhere anywhere >> nova-network-PREROUTING all -- anywhere anywhere >> >> Chain INPUT (policy ACCEPT) >> target prot opt source destination >> >> Chain OUTPUT (policy ACCEPT) >> target prot opt source destination >> nova-compute-OUTPUT all -- anywhere anywhere >> nova-network-OUTPUT all -- anywhere anywhere >> >> Chain POSTROUTING (policy ACCEPT) >> target prot opt source destination >> nova-compute-POSTROUTING all -- anywhere anywhere >> nova-network-POSTROUTING all -- anywhere anywhere >> nova-postrouting-bottom all -- anywhere anywhere >> >> Chain nova-compute-OUTPUT (1 references) >> target prot opt source destination >> >> Chain nova-compute-POSTROUTING (1 references) >> target prot opt source destination >> >> Chain nova-compute-PREROUTING (1 references) >> target prot opt source destination >> >> Chain nova-compute-float-snat (1 references) >> target prot opt source destination >> >> Chain nova-compute-snat (1 references) >> target prot opt source destination >> nova-compute-float-snat all -- anywhere anywhere >> >> Chain nova-network-OUTPUT (1 references) >> target prot opt source destination >> DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 >> to:192.168.1.2:1194 >> DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 >> to:192.168.0.2:1194 >> >> Chain nova-network-POSTROUTING (1 references) >> target prot opt source destination >> ACCEPT all -- 192.168.0.0/16 999.999.999.58 >> ACCEPT all -- 192.168.0.0/16 10.128.0.0/24 >> ACCEPT all -- 192.168.0.0/16 192.168.0.0/16 ! ctstate DNAT >> >> Chain nova-network-PREROUTING (1 references) >> target prot opt source destination >> DNAT tcp -- anywhere 169.254.169.254 tcp dpt:http >> to:999.999.999.58:8775 >> DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 >> to:192.168.1.2:1194 >> DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 >> to:192.168.0.2:1194 >> >> Chain nova-network-float-snat (1 references) >> target prot opt source destination >> >> Chain nova-network-snat (1 references) >> target prot opt source destination >> nova-network-float-snat all -- anywhere anywhere >> SNAT all -- 192.168.0.0/16 anywhere to:10.0.0.103 >> >> Chain nova-postrouting-bottom (1 references) >> target prot opt source destination >> nova-compute-snat all -- anywhere anywhere >> nova-network-snat all -- anywhere anywhere >> _______________________________________________ >> Mailing list: https://launchpad.net/~openstack >> Post to : [email protected] >> Unsubscribe : https://launchpad.net/~openstack >> More help : https://help.launchpad.net/ListHelp > _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : [email protected] Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
