Repro: OVS-DPDK starting up seems fine initializing my non-blacklisted card DPDK_OPTS are '--dpdk -c 0x6 -n 4 --pci-blacklist 0000:00:03.0 -m 2048' Allowing PMDs on two CPUs consuming 2G of huge pages
Before adding Ports config is done with ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=2 ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x6 Port is added like: ovs-vsctl add-port ovsdpdkbr0 dpdk0 -- set Interface dpdk0 type=dpdk Two PMDs are seen dpif_netdev|INFO|Created 2 pmd threads on numa node 0 bridge|INFO|bridge ovsdpdkbr0: added interface dpdk0 on port 1 dpif_netdev(pmd12)|INFO|Core 2 processing port 'dpdk0' dpif_netdev(pmd13)|INFO|Core 1 processing port 'dpdk0' PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 26062 root 10 -10 2816828 105612 16328 R 99.9 1.7 1:09.46 pmd12 26061 root 10 -10 2816828 105612 16328 R 99.9 1.7 1:09.42 pmd13 Now I should be in a similar state as you are. I know the assumption so far was that the reboot (which resets the #queues on the device) might be involved. But I first wanted to try what changing queues without reboot would do. Even setting it down from 4 to 3 (remember I only use 2 actively) goes into the block. ethtool -L eth1 combined 3 => I can see a hang of the ethtool program Good thing: at least it seems we can remove the reboot out of our thinking. Just changing #queues with (?OVS?)DPDK attached seems to be very unhappy. Please note the discussion that could be related: http://dpdk.org/ml/archives/dev/2016-April/037443.html For confirmation @Thiago - does for you also the ethtool program hang, or the full guest? Next steps: - gather debug data on hanging ethtool - check what happens if we ethtool after we stopped OVS-DPDK (match upstream discussion) - check what happens if we have testpmd enabled instead of openvswitch-dpdk -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1570195 Title: Network tools like "ethtool" or "ip" freezes when DPDK Apps are running with VirtIO Status in dpdk package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: Guys, I'm facing an issue here with both "ethtool" and "ip", while trying to manage black-listed by DPDK PCI VirtIO devices. You'll need an Ubuntu Xenial KVM guest, with 4 VirtIO vNIC cards, to run those tests PCI device example from inside a Xenial guest: --- # lspci | grep Ethernet 00:03.0 Ethernet controller: Red Hat, Inc Virtio network device 00:04.0 Ethernet controller: Red Hat, Inc Virtio network device 00:05.0 Ethernet controller: Red Hat, Inc Virtio network device 00:06.0 Ethernet controller: Red Hat, Inc Virtio network device --- Where "ens3" is the first / default interface, attached to Libvirt's "default" network. The "ens4" is reserved for "ethtool / ip" tests (attached to another Libvirt's network without IPs or DHCP), "ens5" will be "dpdk0" and "ens6" "dpdk1"... --- *** How it works? 1- For example, try to enable multi-queue on DPDK's devices, boot your Xenial guest, and run: ethtool -L ens5 combined 4 ethtool -L ens6 combined 4 2- Install openvswitch-switch-dpdk configure DPDK and OVS and fire it up. https://help.ubuntu.com/16.04/serverguide/DPDK.html service openvswitch-switch stop service dpdk stop OVS DPDK Options (/etc/default/openvswitch-switch): -- DPDK_OPTS='--dpdk -c 0x1 -n 4 --socket-mem 1024 --pci-blacklist 0000:00:03.0,0000:00:04.0' -- service dpdk start service openvswitch-switch start - Enable multi-queue on OVS+DPDK inside of the VM: ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=4 ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0xff00 * Multi-queue apparently works! ovs-vswitchd consumes more that 100% of CPU, meaning that it multi-queue is there... *** Where it fails? 1- Reboot the VM and try to run ethtool again (or go straight to 2 below): ethtool -L ens5 combined 4 2- Try to fire up ens4: ip link set dev ens4 up # FAIL! Both commands hangs, consuming 100% of guest's CPU... So, it looks like a Linux fault, because it is "allowing" the DPDK VirtIO App (a user land App), to interfere with kernel devices in a strange way... Best, Thiago ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-18-generic 4.4.0-18.34 ProcVersionSignature: Ubuntu 4.4.0-18.34-generic 4.4.6 Uname: Linux 4.4.0-18-generic x86_64 AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Apr 14 00:35 seq crw-rw---- 1 root audio 116, 33 Apr 14 00:35 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: [Errno 2] No such file or directory: 'fuser' CRDA: N/A Date: Thu Apr 14 01:27:27 2016 HibernationDevice: RESUME=UUID=833e999c-e066-433c-b8a2-4324bb8d56de InstallationDate: Installed on 2016-04-07 (7 days ago) InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Beta amd64 (20160406) IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub MachineType: QEMU Standard PC (i440FX + PIIX, 1996) PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-18-generic root=UUID=9911604e-353b-491f-a0a9-804724350592 ro RelatedPackageVersions: linux-restricted-modules-4.4.0-18-generic N/A linux-backports-modules-4.4.0-18-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 04/01/2014 dmi.bios.vendor: SeaBIOS dmi.bios.version: Ubuntu-1.8.2-1ubuntu1 dmi.chassis.type: 1 dmi.chassis.vendor: QEMU dmi.chassis.version: pc-i440fx-wily dmi.modalias: dmi:bvnSeaBIOS:bvrUbuntu-1.8.2-1ubuntu1:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-wily:cvnQEMU:ct1:cvrpc-i440fx-wily: dmi.product.name: Standard PC (i440FX + PIIX, 1996) dmi.product.version: pc-i440fx-wily dmi.sys.vendor: QEMU To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1570195/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
