[Bug 1841669] [NEW] SR-IOV interface goes down when testing network performance
Public bug reported:
Hello,
We are testing network performance with SR-IOV and bridging on ubuntu
and security onion in Rhev Hypervisor which is connected to a SuperMicro
box running T-rex traffic generator on CentOS which is connected to a
gigmon.
The ubuntu 16.04 vm is running on a RHVH host node and is running SR-
IOV. We have ensured that SR_IOV is enabled in the BIOS. We are finding
that on security onion that the packets are dropping after a few
seconds. This is because the interface is going down after a certain
point. We found that packets are getting dropped as soon as the
interface shows as down in ip -s link. We are running promiscuous mode
and it is required for our use case of measuring network traffic. Once
this process was completed, Kernel 5.3.0-rc3 was built using the 3.2.3
iavf SR-IOV driver. Note that the i40evf driver was renamed to iavf,
which is why the name discrepancy exists. After updating the Kernel we
stopped seeing this issue.
Non-working default security onion kernel information
root@ubuntu:~# uname -r
3.10.0-957.21.3.el7.x86_64
root@ubuntu:~# ethtool -i ens7
driver: iavf
version: 3.0.1-k
firmware-version: N/A
expansion-rom-version:
bus-info: :00:07.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
Working modified security onion kernel information
root@ubuntu:~# uname -r
5.3.0-rc3-custom
root@ubuntu:~# ethtool -i ens7
driver: i40evf
version: 3.2.3-k
firmware-version: N/A
expansion-rom-version:
bus-info: :00:07.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
We were able to perform testing using the 5.3.0 kernel, but long term a better
solution is needed as this would require supporting a modified Linux kernel
with updates, testing, etc.
>From the hypervisor in /var/log/messages:
Aug 27 09:05:00 localhost libvirtd: 2019-08-27 13:05:00.522+: 10748: error
: qemuDomainAgentAvailable:9133 : Guest agent is not responding:
QEMU guest agent is not connected
Aug 27 09:05:07 localhost goferd: [ERROR][worker-0]
gofer.messaging.adapter.conn ect:33 - connect:
proton+amqps://subscription.rhsm.redhat.com:5647, failed: Conn
ection amqps://subscription.rhsm.redhat.com:5647disconnected: Condition('amqp:r
esource-limit-exceeded', 'local-idle-timeout expired')
Aug 27 09:05:07 localhost goferd: [INFO][worker-0]
gofer.messaging.adapter.conne ct:35 - retry in 106 seconds
Aug 27 09:05:22 localhost kernel: i40e :b5:00.3: Unable to send the message
to VF 0 aq_err 12
Aug 27 09:05:22 localhost kernel: i40e :b5:00.3: Unable to send the message
to VF 0 aq_err 12
>From the VM, journalctl -xe gives:
-- The start-up result is done.
Aug 27 14:35:02 ubuntu kernel: i40evf :00:04.0: PF reset warning received
Aug 27 14:35:02 ubuntu kernel: i40evf :00:04.0: Scheduling reset task
Aug 27 14:35:03 ubuntu kernel: i40evf :00:04.0: Hardware reset detected
Aug 27 14:35:03 ubuntu kernel: i40evf :00:04.0: Hardware reset detected
Aug 27 14:35:03 ubuntu kernel: i40evf :00:04.0: Hardware reset detected
Aug 27 14:35:08 ubuntu systemd-logind[1251]: Removed session 16.
-- Subject: Session 16 has been terminated
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat
** Affects: ubuntu
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1841669
Title:
SR-IOV interface goes down when testing network performance
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+bug/1841669/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1841669] Re: SR-IOV interface goes down when testing network performance
** Package changed: ubuntu => kernel-package (Ubuntu)
** Description changed:
Hello,
- We are testing network performance with SR-IOV and bridging on ubuntu
- and security onion in Rhev Hypervisor which is connected to a SuperMicro
- box running T-rex traffic generator on CentOS which is connected to a
- gigmon.
+ We are testing network performance with SR-IOV on ubuntu and security
+ onion in Rhev Hypervisor. We are using a SuperMicro box running T-rex
+ traffic generator on CentOS which is connected to a gigamon to push
+ traffic to the Ubuntu/Security Onion VMs.
The ubuntu 16.04 vm is running on a RHVH host node and is running SR-
- IOV. We have ensured that SR_IOV is enabled in the BIOS. We are finding
- that on security onion that the packets are dropping after a few
- seconds. This is because the interface is going down after a certain
- point. We found that packets are getting dropped as soon as the
- interface shows as down in ip -s link. We are running promiscuous mode
- and it is required for our use case of measuring network traffic. Once
- this process was completed, Kernel 5.3.0-rc3 was built using the 3.2.3
- iavf SR-IOV driver. Note that the i40evf driver was renamed to iavf,
- which is why the name discrepancy exists. After updating the Kernel we
- stopped seeing this issue.
+ IOV. We have ensured that SR-IOV is enabled in the BIOS. We are finding
+ that packets stop coming into the interface after a few seconds. This is
+ because the interface is going down after a few seconds of receiving
+ traffic. We are running promiscuous mode and it is required for our use
+ case of measuring network traffic.
- Non-working default security onion kernel information
+ As a troubleshooting measure, 5.3.0-rc3 was built using the 3.2.3 iavf
+ SR-IOV driver. Note that the i40evf driver was renamed to iavf, which is
+ why the name discrepancy exists. After updating the Kernel we stopped
+ seeing this issue.
+
+ Non-working default security onion kernel information
root@ubuntu:~# uname -r
3.10.0-957.21.3.el7.x86_64
root@ubuntu:~# ethtool -i ens7
driver: iavf
version: 3.0.1-k
firmware-version: N/A
expansion-rom-version:
bus-info: :00:07.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
- Working modified security onion kernel information
+ Working modified security onion kernel information
root@ubuntu:~# uname -r
5.3.0-rc3-custom
root@ubuntu:~# ethtool -i ens7
driver: i40evf
version: 3.2.3-k
firmware-version: N/A
expansion-rom-version:
bus-info: :00:07.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
-
- We were able to perform testing using the 5.3.0 kernel, but long term a
better solution is needed as this would require supporting a modified Linux
kernel with updates, testing, etc.
+ We were able to perform testing using the 5.3.0 kernel, but long term a
+ better solution is needed as this would require supporting a modified
+ Linux kernel with updates, testing, etc.
From the hypervisor in /var/log/messages:
Aug 27 09:05:00 localhost libvirtd: 2019-08-27 13:05:00.522+: 10748:
error : qemuDomainAgentAvailable:9133 : Guest agent is not
responding: QEMU guest agent is not connected
Aug 27 09:05:07 localhost goferd: [ERROR][worker-0]
gofer.messaging.adapter.conn ect:33 - connect:
proton+amqps://subscription.rhsm.redhat.com:5647, failed: Conn
ection amqps://subscription.rhsm.redhat.com:5647disconnected: Condition('amqp:r
esource-limit-exceeded', 'local-idle-timeout expired')
Aug 27 09:05:07 localhost goferd: [INFO][worker-0]
gofer.messaging.adapter.conne ct:35 - retry in 106 seconds
Aug 27 09:05:22 localhost kernel: i40e :b5:00.3: Unable to send the
message to VF 0 aq_err 12
Aug 27 09:05:22 localhost kernel: i40e :b5:00.3: Unable to send the
message to VF 0 aq_err 12
-
From the VM, journalctl -xe gives:
-- The start-up result is done.
Aug 27 14:35:02 ubuntu kernel: i40evf :00:04.0: PF reset warning received
Aug 27 14:35:02 ubuntu kernel: i40evf :00:04.0: Scheduling reset task
Aug 27 14:35:03 ubuntu kernel: i40evf :00:04.0: Hardware reset detected
Aug 27 14:35:03 ubuntu kernel: i40evf :00:04.0: Hardware reset detected
Aug 27 14:35:03 ubuntu kernel: i40evf :00:04.0: Hardware reset detected
Aug 27 14:35:08 ubuntu systemd-logind[1251]: Removed session 16.
-- Subject: Session 16 has been terminated
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchp
[Bug 1841669] ProcCpuinfoMinimal.txt
apport information ** Attachment added: "ProcCpuinfoMinimal.txt" https://bugs.launchpad.net/bugs/1841669/+attachment/5285448/+files/ProcCpuinfoMinimal.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1841669 Title: SR-IOV interface goes down when testing network performance To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1841669/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1841669] ProcModules.txt
apport information ** Attachment added: "ProcModules.txt" https://bugs.launchpad.net/bugs/1841669/+attachment/5285450/+files/ProcModules.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1841669 Title: SR-IOV interface goes down when testing network performance To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1841669/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1841669] UdevDb.txt
apport information ** Attachment added: "UdevDb.txt" https://bugs.launchpad.net/bugs/1841669/+attachment/5285451/+files/UdevDb.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1841669 Title: SR-IOV interface goes down when testing network performance To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1841669/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1841669] ProcCpuinfo.txt
apport information ** Attachment added: "ProcCpuinfo.txt" https://bugs.launchpad.net/bugs/1841669/+attachment/5285447/+files/ProcCpuinfo.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1841669 Title: SR-IOV interface goes down when testing network performance To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1841669/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1841669] Lspci.txt
apport information ** Attachment added: "Lspci.txt" https://bugs.launchpad.net/bugs/1841669/+attachment/5285446/+files/Lspci.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1841669 Title: SR-IOV interface goes down when testing network performance To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1841669/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1841669] CurrentDmesg.txt
apport information ** Attachment added: "CurrentDmesg.txt" https://bugs.launchpad.net/bugs/1841669/+attachment/5285445/+files/CurrentDmesg.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1841669 Title: SR-IOV interface goes down when testing network performance To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1841669/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1841669] ProcInterrupts.txt
apport information ** Attachment added: "ProcInterrupts.txt" https://bugs.launchpad.net/bugs/1841669/+attachment/5285449/+files/ProcInterrupts.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1841669 Title: SR-IOV interface goes down when testing network performance To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1841669/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1841669] Re: SR-IOV interface goes down when testing network performance
apport information
** Tags added: apport-collected xenial
** Description changed:
Hello,
We are testing network performance with SR-IOV on ubuntu and security
onion in Rhev Hypervisor. We are using a SuperMicro box running T-rex
traffic generator on CentOS which is connected to a gigamon to push
traffic to the Ubuntu/Security Onion VMs.
The ubuntu 16.04 vm is running on a RHVH host node and is running SR-
IOV. We have ensured that SR-IOV is enabled in the BIOS. We are finding
that packets stop coming into the interface after a few seconds. This is
because the interface is going down after a few seconds of receiving
traffic. We are running promiscuous mode and it is required for our use
case of measuring network traffic.
As a troubleshooting measure, 5.3.0-rc3 was built using the 3.2.3 iavf
SR-IOV driver. Note that the i40evf driver was renamed to iavf, which is
why the name discrepancy exists. After updating the Kernel we stopped
seeing this issue.
Non-working default security onion kernel information
root@ubuntu:~# uname -r
3.10.0-957.21.3.el7.x86_64
root@ubuntu:~# ethtool -i ens7
driver: iavf
version: 3.0.1-k
firmware-version: N/A
expansion-rom-version:
bus-info: :00:07.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
Working modified security onion kernel information
root@ubuntu:~# uname -r
5.3.0-rc3-custom
root@ubuntu:~# ethtool -i ens7
driver: i40evf
version: 3.2.3-k
firmware-version: N/A
expansion-rom-version:
bus-info: :00:07.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
We were able to perform testing using the 5.3.0 kernel, but long term a
better solution is needed as this would require supporting a modified
Linux kernel with updates, testing, etc.
From the hypervisor in /var/log/messages:
Aug 27 09:05:00 localhost libvirtd: 2019-08-27 13:05:00.522+: 10748:
error : qemuDomainAgentAvailable:9133 : Guest agent is not
responding: QEMU guest agent is not connected
Aug 27 09:05:07 localhost goferd: [ERROR][worker-0]
gofer.messaging.adapter.conn ect:33 - connect:
proton+amqps://subscription.rhsm.redhat.com:5647, failed: Conn
ection amqps://subscription.rhsm.redhat.com:5647disconnected: Condition('amqp:r
esource-limit-exceeded', 'local-idle-timeout expired')
Aug 27 09:05:07 localhost goferd: [INFO][worker-0]
gofer.messaging.adapter.conne ct:35 - retry in 106 seconds
Aug 27 09:05:22 localhost kernel: i40e :b5:00.3: Unable to send the
message to VF 0 aq_err 12
Aug 27 09:05:22 localhost kernel: i40e :b5:00.3: Unable to send the
message to VF 0 aq_err 12
From the VM, journalctl -xe gives:
-- The start-up result is done.
Aug 27 14:35:02 ubuntu kernel: i40evf :00:04.0: PF reset warning received
Aug 27 14:35:02 ubuntu kernel: i40evf :00:04.0: Scheduling reset task
Aug 27 14:35:03 ubuntu kernel: i40evf :00:04.0: Hardware reset detected
Aug 27 14:35:03 ubuntu kernel: i40evf :00:04.0: Hardware reset detected
Aug 27 14:35:03 ubuntu kernel: i40evf :00:04.0: Hardware reset detected
Aug 27 14:35:08 ubuntu systemd-logind[1251]: Removed session 16.
-- Subject: Session 16 has been terminated
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat
+ ---
+ AlsaDevices:
+ total 0
+ crw-rw 1 root audio 116, 1 Aug 28 17:06 seq
+ crw-rw 1 root audio 116, 33 Aug 28 17:06 timer
+ AplayDevices: Error: [Errno 2] No such file or directory
+ ApportVersion: 2.20.1-0ubuntu2.18
+ Architecture: amd64
+ ArecordDevices: Error: [Errno 2] No such file or directory
+ AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit code 1:
+ DistroRelease: Ubuntu 16.04
+ HibernationDevice: RESUME=/dev/mapper/ubuntu--vg-swap_1
+ InstallationDate: Installed on 2019-08-21 (6 days ago)
+ InstallationMedia: Ubuntu-Server 16.04.6 LTS "Xenial Xerus" - Release amd64
(20190226)
+ IwConfig: Error: [Errno 2] No such file or directory
+ Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
+ MachineType: Red Hat RHEV Hypervisor
+ Package: linux (not installed)
+ PciMultimedia:
+
+ ProcEnviron:
+ TERM=linux
+ PATH=(custom, no user)
+ XDG_RUNTIME_DIR=
+ LANG=en_US.UTF-8
+ SHELL=/bin/bash
+ ProcFB: 0 qxldrmfb
+ ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.15.0-45-generic
root=/dev/mapper/hostname--vg-root ro quiet splash isolcpus=2-27 vt.handoff=7
+ ProcVersionSignature: Ubuntu 4.15.0-45.48~16.04.1-generic 4.15.18
+ RelatedPackageVersions:
+ linux-restricted-modules-4.15.0-45-generic N/A
+ linux-backports-modules-4.15
[Bug 1841669] WifiSyslog.txt
apport information ** Attachment added: "WifiSyslog.txt" https://bugs.launchpad.net/bugs/1841669/+attachment/5285452/+files/WifiSyslog.txt ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1841669 Title: SR-IOV interface goes down when testing network performance To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1841669/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
