Public bug reported: Charmed OpenStack on Jammy.
Hardware offloaded instances that reside on the hypervisor which is also the active OVN gateway for their network have degraded connectivity. $ time wget --debug -O /dev/null http://nova.clouds.archive.ubuntu.com/ubuntu/ Setting --output-document (outputdocument) to /dev/null DEBUG output created by Wget 1.21.2 on linux-gnu. Reading HSTS entries from /home/ubuntu/.wget-hsts URI encoding = ‘UTF-8’ --2022-08-25 07:16:54-- http://nova.clouds.archive.ubuntu.com/ubuntu/ Resolving nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)... 91.189.91.121, 185.125.190.38, 185.125.190.41, ... Caching nova.clouds.archive.ubuntu.com => 91.189.91.121 185.125.190.38 185.125.190.41 91.189.91.122 2001:67c:1562::29 2620:2d:4000:1::1b 2620:2d:4000:1::18 2001:67c:1562::26 Connecting to nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)|91.189.91.121|:80... connected. Created socket 4. Releasing 0x000055ef05654490 (new refcount 1). ---request begin--- GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Date: Thu, 25 Aug 2022 07:16:54 GMT Server: Apache/2.4.29 (Ubuntu) Vary: Accept-Encoding Content-Length: 1779 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/html;charset=UTF-8 ---response end--- 200 OK Registered socket 4 for persistent reuse. URI content encoding = ‘UTF-8’ Length: 1779 (1.7K) [text/html] Saving to: ‘/dev/null’ /dev/null 100%[====================================================================================================================================>] 1.74K --.-KB/s in 0.07s 2022-08-25 07:16:59 (24.3 KB/s) - ‘/dev/null’ saved [1779/1779] real 0m5.591s user 0m0.000s sys 0m0.005s $ sudo tcpdump -nevi ens3 not port 22 07:16:54.017265 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 50207, offset 0, flags [DF], proto TCP (6), length 60) 10.42.1.8.43060 > 91.189.91.121.80: Flags [S], cksum 0xc296 (incorrect -> 0x4190), seq 2905559525, win 62314, options [mss 8902,sackOK,TS val 4081686695 ecr 0,nop,wscale 7], length 0 07:16:54.091378 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 58, id 0, offset 0, flags [DF], proto TCP (6), length 60) 91.189.91.121.80 > 10.42.1.8.43060: Flags [S.], cksum 0x809d (correct), seq 3404088064, ack 2905559526, win 65160, options [mss 1460,sackOK,TS val 932285789 ecr 4081686695,nop,wscale 7], length 0 07:16:54.091402 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 50208, offset 0, flags [DF], proto TCP (6), length 52) 10.42.1.8.43060 > 91.189.91.121.80: Flags [.], cksum 0xc28e (incorrect -> 0xabc0), ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 0 07:16:54.091512 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 218: (tos 0x0, ttl 64, id 50209, offset 0, flags [DF], proto TCP (6), length 204) 10.42.1.8.43060 > 91.189.91.121.80: Flags [P.], cksum 0xc326 (incorrect -> 0x7d0c), seq 1:153, ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 152: HTTP, length: 152 GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive 07:16:54.166170 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15660, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [.], cksum 0xaac7 (correct), ack 153, win 508, options [nop,nop,TS val 932285865 ecr 4081686770], length 0 07:16:59.171924 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15669, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [F.], cksum 0x8f61 (correct), seq 2008, ack 153, win 508, options [nop,nop,TS val 932290871 ecr 4081686770], length 0 07:16:59.171956 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 78: (tos 0x0, ttl 64, id 50210, offset 0, flags [DF], proto TCP (6), length 64) 10.42.1.8.43060 > 91.189.91.121.80: Flags [.], cksum 0xc29a (incorrect -> 0x356e), ack 1, win 487, options [nop,nop,TS val 4081691850 ecr 932285865,nop,nop,sack 1 {2008:2009}], length 0 07:16:59.531552 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 1440: (tos 0x0, ttl 58, id 15671, offset 0, flags [DF], proto TCP (6), length 1426) 91.189.91.121.80 > 10.42.1.8.43060: Flags [.], cksum 0x0d7c (correct), seq 1:1375, ack 153, win 508, options [nop,nop,TS val 932291230 ecr 4081691850], length 1374: HTTP, length: 1374 HTTP/1.1 200 OK Date: Thu, 25 Aug 2022 07:16:54 GMT Server: Apache/2.4.29 (Ubuntu) Vary: Accept-Encoding Content-Length: 1779 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/html;charset=UTF-8 ... The issue appears to be present in mainline kernels up to v5.17, and appears to be fixed in kernel v5.18. We have yet to identify exactly what in the diff from v5.17 to v5.18 fixes the issue. ** Affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1987618 Title: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways Status in linux package in Ubuntu: New Bug description: Charmed OpenStack on Jammy. Hardware offloaded instances that reside on the hypervisor which is also the active OVN gateway for their network have degraded connectivity. $ time wget --debug -O /dev/null http://nova.clouds.archive.ubuntu.com/ubuntu/ Setting --output-document (outputdocument) to /dev/null DEBUG output created by Wget 1.21.2 on linux-gnu. Reading HSTS entries from /home/ubuntu/.wget-hsts URI encoding = ‘UTF-8’ --2022-08-25 07:16:54-- http://nova.clouds.archive.ubuntu.com/ubuntu/ Resolving nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)... 91.189.91.121, 185.125.190.38, 185.125.190.41, ... Caching nova.clouds.archive.ubuntu.com => 91.189.91.121 185.125.190.38 185.125.190.41 91.189.91.122 2001:67c:1562::29 2620:2d:4000:1::1b 2620:2d:4000:1::18 2001:67c:1562::26 Connecting to nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)|91.189.91.121|:80... connected. Created socket 4. Releasing 0x000055ef05654490 (new refcount 1). ---request begin--- GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Date: Thu, 25 Aug 2022 07:16:54 GMT Server: Apache/2.4.29 (Ubuntu) Vary: Accept-Encoding Content-Length: 1779 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/html;charset=UTF-8 ---response end--- 200 OK Registered socket 4 for persistent reuse. URI content encoding = ‘UTF-8’ Length: 1779 (1.7K) [text/html] Saving to: ‘/dev/null’ /dev/null 100%[====================================================================================================================================>] 1.74K --.-KB/s in 0.07s 2022-08-25 07:16:59 (24.3 KB/s) - ‘/dev/null’ saved [1779/1779] real 0m5.591s user 0m0.000s sys 0m0.005s $ sudo tcpdump -nevi ens3 not port 22 07:16:54.017265 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 50207, offset 0, flags [DF], proto TCP (6), length 60) 10.42.1.8.43060 > 91.189.91.121.80: Flags [S], cksum 0xc296 (incorrect -> 0x4190), seq 2905559525, win 62314, options [mss 8902,sackOK,TS val 4081686695 ecr 0,nop,wscale 7], length 0 07:16:54.091378 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 58, id 0, offset 0, flags [DF], proto TCP (6), length 60) 91.189.91.121.80 > 10.42.1.8.43060: Flags [S.], cksum 0x809d (correct), seq 3404088064, ack 2905559526, win 65160, options [mss 1460,sackOK,TS val 932285789 ecr 4081686695,nop,wscale 7], length 0 07:16:54.091402 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 50208, offset 0, flags [DF], proto TCP (6), length 52) 10.42.1.8.43060 > 91.189.91.121.80: Flags [.], cksum 0xc28e (incorrect -> 0xabc0), ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 0 07:16:54.091512 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 218: (tos 0x0, ttl 64, id 50209, offset 0, flags [DF], proto TCP (6), length 204) 10.42.1.8.43060 > 91.189.91.121.80: Flags [P.], cksum 0xc326 (incorrect -> 0x7d0c), seq 1:153, ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 152: HTTP, length: 152 GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive 07:16:54.166170 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15660, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [.], cksum 0xaac7 (correct), ack 153, win 508, options [nop,nop,TS val 932285865 ecr 4081686770], length 0 07:16:59.171924 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15669, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [F.], cksum 0x8f61 (correct), seq 2008, ack 153, win 508, options [nop,nop,TS val 932290871 ecr 4081686770], length 0 07:16:59.171956 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 78: (tos 0x0, ttl 64, id 50210, offset 0, flags [DF], proto TCP (6), length 64) 10.42.1.8.43060 > 91.189.91.121.80: Flags [.], cksum 0xc29a (incorrect -> 0x356e), ack 1, win 487, options [nop,nop,TS val 4081691850 ecr 932285865,nop,nop,sack 1 {2008:2009}], length 0 07:16:59.531552 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 1440: (tos 0x0, ttl 58, id 15671, offset 0, flags [DF], proto TCP (6), length 1426) 91.189.91.121.80 > 10.42.1.8.43060: Flags [.], cksum 0x0d7c (correct), seq 1:1375, ack 153, win 508, options [nop,nop,TS val 932291230 ecr 4081691850], length 1374: HTTP, length: 1374 HTTP/1.1 200 OK Date: Thu, 25 Aug 2022 07:16:54 GMT Server: Apache/2.4.29 (Ubuntu) Vary: Accept-Encoding Content-Length: 1779 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/html;charset=UTF-8 ... The issue appears to be present in mainline kernels up to v5.17, and appears to be fixed in kernel v5.18. We have yet to identify exactly what in the diff from v5.17 to v5.18 fixes the issue. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1987618/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
