Re: Kernel 4.19 network performance - forwarding/routing normal users traffic

Paweł Staszewski Thu, 08 Nov 2018 08:27:45 -0800



W dniu 08.11.2018 o 17:25, Paweł Staszewski pisze:



W dniu 08.11.2018 o 17:06, David Ahern pisze:

On 11/8/18 6:33 AM, Paweł Staszewski wrote:


W dniu 07.11.2018 o 22:06, David Ahern pisze:

On 11/3/18 6:24 PM, Paweł Staszewski wrote:

Does your setup have any other device types besides physicalports with
VLANs (e.g., any macvlans or bonds)?
no.
just
phy(mlnx)->vlans only config

VLAN and non-VLAN (and a mix) seem to work ok. Patches are here:
     https://github.com/dsahern/linux.git bpf/kernel-tables-wip

I got lazy with the vlan exports; right now it requires 8021q to be
builtin (CONFIG_VLAN_8021Q=y)

You can use the xdp_fwd sample:
    make O=kbuild -C samples/bpf -j 8

Copy samples/bpf/xdp_fwd_kern.o and samples/bpf/xdp_fwd to the server
and run:
     ./xdp_fwd <list of NIC ports>

e.g., in my testing I run:
     xdp_fwd eth1 eth2 eth3 eth4

All of the relevant forwarding ports need to be on the same command

line. This version populates a second map to verify the egress porthas

XDP enabled.

Installed today on some lab server with mellanox connectx4

And trying some simple static routing first - but after enabling xdp
program - receiver is not receiving frames

Route table is simple as possible for tests :)

icmp ping test send from 192.168.22.237 to 172.16.0.2 - incomming
packets on vlan 4081

ip r
default via 192.168.22.236 dev vlan4081
172.16.0.0/30 dev vlan1740 proto kernel scope link src 172.16.0.1
192.168.22.0/24 dev vlan4081 proto kernel scope link src 192.168.22.205

neigh table:
ip neigh ls

192.168.22.237 dev vlan4081 lladdr 00:25:90:fb:a6:8d REACHABLE
172.16.0.2 dev vlan1740 lladdr ac:1f:6b:2c:2e:5a REACHABLE

and interfaces:

4: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mqstate

UP mode DEFAULT group default qlen 1000
     link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff

5: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mqstate

UP mode DEFAULT group default qlen 1000
     link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff

6: vlan4081@enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500qdisc

noqueue state UP mode DEFAULT group default qlen 1000
     link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff

7: vlan1740@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500qdisc

noqueue state UP mode DEFAULT group default qlen 1000
     link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff

5: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp/id:5qdisc

mq state UP group default qlen 1000
     link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
     inet6 fe80::ae1f:6bff:fe07:c891/64 scope link
        valid_lft forever preferred_lft forever

6: vlan4081@enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500qdisc

noqueue state UP group default qlen 1000
     link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
     inet 192.168.22.205/24 scope global vlan4081
        valid_lft forever preferred_lft forever
     inet6 fe80::ae1f:6bff:fe07:c890/64 scope link
        valid_lft forever preferred_lft forever

7: vlan1740@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500qdisc

noqueue state UP group default qlen 1000
     link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
     inet 172.16.0.1/30 scope global vlan1740
        valid_lft forever preferred_lft forever
     inet6 fe80::ae1f:6bff:fe07:c891/64 scope link
        valid_lft forever preferred_lft forever


xdp program detached:
Receiving side tcpdump:
14:28:09.141233 IP 192.168.22.237 > 172.16.0.2: ICMP echo request, id
30227, seq 487, length 64

I can see icmp requests


enabling xdp
./xdp_fwd enp175s0f1 enp175s0f0

4: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc mq
state UP mode DEFAULT group default qlen 1000
     link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
     prog/xdp id 5 tag 3c231ff1e5e77f3f
5: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc mq
state UP mode DEFAULT group default qlen 1000
     link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
     prog/xdp id 5 tag 3c231ff1e5e77f3f

6: vlan4081@enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500qdisc

noqueue state UP mode DEFAULT group default qlen 1000
     link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff

7: vlan1740@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500qdisc

noqueue state UP mode DEFAULT group default qlen 1000
     link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff

What hardware is this?

mellanox connectx 4
ethtool -i enp175s0f0
driver: mlx5_core
version: 5.0-0
firmware-version: 12.21.1000 (SM_2001000001033)
expansion-rom-version:
bus-info: 0000:af:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

ethtool -i enp175s0f1
driver: mlx5_core
version: 5.0-0
firmware-version: 12.21.1000 (SM_2001000001033)
expansion-rom-version:
bus-info: 0000:af:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

Start with:

echo 1 > /sys/kernel/debug/tracing/events/xdp/enable
cat /sys/kernel/debug/tracing/trace_pipe
 cat /sys/kernel/debug/tracing/trace_pipe
<idle>-0 [045] ..s. 68469.467752: xdp_devmap_xmit:ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1from_ifindex=4 to_ifindex=5 err=-6 <idle>-0 [045] ..s. 68470.483836: xdp_redirect_map:prog_id=30 action=REDIRECT ifindex=4 to_ifindex=5 err=0 map_id=32map_index=5 <idle>-0 [045] ..s. 68470.483837: xdp_devmap_xmit:ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1from_ifindex=4 to_ifindex=5 err=-6 <idle>-0 [045] ..s. 68471.503853: xdp_redirect_map:prog_id=30 action=REDIRECT ifindex=4 to_ifindex=5 err=0 map_id=32map_index=5 <idle>-0 [045] ..s. 68471.503853: xdp_devmap_xmit:ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1from_ifindex=4 to_ifindex=5 err=-6 <idle>-0 [045] ..s. 68472.527871: xdp_redirect_map:prog_id=30 action=REDIRECT ifindex=4 to_ifindex=5 err=0 map_id=32map_index=5 <idle>-0 [045] ..s. 68472.527877: xdp_devmap_xmit:ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1from_ifindex=4 to_ifindex=5 err=-6 <idle>-0 [045] ..s. 68473.551876: xdp_redirect_map:prog_id=30 action=REDIRECT ifindex=4 to_ifindex=5 err=0 map_id=32map_index=5 <idle>-0 [045] ..s. 68473.551880: xdp_devmap_xmit:ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1from_ifindex=4 to_ifindex=5 err=-6 <idle>-0 [045] ..s. 68474.575893: xdp_redirect_map:prog_id=30 action=REDIRECT ifindex=4 to_ifindex=5 err=0 map_id=32map_index=5 <idle>-0 [045] ..s. 68474.575897: xdp_devmap_xmit:ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1from_ifindex=4 to_ifindex=5 err=-6 <idle>-0 [045] ..s. 68475.599909: xdp_redirect_map:prog_id=30 action=REDIRECT ifindex=4 to_ifindex=5 err=0 map_id=32map_index=5 <idle>-0 [045] ..s. 68475.599912: xdp_devmap_xmit:ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1from_ifindex=4 to_ifindex=5 err=-6
>From there, you can check the FIB lookups:
sysctl -w kernel.perf_event_max_stack=16
perf record -e fib:* -a -g -- sleep 5
perf script
swapper 0 [045] 68493.746274: fib:fib_table_lookup: table 254 oif0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
            7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
swapper 0 [045] 68494.770287: fib:fib_table_lookup: table 254 oif0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
            7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
swapper 0 [045] 68495.794304: fib:fib_table_lookup: table 254 oif0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
            7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
swapper 0 [045] 68496.818308: fib:fib_table_lookup: table 254 oif0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
            7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
swapper 0 [045] 68497.842313: fib:fib_table_lookup: table 254 oif0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
            7fff818c13b5 fib_table_lookup ([kernel.kallsyms])

Re: Kernel 4.19 network performance - forwarding/routing normal users traffic

Reply via email to