On Sun, 01 Nov 2020 13:16:32 +0000 Alexander Lobakin wrote: > NETIF_F_GSO_UDP_L4 and NETIF_F_GSO_FRAGLIST allow drivers to offload > GSO UDP L4. This works well on simple setups, but when any logical > netdev (e.g. VLAN) is present, kernel stack always performs software > resegmentation which actually kills the performance. > > The full path in such cases is like: > 1. Our NIC driver advertises a support for fraglists, GSO UDP L4, GSO > fraglists. > 2. User enables fraglisted GRO via Ethtool. > 3. GRO subsystem receives UDP frames from driver and merges the packets > into fraglisted GSO skb(s). > 4. Networking stack queues it up for xmitting. > 5. Virtual device like VLAN doesn't advertise a support for GSO UDP L4 > and GSO fraglists, so skb_gso_check() doesn't allow to pass this skb > as is to the real driver. > 6. Kernel then has to form a bunch of regular UDP skbs from that one and > pass it to the driver instead. This fallback is *extremely* slow for > any GSO types, but especially for GSO fraglists. > 7. All further processing performs with a series of plain UDP skbs, and > the driver gets it one-by-one, despite that it supports UDP L4 and > fraglisted GSO. > > That's not OK because: > a) logical/virtual netdevs like VLANs, bridges etc. should pass GSO skbs > as is; > b) even if the final driver doesn't support such type of GSO, this > software resegmenting should be performed right before it, not in the > middle of processing -- I think I even saw that note somewhere in > kernel documentation, and it's totally reasonable in terms of > performance. > > Despite the fact that no mainline drivers currently supports fraglist > GSO, this should and can be easily fixed by adding UDP L4 and fraglist > GSO to the list of GSO types that can be passed-through the logical > interfaces (NETIF_F_GSO_SOFTWARE). After this change, no resegmentation > occurs (if a particular driver supports and advertises this), and the > performance goes on par with e.g. 1:1 forwarding. > The only logical netdevs that seem to be unaffected to this are bridge > interfaces, as their code uses full NETIF_F_GSO_MASK. > > Tested on MIPS32 R2 router board with a WIP NIC driver in VLAN NAT: > 20 Mbps baseline, 1 Gbps / link speed with this patch. > > Since v1 [1]: > - handle bonding and team drivers as suggested by Willem de Bruijn; > - reword and expand the introduction with the particular example. > > [1] > https://lore.kernel.org/netdev/mx3bwgop6fgorn6cpo4mhihz2b1bb0elxemg8vsi...@cp3-web-020.plabs.ch
Applied, thanks!
