From: David Ahern <d...@cumulusnetworks.com> Date: Mon, 20 Mar 2017 11:19:43 -0700
> Device based features for VRF such as qdisc, netfilter and packet > captures are implemented by switching the dst on skbuffs to its per-VRF > dst. This has the effect of controlling the output function which points > a function in the VRF driver. [1] The skb proceeds down the stack with > dst->dev pointing to the VRF device. Netfilter, qdisc and tc rules and > network taps are evaluated based on this device. Finally, the skb makes > it to the vrf_xmit function which resets the dst based on a FIB lookup. > > The feature comes at cost - between 5 and 10% depending on test (TCP vs > UDP, stream vs RR and IPv4 vs IPv6). The main cost is requiring a FIB > lookup in the VRF driver for each packet sent through it. The FIB lookup > is required because the real dst gets dropped so that the skb can > traverse the stack with dst->dev set to the VRF device. > > All of that is really driven by the qdisc and not replicating the > processing of __dev_queue_xmit if a qdisc is set up on the device. But, > VRF devices by default do not have a qdisc and really have no need for > multiple Tx queues. This means the performance overhead is inflicted upon > all users for the potential use case of a qdisc being configured. > > The overhead can be avoided by checking if the default configuration > applies to a specific VRF device before switching the dst. If a device > does not have a qdisc, the pass through netfilter hooks and packet taps > can be done inline without dropping the dst and thus avoiding the > performance penalty. With this change performance overhead of VRF drops > to neglible (difference with run-over-run variance) to 3% depending on > test type. ... > * UDP is consistently better with VRF for two reasons: > 1. Source address selection with L3 domains is considering fewer > addresses since only addresses on interfaces in the domain are > considered for the selection. Specifically, perf-top shows > shows ipv6_get_saddr_eval, ipv6_dev_get_saddr and __ipv6_dev_get_saddr > running much lower with vrf than without. > > 2. The VRF table contains all routes (i.e, there are no separate local > and main tables per VRF). That means ip6_pol_route_output only has 1 > lookup for VRF where it does 2 without it (1 in the local table and 1 > in the main table). > > [1] http://netdevconf.org/1.2/papers/ahern-what-is-l3mdev-paper.pdf Series applied, thanks David.
