This series listifies part of GRO processing, in a manner which allows those packets which are not GROed (i.e. for which dev_gro_receive returns GRO_NORMAL) to be passed on to the listified regular receive path. dev_gro_receive() itself is not listified, nor the per-protocol GRO callback, since GRO's need to hold packets on lists under napi->gro_hash makes keeping the packets on other lists awkward, and since the GRO control block state of held skbs can refer only to one 'new' skb at a time. Instead, when napi_frags_finish() handles a GRO_NORMAL result, stash the skb onto a list in the napi struct, which is received at the end of the napi poll or when its length exceeds the (new) sysctl net.core.gro_normal_batch. Unlike my previous design ([1]), this does not require changes in drivers, and also does not prevent the re-use of napi->skb after GRO_MERGED_FREE or GRO_DROP.
Performance figures with this series, collected on a back-to-back pair of Solarflare sfn8522-r2 NICs with 120-second NetPerf tests. In the stats, sample size n for old and new code is 6 runs each; p is from a Welch t-test. Tests were run both with GRO enabled and disabled, the latter simulating uncoalesceable packets (e.g. due to IP or TCP options). The receive side (which was the device under test) had the NetPerf process pinned to one CPU, and the device interrupts pinned to a second CPU. CPU utilisation figures (used in cases of line-rate performance) are summed across all CPUs. Where not specified (as batch=), net.core.gro_normal_batch was set to 8. The net-next baseline used for these tests was commit 7d30a7f6424e. TCP 4 streams, GRO on: all results line rate (9.415Gbps) net-next: 210.3% cpu after #1: 181.5% cpu (-13.7%, p=0.031 vs net-next) after #3: 191.7% cpu (- 8.9%, p=0.102 vs net-next) TCP 4 streams, GRO off: after #1: 7.785 Gbps after #3: 8.387 Gbps (+ 7.7%, p=0.215 vs #1, but note *) TCP 1 stream, GRO on: all results line rate & ~200% cpu. TCP 1 stream, GRO off: after #1: 6.444 Gbps after #3: 7.363 Gbps (+14.3%, p=0.003 vs #1) batch=16: 7.199 Gbps batch= 4: 7.354 Gbps batch= 0: 5.899 Gbps TCP 100 RR, GRO off: net-next: 995.083 us after #1: 969.167 us (- 2.6%, p=0.204 vs net-next) after #3: 976.433 us (- 1.9%, p=0.254 vs net-next) (*) These tests produced a mixture of line-rate and below-line-rate results, meaning that statistically speaking the results were 'censored' by the upper bound, and were thus not normally distributed, making a Welch t-test mathematically invalid. I therefore also calculated estimators according to [2], which gave the following: after #1: 8.155 Gbps after #3: 8.716 Gbps (+ 6.9%, p=0.291 vs #1) (though my procedure for determining ν wasn't mathematically well-founded either, so take that p-value with a grain of salt). Conclusion: * Patch #1 is a fairly unambiguous improvement. * Patch #3 has no statistically significant effect when GRO is active. * Any effect of patch #3 on latency is within statistical noise. * When GRO is inactive, patch #3 improves bandwidth, though for multiple streams the effect is smaller (possibly owing to the line-rate limit). * The optimal batch size for this setup appears to be around 8. * Setting the batch size to zero gives worse performance than before the patch; perhaps a static key is needed? * Drivers which, unlike sfc, pass UDP traffic to GRO would expect to see a benefit from gaining access to batching. Notes for future thought: in principle if we passed the napi pointer to napi_gro_complete(), it could add its superframe skb to napi->rx_list, rather than immediately netif_receive_skb_internal()ing it. Without that I'm not sure if there's a possibility of OoO between normal and GROed SKBs on the same flow. [1]: http://patchwork.ozlabs.org/cover/997844/ [2]: Cohen 1959, doi: 10.1080/00401706.1959.10489859 Edward Cree (3): sfc: don't score irq moderation points for GRO sfc: falcon: don't score irq moderation points for GRO net: use listified RX for handling GRO_NORMAL skbs drivers/net/ethernet/sfc/falcon/rx.c | 5 +---- drivers/net/ethernet/sfc/rx.c | 5 +---- include/linux/netdevice.h | 3 +++ net/core/dev.c | 32 ++++++++++++++++++++++++++-- net/core/sysctl_net_core.c | 8 +++++++ 5 files changed, 43 insertions(+), 10 deletions(-)
