After applying Eric's proposed change (see below) to a 4.17 RC3 kernel, the regressions that we had observed in our TCP_STREAM small message tests with TCP_NODELAY enabled are now drastically reduced. Instead of the original 3x thruput and cpu cost regressions, the regression depth is now < 10% for thruput and between 10% - 20% for cpu cost. The improvements in the TCP_RR tests that we had observed after Eric's original commit are not impacted by the change. It would be great if this change could make it into a patch.
Michael Wenig VMware Performance Engineering -----Original Message----- From: Eric Dumazet [mailto:eric.duma...@gmail.com] Sent: Monday, April 30, 2018 10:48 AM To: Ben Greear <gree...@candelatech.com>; Steven Rostedt <rost...@goodmis.org>; Michael Wenig <mwe...@vmware.com> Cc: netdev@vger.kernel.org; Shilpi Agarwal <sagar...@vmware.com>; Boon Ang <b...@vmware.com>; Darren Hart <dvh...@vmware.com>; Steven Rostedt <srost...@vmware.com>; Abdul Anshad Azeez <aaz...@vmware.com> Subject: Re: Performance regressions in TCP_STREAM tests in Linux 4.15 (and later) On 04/30/2018 09:36 AM, Eric Dumazet wrote: > > > On 04/30/2018 09:14 AM, Ben Greear wrote: >> On 04/27/2018 08:11 PM, Steven Rostedt wrote: >>> >>> We'd like this email archived in netdev list, but since netdev is >>> notorious for blocking outlook email as spam, it didn't go through. >>> So I'm replying here to help get it into the archives. >>> >>> Thanks! >>> >>> -- Steve >>> >>> >>> On Fri, 27 Apr 2018 23:05:46 +0000 >>> Michael Wenig <mwe...@vmware.com> wrote: >>> >>>> As part of VMware's performance testing with the Linux 4.15 kernel, >>>> we identified CPU cost and throughput regressions when comparing to >>>> the Linux 4.14 kernel. The impacted test cases are mostly >>>> TCP_STREAM send tests when using small message sizes. The >>>> regressions are significant (up 3x) and were tracked down to be a >>>> side effect of Eric Dumazat's RB tree changes that went into the Linux >>>> 4.15 kernel. >>>> Further investigation showed our use of the TCP_NODELAY flag in >>>> conjunction with Eric's change caused the regressions to show and >>>> simply disabling TCP_NODELAY brought performance back to normal. >>>> Eric's change also resulted into significant improvements in our >>>> TCP_RR test cases. >>>> >>>> >>>> >>>> Based on these results, our theory is that Eric's change made the >>>> system overall faster (reduced latency) but as a side effect less >>>> aggregation is happening (with TCP_NODELAY) and that results in >>>> lower throughput. Previously even though TCP_NODELAY was set, >>>> system was slower and we still got some benefit of aggregation. >>>> Aggregation helps in better efficiency and higher throughput >>>> although it can increase the latency. If you are seeing a >>>> regression in your application throughput after this change, using >>>> TCP_NODELAY might help bring performance back however that might increase >>>> latency. >> >> I guess you mean _disabling_ TCP_NODELAY instead of _using_ TCP_NODELAY? >> > > Yeah, I guess auto-corking does not work as intended. I would try the following patch : diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 44be7f43455e4aefde8db61e2d941a69abcc642a..c9d00ef54deca15d5760bcbe154001a96fa1e2a7 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -697,7 +697,7 @@ static bool tcp_should_autocork(struct sock *sk, struct sk_buff *skb, { return skb->len < size_goal && sock_net(sk)->ipv4.sysctl_tcp_autocorking && - skb != tcp_write_queue_head(sk) && + !tcp_rtx_queue_empty(sk) && refcount_read(&sk->sk_wmem_alloc) > skb->truesize; }
