CC some bonding folks
On Tue, 20 Oct 2020 07:54:29 -0700 Stephen Hemminger wrote:
> Begin forwarded message:
>
> Date: Tue, 20 Oct 2020 10:42:34 +0000
> From: bugzilla-dae...@bugzilla.kernel.org
> To: step...@networkplumber.org
> Subject: [Bug 209767] New: Bonding 802.3ad layer2+3 transmits on both slaves
> within single connection
>
>
> https://bugzilla.kernel.org/show_bug.cgi?id=209767
>
> Bug ID: 209767
> Summary: Bonding 802.3ad layer2+3 transmits on both slaves
> within single connection
> Product: Networking
> Version: 2.5
> Kernel Version: 5.8.11-1.el8.elrepo.x86_64 and
> 3.10.0-1127.19.1.el7.x86_64
> Hardware: All
> OS: Linux
> Tree: Mainline
> Status: NEW
> Severity: normal
> Priority: P1
> Component: Other
> Assignee: step...@networkplumber.org
> Reporter: onno.zwe...@surf.nl
> Regression: No
>
> Dear people,
>
> I'm seeing bonding behavior I don't understand and neither do several network
> experts I've consulted.
>
> We have two servers, both with two 25 Gbit interfaces in a bond (802.3ad with
> layer2+3 hashing). We tuned the systems according to
> https://fasterdata.es.net/host-tuning/linux/. I run `iperf3 --server` on
> server
> 1 and connect to it with `iperf3 --client server1` from server 2. We notice
> that sometimes the connection is good (24.7 Gbit/s, no retransmits) and
> sometimes there are many retransmits (sometimes as many as >30,000 in a 10
> second run) and then the bandwidth may drop to 15 Gbit/s or even lower. The
> servers are idle except for the iperf3 runs. When we bring down one slave on
> server 1, the result is always perfect; no retransmits and good throughput.
>
> We have captured traffic with tcpdump on server 1 at the slave level (I'll try
> to add the pcap files). To our surprise, we see that the data channel ACK
> packets are sometimes sent over one slave and sometimes over the other. We
> think this causes packet misordering in the network switches, and thus
> retransmits and loss of bandwidth.
>
> Our understanding of layer2+3 hashing is that for a single connection, all
> traffic should go over the same slave. Therefore, we don't understand why
> server 1 sends ACK packets out over both slaves.
>
> I've read the documentation at
> https://www.kernel.org/doc/Documentation/networking/bonding.txt but I couldn't
> find the observed behaviour explained there.
>
> We have tested several Centos 7 and Centos 8 kernels, including recent elrepo
> kernels, but all show this behaviour. We have tried teaming instead of bonding
> but it has the same behaviour. We have tried other hashing algorithms like
> layer3+4 but they seem to have the same issue. It occurs with both IPv4 and
> IPv6.
>
> Is this behaviour to be expected? If yes, is it documented anywhere? Will it
> degrade throughput in real life traffic (with multiple concurrent data
> streams)?
> If the behaviour is not expected, are we doing something wrong, or might it be
> a bug?
>
> Thanks,
> Onno
>
