Subject changed from “Cake on elements of a bridge”... On Sep 10, 2018, at 9:55 PM, Dave Taht <[email protected]> wrote: > > On Mon, Sep 10, 2018 at 12:29 PM Pete Heist <[email protected]> wrote: >> >> For anyone who followed this, yes, the regular soft bridge (i.e. set >> interfaces bridge br0) works fine on the ER-X, as I suspect it would on most >> any Linux. A few notes about it: >> >> - Your qdisc must be added to the physical interface (e.g. eth4), not the >> bridge interface >> - Unlike the hardware bridge which has its own MAC, the soft bridge seems to >> take the MAC of the lowest (or first listed?) interface port >> - On ER-X, bridge-nf-call-iptables=1 is the default so nothing needs to be >> changed there for firewalling >> - When firewalling the bridged WAN interface, ‘in’ corresponds to bridged >> traffic and ‘local’ to routed traffic, which is different from the semantics >> for ordinary routed traffic >> - I can do stateful firewalling for bridged hosts with “accept established >> and related”, but have to explicitly allow DHCP (UDP source/dest port 67-68) >> in the WAN interface’s ‘in’ rules for DHCP traffic to pass through the bridge >> >> Performance: >> >> Using Cake with this setup, the fun ends at around 110 Mbit with ksoftirqd >> thrashing. Unsurprisingly, there’s probably some overhead here with the soft >> bridge. For my purposes though (50 Mbit), it’s enough, barely… > > Can I encourage you to give regular ole htb+fq_codel sqm a shot with a > bigger burst and cburst size for htb? Fiddling with the htb quantum > isn't helping much, > but try this, from: https://github.com/tohojo/sqm-scripts/issues/71 > > (I am thinking burst and cburst should be about 1.1ms of buffering in size)
So this has turned info an interesting exercise that produced a result counter to what the common wisdom has been (that fq_codel is “faster” than cake). Because of that, I’m open to criticism of my methodology and different criteria for a successful bitrate for the shaper. First, note that these tests still through a bridge as above, but are for a more typical setup with separate qdisc instances on egress and ingress, as opposed to my “110 Mbit” result from above, which was for egress and ingress through a common IFB. It occurs to me that what I really want to know is the maximum set bitrate for the shaper where it still appears to be behaving properly and consistently, meaning, the actual measured TCP throughput is held steady, and at the expected percentage less than the set bitrate. I first find this out by setting a “comfortable” rate of 100Mbit and checking TCP throughput with iperf3, which is typically around 5% less than the set bitrate. Then I increase the shaper’s bitrate 5Mbit at a time and re-run the test until I find the last bitrate at which iperf3 reports a stable (within a few percent) and correct rate over 10 seconds for several runs in a row. See the attached iperf3 results for sample runs around the threshold rates. qdisc: egress Mbit / ingress Mbit cake nat dual-srchost / cake nat dual-dsthost ingress: 135 / 145 htb+fq_codel: 125 / 125 htb+fq_codel with burst/cburst=96000: 155 / 155 So with this testing criteria, I’m actually seeing cake “win” (with the exception of setting htb's burst/cburst to 96000, which shows a clear improvement, probably at the expense of something). I also see that the ingress rate for cake can be held steady to a bit higher of a bitrate than egress. I am using the ‘ingress’ keyword on ingress. I have to be careful here because from run to run there can be slight variations in behavior, but having repeated it several times at each bitrate around the threshold, I’m fairly certain about the results. In the ER-X manual (https://dl.ubnt.com/guides/edgemax/EdgeOS_UG.pdf), they give a guideline of 100-250Mbps on the “expected Smart Queue shaping performance” (which means fq_codel) for the ER-X. In reality, 100Mbps is comfortable, and 250Mbps seems impossible. You might be able to get that rate by setting fq_codel to 300+Mbit (and you can’t, through a bridge anyway), but is the queue really controlled? I think I’m applying at least a little more consistent criteria for “success" here at a given bitrate than we have before. I suppose I should repeat this test with different hardware to be surer of the claim, but I’m not sure when I’ll have the time. I will say that Cake’s shaper overall produces more satisfyingly consistent rates, and given its NAT support and host fairness, that’s why I’m likely to continue to use it when I can.
hostname:~/.ssh:% iperf3 -c a.b.c.230 Connecting to host a.b.c.230, port 5201 [ 5] local 192.168.1.20 port 51310 connected to a.b.c.230 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 14.5 MBytes 122 Mbits/sec [ 5] 1.00-2.00 sec 14.2 MBytes 119 Mbits/sec [ 5] 2.00-3.00 sec 14.3 MBytes 120 Mbits/sec [ 5] 3.00-4.00 sec 14.3 MBytes 120 Mbits/sec [ 5] 4.00-5.00 sec 14.3 MBytes 120 Mbits/sec [ 5] 5.00-6.00 sec 14.2 MBytes 119 Mbits/sec [ 5] 6.00-7.00 sec 14.3 MBytes 120 Mbits/sec [ 5] 7.00-8.00 sec 14.3 MBytes 120 Mbits/sec [ 5] 8.00-9.00 sec 14.2 MBytes 119 Mbits/sec [ 5] 9.00-10.00 sec 14.3 MBytes 120 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 143 MBytes 120 Mbits/sec sender [ 5] 0.00-10.00 sec 143 MBytes 120 Mbits/sec receiver
hostname:~/Downloads:% iperf3 -c a.b.c.230 Connecting to host a.b.c.230, port 5201 [ 5] local 192.168.1.20 port 51324 connected to a.b.c.230 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 13.3 MBytes 111 Mbits/sec [ 5] 1.00-2.00 sec 14.0 MBytes 117 Mbits/sec [ 5] 2.00-3.00 sec 13.9 MBytes 117 Mbits/sec [ 5] 3.00-4.00 sec 13.9 MBytes 116 Mbits/sec [ 5] 4.00-5.00 sec 13.6 MBytes 114 Mbits/sec [ 5] 5.00-6.00 sec 13.9 MBytes 116 Mbits/sec [ 5] 6.00-7.00 sec 14.0 MBytes 117 Mbits/sec [ 5] 7.00-8.00 sec 13.9 MBytes 116 Mbits/sec [ 5] 8.00-9.00 sec 13.8 MBytes 116 Mbits/sec [ 5] 9.00-10.00 sec 13.8 MBytes 116 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 138 MBytes 116 Mbits/sec sender [ 5] 0.00-10.00 sec 138 MBytes 116 Mbits/sec receiver
hostname:~/Downloads:% iperf3 -c a.b.c.230 Connecting to host a.b.c.230, port 5201 [ 5] local 192.168.1.20 port 51328 connected to a.b.c.230 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 15.5 MBytes 130 Mbits/sec [ 5] 1.00-2.00 sec 15.4 MBytes 129 Mbits/sec [ 5] 2.00-3.00 sec 15.4 MBytes 129 Mbits/sec [ 5] 3.00-4.00 sec 15.2 MBytes 128 Mbits/sec [ 5] 4.00-5.00 sec 15.4 MBytes 129 Mbits/sec [ 5] 5.00-6.00 sec 15.4 MBytes 129 Mbits/sec [ 5] 6.00-7.00 sec 15.4 MBytes 129 Mbits/sec [ 5] 7.00-8.00 sec 15.4 MBytes 129 Mbits/sec [ 5] 8.00-9.00 sec 15.3 MBytes 128 Mbits/sec [ 5] 9.00-10.00 sec 14.8 MBytes 124 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 153 MBytes 129 Mbits/sec sender [ 5] 0.00-10.00 sec 153 MBytes 128 Mbits/sec receiver
hostname:~/Downloads:% iperf3 -c a.b.c.230 Connecting to host a.b.c.230, port 5201 [ 5] local 192.168.1.20 port 51332 connected to a.b.c.230 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 16.3 MBytes 136 Mbits/sec [ 5] 1.00-2.00 sec 16.3 MBytes 137 Mbits/sec [ 5] 2.00-3.00 sec 15.6 MBytes 131 Mbits/sec [ 5] 3.00-4.00 sec 15.6 MBytes 131 Mbits/sec [ 5] 4.00-5.00 sec 15.3 MBytes 128 Mbits/sec [ 5] 5.00-6.00 sec 16.5 MBytes 139 Mbits/sec [ 5] 6.00-7.00 sec 15.5 MBytes 130 Mbits/sec [ 5] 7.00-8.00 sec 15.3 MBytes 129 Mbits/sec [ 5] 8.00-9.00 sec 16.6 MBytes 139 Mbits/sec [ 5] 9.00-10.00 sec 15.3 MBytes 128 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 158 MBytes 133 Mbits/sec sender [ 5] 0.00-10.00 sec 158 MBytes 133 Mbits/sec receiver
_______________________________________________ Cake mailing list [email protected] https://lists.bufferbloat.net/listinfo/cake
