Sebastian Moeller wrote:
Hi Andy,
On Apr 25, 2017, at 14:58, Andy Furniss <[email protected]>
wrote:
Dendari Marini wrote:
Also I have done some more testing, I was able to limit Steam
connections just to one thanks to some console commands
("@cMaxContentServersToRequest" and
"@cCSClientMaxNumSocketsPerHost") and while the situation
improved (no more packet loss, latency variation within 10ms, but
still seeing ping spikes of ~50ms) it's still not what I'd
consider ideal, which would be like with any other download. So
my guess is there's something else going on other than just the
multiple connections, which are definitely big part of the
problem but not the only thing. Anyway these are my current
settings for Cake and I've been using them for the last four days
without issues: *qdisc cake 8005: root refcnt 2 bandwidth 950Kbit
diffserv3 triple-isolate nat wash rtt 100.0ms atm overhead 40
via-ethernet* *qdisc cake 8006: root refcnt 2 bandwidth 17500Kbit
diffserv3 triple-isolate nat wash ingress rtt 100.0ms atm
overhead 40 via-ethernet*
I still think that once via-ethernet appears you really need the
raw parameter.
Why? As far as I can tell the default mode is to adjust for the
kernel auto-added overhead (which seems to be 14 bytes for pakets
headed for an ethernet device and zero for packets headed for a ppp
device). If he adds raw, the kernel willl make its additions
silently…
I shape on pppoe - though it's ptm. My overhead is 34, it seems as soon
as you add overhead as a parameter via-ethernet appears and in the case
of pppoe apparently does the wrong thing. Recalling my now closed issue
about overheads on github, cake beleives what the kernel tells it about
overheads and this seems to 22. In fact cake does not see the traffic as
+22 it sees ip length.
If I don't use raw below my shaper fails. Also note that when using raw
the overhead gets increased by 22 from that specified.
tc qdisc add dev ppp0 handle 1:0 root cake bandwidth 19690kbit raw
overhead 34 diffserv4 dual-srchost nat rtt 200ms
asr[/home/andy]# tc -s qdisc ls dev ppp0
qdisc cake 1: root refcnt 2 bandwidth 19690Kbit diffserv4 dual-srchost
nat rtt 200.0ms noatm overhead 56 via-ethernet
Sent 1311201393 bytes 5277696 pkt (dropped 128, overlimits 1606291
requeues 0)
backlog 0b 0p requeues 0
memory used: 115328b of 4Mb
capacity estimate: 19690Kbit
Bulk Best Effort Video Voice
thresh 1230Kbit 19690Kbit 9845Kbit 4922Kbit
target 14.8ms 10.0ms 10.0ms 10.0ms
interval 204.8ms 200.0ms 200.0ms 200.0ms
pk_delay 6us 25us 2us 209us
av_delay 3us 2us 2us 31us
sp_delay 1us 1us 1us 3us
pkts 927088 4133045 118552 99139
bytes 974738943 320755191 3326412 12572807
way_inds 7798 33031 0 2
way_miss 45787 84501 40 8687
way_cols 0 0 0 0
drops 0 128 0 0
marks 0 0 0 0
sp_flows 1 0 0 0
bk_flows 1 0 0 0
un_flows 0 0 0 0
max_len 1500 1500 84 1428
Independent of that matter one needs to specify different overheads
on ethN and pppeN interfaces simply as the packets on the pppoN
interface have not yet been encpsulated and are 8 byte smaller than
on the respective ethN interface. I am by now thoroghly confused, so
I might have missed the subtleties in which either cake or my
understanding is wrong.
On egress ppp it likely subtracts 22 bytes on ifb that is attached
to ingress ppp 14 bytes.
My understanding is again that on pppoe devices the kernel adds zero
bytes auto matically and attaching the ifb does not seem to change
that?
The packet size is ip length as seen by cake on ifb redirected from
pppoe - but this time it seems the difference is 14 not 22 ...
tc qdisc add dev ifb0 handle 1:0 root cake bandwidth 60mbit raw overhead
34 diffserv4 nat dual-dsthost
asr[/home/andy]# tc -s qdisc ls dev ifb0
qdisc cake 1: root refcnt 2 bandwidth 60Mbit diffserv4 dual-dsthost nat
rtt 100.0ms noatm overhead 48 via-ethernet
Sent 14315263754 bytes 12978554 pkt (dropped 51315, overlimits
18015371 requeues 0)
backlog 0b 0p requeues 0
memory used: 1006656b of 4Mb
capacity estimate: 60Mbit
Bulk Best Effort Video Voice
thresh 3750Kbit 60Mbit 30Mbit 15Mbit
target 5.0ms 5.0ms 5.0ms 5.0ms
interval 100.0ms 100.0ms 100.0ms 100.0ms
pk_delay 243us 1.2ms 42us 82us
av_delay 18us 821us 7us 8us
sp_delay 2us 3us 4us 3us
pkts 1942903 10462709 272777 351480
bytes 130598812 14085111253 8318580 168132633
way_inds 220 46306 0 0
way_miss 68580 114792 188 84890
way_cols 0 0 0 0
drops 0 51308 0 7
marks 0 0 0 0
sp_flows 0 0 1 0
bk_flows 0 0 0 0
un_flows 0 0 0 0
max_len 1500 1500 1500 1441
If you shape on real eth (eg. lan side) or ifb hooked from real eth
then you shouldn't use raw.
Ah, I thought that the non-raw mode inquires at the kernel what
overhead it intends to account for silently and simply undoes that
probably by changing the overhead passed to the kernel so that the
sum of both match what the user requested?
I think that things change when you add overhead XX (though I may need
to retest that on a normal eth), but the reason I thought not to use raw
on lan side was that cake really will see the packets as +14 so you want
via-ethernet to subtract 14.
Thinking about it, mostly you will luck into it not making any
difference, so the test I suggested earlier won't work. The reason
being that whether the overhead is 40 or 40 - 22 an MTU sized
packet or an empty ack/single sack will end up the same due to atm.
sack > 1 will be wrong as of course will be other traffic of
certain sizes, so a carefully crafted test would be needed to show
the issue.
Talking of acks/sacks 17500 vs 950kbit doesn't even have room for
one sack per packet, though mostly there will be < 1 ack per
packet.
Intersting, looking into this I learned that SACK can cause up to 40
bytes of TCP options added to the packet...
Yea, not nice with overhead of 40 as 3 cells for multi sacks - but even
at 2 cells Dendari doesn't have enough bandwidth for 1 ack per packet
during recovery.
_______________________________________________
Cake mailing list
[email protected]
https://lists.bufferbloat.net/listinfo/cake
