Hi Bill,
Thank you for your good work!
As you mentioned that we've been considering the problems of less
aggressiveness of BIC and CUBIC. We are testing BIC and CUBIC for as
many bottleneck bandwidths (from 1MB - 1GB) and possibly up to 10GB
capacity.
One of the reasons we clamp the "slow start" is that earlier kernel
version sometimes we had a lot of blackouts when we increase the rate
exponentially. It was not because of network but because of local
queues in the kernel.
Now, TCP stack is quite stable and looks handle the burst following
the slow start.
So, your suggestion to change to make BIC and CUBIC use a standard
slow start quite makes sense to me. Actually, we've been consdering to
adopt "limited slow start" or some new derivatives of "slow start".
For any updates of these protocols, we've been testing extensive set
of testing scenarios. Also, we've been comparing the results between
"slow start", "limited slow start" and our clamping for many cases.
Probably, we will update the result to the latest kernel.
Also, "Documents" for congestion control parameters are a good
suggestion and not much work for us, so we will collaborate with
kernel developers to make them available.
Recently, CUBIC is updated to v2.1 to enhance the scalability of the
protocol and you might want to try to use it.
Thanks,
Sangtae
On 5/10/07, Bill Fink <[EMAIL PROTECTED]> wrote:
As a followup, I ran a somewhat interesting test. I increased the
requested socket buffer size to 100 MB, which is sufficient to
overstress the capabilities of the netem delay emulator (which can
handle up to about 8.5 Gbps). This causes some packet loss when
using the standard Reno agressive "slow start".
[EMAIL PROTECTED] ~]# netstat -s | grep -i retrans
0 segments retransmited
[EMAIL PROTECTED] ~]# cat /proc/sys/net/ipv4/tcp_congestion_control
cubic
[EMAIL PROTECTED] ~]# echo 0 > /sys/module/tcp_cubic/parameters/initial_ssthresh
[EMAIL PROTECTED] ~]# cat /sys/module/tcp_cubic/parameters/initial_ssthresh
0
[EMAIL PROTECTED] ~]# nuttcp -T10 -i1 -w100m 192.168.89.15
69.9829 MB / 1.00 sec = 585.1895 Mbps
311.9521 MB / 1.00 sec = 2616.9019 Mbps
0.2332 MB / 1.00 sec = 1.9559 Mbps
37.9907 MB / 1.00 sec = 318.6912 Mbps
702.7856 MB / 1.00 sec = 5895.4640 Mbps
817.0142 MB / 1.00 sec = 6853.7006 Mbps
820.3125 MB / 1.00 sec = 6881.3626 Mbps
820.5625 MB / 1.00 sec = 6883.2601 Mbps
813.0125 MB / 1.00 sec = 6820.2678 Mbps
815.7756 MB / 1.00 sec = 6842.8867 Mbps
5253.2500 MB / 10.07 sec = 4378.0109 Mbps 72 %TX 35 %RX
[EMAIL PROTECTED] ~]# netstat -s | grep -i retrans
464 segments retransmited
464 fast retransmits
Contrast that with the default behavior.
[EMAIL PROTECTED] ~]# echo 100 >
/sys/module/tcp_cubic/parameters/initial_ssthresh
[EMAIL PROTECTED] ~]# cat /sys/module/tcp_cubic/parameters/initial_ssthresh
100
[EMAIL PROTECTED] ~]# nuttcp -T10 -i1 -w100m 192.168.89.15
6.8188 MB / 1.00 sec = 57.1670 Mbps
16.2097 MB / 1.00 sec = 135.9795 Mbps
25.4810 MB / 1.00 sec = 213.7525 Mbps
38.7256 MB / 1.00 sec = 324.8580 Mbps
49.7998 MB / 1.00 sec = 417.7565 Mbps
62.5745 MB / 1.00 sec = 524.9189 Mbps
78.6646 MB / 1.00 sec = 659.8947 Mbps
98.9673 MB / 1.00 sec = 830.2086 Mbps
124.3201 MB / 1.00 sec = 1038.7288 Mbps
156.1584 MB / 1.00 sec = 1309.9730 Mbps
775.2500 MB / 10.64 sec = 611.0181 Mbps 7 %TX 7 %RX
[EMAIL PROTECTED] ~]# netstat -s | grep -i retrans
464 segments retransmited
464 fast retransmits
The standard Reno aggressive "slow start" gets much better overall
performance even in this case, because even though the default cubic
behavior manages to avoid the "congestion" event, its lack of
aggressiveness during the initial slow start period puts it at a
major disadvantage. It would take a long time for the tortoise
in this race to catch up with the hare.
It seems best to ramp up as quickly as possible to any congestion,
using the standard Reno aggressive "slow start" behavior, and then
let the power of cubic take over from there, getting the best of
both worlds.
For completeness here's the same test with bic.
First with the standard Reno aggessive "slow start" behavior:
[EMAIL PROTECTED] ~]# netstat -s | grep -i retrans
464 segments retransmited
464 fast retransmits
[EMAIL PROTECTED] ~]# echo bic > /proc/sys/net/ipv4/tcp_congestion_control
[EMAIL PROTECTED] ~]# cat /proc/sys/net/ipv4/tcp_congestion_control
bic
[EMAIL PROTECTED] ~]# echo 0 > /sys/module/tcp_bic/parameters/initial_ssthresh
[EMAIL PROTECTED] ~]# cat /sys/module/tcp_bic/parameters/initial_ssthresh
0
[EMAIL PROTECTED] ~]# nuttcp -T10 -i1 -w100m 192.168.89.15
69.9829 MB / 1.00 sec = 585.2770 Mbps
302.3921 MB / 1.00 sec = 2536.7045 Mbps
0.0000 MB / 1.00 sec = 0.0000 Mbps
0.7520 MB / 1.00 sec = 6.3079 Mbps
114.1570 MB / 1.00 sec = 957.5914 Mbps
792.9634 MB / 1.00 sec = 6651.5131 Mbps
845.9099 MB / 1.00 sec = 7096.4182 Mbps
865.0825 MB / 1.00 sec = 7257.1575 Mbps
890.4663 MB / 1.00 sec = 7470.0567 Mbps
911.5039 MB / 1.00 sec = 7646.3560 Mbps
4829.9375 MB / 10.05 sec = 4033.0191 Mbps 76 %TX 32 %RX
[EMAIL PROTECTED] ~]# netstat -s | grep -i retrans
1093 segments retransmited
1093 fast retransmits
And then with the default bic behavior:
[EMAIL PROTECTED] ~]# echo 100 > /sys/module/tcp_bic/parameters/initial_ssthresh
[EMAIL PROTECTED] ~]# cat /sys/module/tcp_bic/parameters/initial_ssthresh
100
[EMAIL PROTECTED] ~]# nuttcp -T10 -i1 -w100m 192.168.89.15
9.9548 MB / 1.00 sec = 83.1028 Mbps
47.5439 MB / 1.00 sec = 398.8351 Mbps
107.6147 MB / 1.00 sec = 902.7506 Mbps
183.9038 MB / 1.00 sec = 1542.7124 Mbps
313.4875 MB / 1.00 sec = 2629.7689 Mbps
531.0012 MB / 1.00 sec = 4454.3032 Mbps
841.7866 MB / 1.00 sec = 7061.5098 Mbps
837.5867 MB / 1.00 sec = 7026.4041 Mbps
834.8889 MB / 1.00 sec = 7003.3667 Mbps
4539.6250 MB / 10.00 sec = 3806.5410 Mbps 50 %TX 34 %RX
[EMAIL PROTECTED] ~]# netstat -s | grep -i retrans
1093 segments retransmited
1093 fast retransmits
bic actually does much better than cubic for this scenario, and only
loses out to the standard Reno aggressive "slow start" behavior by a
small amount. Of course in the case of no congestion, it loses out
by a much more significant margin.
This reinforces my belief that it's best to marry the standard Reno
aggressive initial "slow start" behavior with the better performance
of bic or cubic during the subsequent steady state portion of the
TCP session.
I can of course achieve that objective by setting initial_ssthresh
to 0, but perhaps that should be made the default behavior.
-Bill
On Wed, 9 May 2007, I wrote:
> Hi Sangtae,
>
> On Tue, 8 May 2007, SANGTAE HA wrote:
>
> > Hi Bill,
> >
> > At this time, BIC and CUBIC use a less aggressive slow start than
> > other protocols. Because we observed "slow start" is somewhat
> > aggressive and introduced a lot of packet losses. This may be changed
> > to standard "slow start" in later version of BIC and CUBIC, but, at
> > this time, we still using a modified slow start.
>
> "slow start" is somewhat of a misnomer. However, I'd argue in favor
> of using the standard "slow start" for BIC and CUBIC as the default.
> Is the rationale for using a less agressive "slow start" to be gentler
> to certain receivers, which possibly can't handle a rapidly increasing
> initial burst of packets (and the resultant necessary allocation of
> system resources)? Or is it related to encountering actual network
> congestion during the initial "slow start" period, and how well that
> is responded to?
>
> > So, as you observed, this modified slow start behavior may slow for
> > 10G testing. You can alleviate this for your 10G testing by changing
> > BIC and CUBIC to use a standard "slow start" by loading these modules
> > with "initial_ssthresh=0".
>
> I saw the initial_ssthresh parameter, but didn't know what it did or
> even what its units were. I saw the default value was 100 and tried
> increasing it, but I didn't think to try setting it to 0.
>
> [EMAIL PROTECTED] ~]# grep -r initial_ssthresh
/usr/src/kernels/linux-2.6.20.7/Documentation/
> [EMAIL PROTECTED] ~]#
>
> It would be good to have some documentation for these bic and cubic
> parameters similar to the documentation in ip-sysctl.txt for the
> /proc/sys/net/ipv[46]/* variables (I know, I know, I should just
> "use the source").
>
> Is it expected that the cubic "slow start" is that much less agressive
> than the bic "slow start" (from 10 secs to max rate for bic in my test
> to 25 secs to max rate for cubic). This could be considered a performance
> regression since the default TCP was changed from bic to cubic.
>
> In any event, I'm now happy as setting initial_ssthresh to 0 works
> well for me.
>
> [EMAIL PROTECTED] ~]# netstat -s | grep -i retrans
> 0 segments retransmited
>
> [EMAIL PROTECTED] ~]# cat /proc/sys/net/ipv4/tcp_congestion_control
> cubic
>
> [EMAIL PROTECTED] ~]# cat /sys/module/tcp_cubic/parameters/initial_ssthresh
> 0
>
> [EMAIL PROTECTED] ~]# nuttcp -T10 -i1 -w60m 192.168.89.15
> 69.9829 MB / 1.00 sec = 584.2065 Mbps
> 843.1467 MB / 1.00 sec = 7072.9052 Mbps
> 844.3655 MB / 1.00 sec = 7082.6544 Mbps
> 842.2671 MB / 1.00 sec = 7065.7169 Mbps
> 839.9204 MB / 1.00 sec = 7045.8335 Mbps
> 840.1780 MB / 1.00 sec = 7048.3114 Mbps
> 834.1475 MB / 1.00 sec = 6997.4270 Mbps
> 835.5972 MB / 1.00 sec = 7009.3148 Mbps
> 835.8152 MB / 1.00 sec = 7011.7537 Mbps
> 830.9333 MB / 1.00 sec = 6969.9281 Mbps
>
> 7617.1875 MB / 10.01 sec = 6386.2622 Mbps 90 %TX 46 %RX
>
> [EMAIL PROTECTED] ~]# netstat -s | grep -i retrans
> 0 segments retransmited
>
> -Thanks a lot!
>
> -Bill
>
>
>
> > Regards,
> > Sangtae
> >
> >
> > On 5/6/07, Bill Fink <[EMAIL PROTECTED]> wrote:
> > > The initial TCP slow start on 2.6.20.7 cubic (and to a lesser
> > > extent bic) seems to be way too slow. With an ~80 ms RTT, this
> > > is what cubic delivers (thirty second test with one second interval
> > > reporting and specifying a socket buffer size of 60 MB):
> > >
> > > [EMAIL PROTECTED] ~]# netstat -s | grep -i retrans
> > > 0 segments retransmited
> > >
> > > [EMAIL PROTECTED] ~]# cat /proc/sys/net/ipv4/tcp_congestion_control
> > > cubic
> > >
> > > [EMAIL PROTECTED] ~]# nuttcp -T30 -i1 -w60m 192.168.89.15
> > > 6.8188 MB / 1.00 sec = 57.0365 Mbps
> > > 16.2097 MB / 1.00 sec = 135.9824 Mbps
> > > 25.4553 MB / 1.00 sec = 213.5420 Mbps
> > > 35.5127 MB / 1.00 sec = 297.9119 Mbps
> > > 43.0066 MB / 1.00 sec = 360.7770 Mbps
> > > 50.3210 MB / 1.00 sec = 422.1370 Mbps
> > > 59.0796 MB / 1.00 sec = 495.6124 Mbps
> > > 69.1284 MB / 1.00 sec = 579.9098 Mbps
> > > 76.6479 MB / 1.00 sec = 642.9130 Mbps
> > > 90.6189 MB / 1.00 sec = 760.2835 Mbps
> > > 109.4348 MB / 1.00 sec = 918.0361 Mbps
> > > 128.3105 MB / 1.00 sec = 1076.3813 Mbps
> > > 150.4932 MB / 1.00 sec = 1262.4686 Mbps
> > > 175.9229 MB / 1.00 sec = 1475.7965 Mbps
> > > 205.9412 MB / 1.00 sec = 1727.6150 Mbps
> > > 240.8130 MB / 1.00 sec = 2020.1504 Mbps
> > > 282.1790 MB / 1.00 sec = 2367.1644 Mbps
> > > 318.1841 MB / 1.00 sec = 2669.1349 Mbps
> > > 372.6814 MB / 1.00 sec = 3126.1687 Mbps
> > > 440.8411 MB / 1.00 sec = 3698.5200 Mbps
> > > 524.8633 MB / 1.00 sec = 4403.0220 Mbps
> > > 614.3542 MB / 1.00 sec = 5153.7367 Mbps
> > > 718.9917 MB / 1.00 sec = 6031.5386 Mbps
> > > 829.0474 MB / 1.00 sec = 6954.6438 Mbps
> > > 867.3289 MB / 1.00 sec = 7275.9510 Mbps
> > > 865.7759 MB / 1.00 sec = 7262.9813 Mbps
> > > 864.4795 MB / 1.00 sec = 7251.7071 Mbps
> > > 864.5425 MB / 1.00 sec = 7252.8519 Mbps
> > > 867.3372 MB / 1.00 sec = 7246.9232 Mbps
> > >
> > > 10773.6875 MB / 30.00 sec = 3012.3936 Mbps 38 %TX 25 %RX
> > >
> > > [EMAIL PROTECTED] ~]# netstat -s | grep -i retrans
> > > 0 segments retransmited
> > >
> > > It takes 25 seconds for cubic TCP to reach its maximal rate.
> > > Note that there were no TCP retransmissions (no congestion
> > > experienced).
> > >
> > > Now with bic (only 20 second test this time):
> > >
> > > [EMAIL PROTECTED] ~]# echo bic > /proc/sys/net/ipv4/tcp_congestion_control
> > > [EMAIL PROTECTED] ~]# cat /proc/sys/net/ipv4/tcp_congestion_control
> > > bic
> > >
> > > [EMAIL PROTECTED] ~]# nuttcp -T20 -i1 -w60m 192.168.89.15
> > > 9.9548 MB / 1.00 sec = 83.1497 Mbps
> > > 47.2021 MB / 1.00 sec = 395.9762 Mbps
> > > 92.4304 MB / 1.00 sec = 775.3889 Mbps
> > > 134.3774 MB / 1.00 sec = 1127.2758 Mbps
> > > 194.3286 MB / 1.00 sec = 1630.1987 Mbps
> > > 280.0598 MB / 1.00 sec = 2349.3613 Mbps
> > > 404.3201 MB / 1.00 sec = 3391.8250 Mbps
> > > 559.1594 MB / 1.00 sec = 4690.6677 Mbps
> > > 792.7100 MB / 1.00 sec = 6650.0257 Mbps
> > > 857.2241 MB / 1.00 sec = 7190.6942 Mbps
> > > 852.6912 MB / 1.00 sec = 7153.3283 Mbps
> > > 852.6968 MB / 1.00 sec = 7153.2538 Mbps
> > > 851.3162 MB / 1.00 sec = 7141.7575 Mbps
> > > 851.4927 MB / 1.00 sec = 7143.0240 Mbps
> > > 850.8782 MB / 1.00 sec = 7137.8762 Mbps
> > > 852.7119 MB / 1.00 sec = 7153.2949 Mbps
> > > 852.3879 MB / 1.00 sec = 7150.2982 Mbps
> > > 850.2163 MB / 1.00 sec = 7132.5165 Mbps
> > > 849.8340 MB / 1.00 sec = 7129.0026 Mbps
> > >
> > > 11882.7500 MB / 20.00 sec = 4984.0068 Mbps 67 %TX 41 %RX
> > >
> > > [EMAIL PROTECTED] ~]# netstat -s | grep -i retrans
> > > 0 segments retransmited
> > >
> > > bic does better but still takes 10 seconds to achieve its maximal
> > > rate.
> > >
> > > Surprisingly venerable reno does the best (only a 10 second test):
> > >
> > > [EMAIL PROTECTED] ~]# echo reno >
/proc/sys/net/ipv4/tcp_congestion_control
> > > [EMAIL PROTECTED] ~]# cat /proc/sys/net/ipv4/tcp_congestion_control
> > > reno
> > >
> > > [EMAIL PROTECTED] ~]# nuttcp -T10 -i1 -w60m 192.168.89.15
> > > 69.9829 MB / 1.01 sec = 583.5822 Mbps
> > > 844.3870 MB / 1.00 sec = 7083.2808 Mbps
> > > 862.7568 MB / 1.00 sec = 7237.7342 Mbps
> > > 859.5725 MB / 1.00 sec = 7210.8981 Mbps
> > > 860.1365 MB / 1.00 sec = 7215.4487 Mbps
> > > 865.3940 MB / 1.00 sec = 7259.8434 Mbps
> > > 863.9678 MB / 1.00 sec = 7247.4942 Mbps
> > > 864.7493 MB / 1.00 sec = 7254.4634 Mbps
> > > 864.6660 MB / 1.00 sec = 7253.5183 Mbps
> > >
> > > 7816.9375 MB / 10.00 sec = 6554.4883 Mbps 90 %TX 53 %RX
> > >
> > > [EMAIL PROTECTED] ~]# netstat -s | grep -i retrans
> > > 0 segments retransmited
> > >
> > > reno achieves its maximal rate in about 2 seconds. This is what I
> > > would expect from the exponential increase during TCP's initial
> > > slow start. To achieve 10 Gbps on an 80 ms RTT with 9000 byte
> > > jumbo frame packets would require:
> > >
> > > [EMAIL PROTECTED] ~]# bc -l
> > > scale=10
> > > 10^10*0.080/9000/8
> > > 11111.1111111111
> > >
> > > So 11111 packets would have to be in flight during one RTT.
> > > It should take log2(11111)+1 round trips to achieve 10 Gbps
> > > (note bc's l() function is logE);
> > >
> > > [EMAIL PROTECTED] ~]# bc -l
> > > scale=10
> > > l(11111)/l(2)+1
> > > 14.4397010470
> > >
> > > And 15 round trips at 80 ms each gives a total time of:
> > >
> > > [EMAIL PROTECTED] ~]# bc -l
> > > scale=10
> > > 15*0.080
> > > 1.200
> > >
> > > So if there is no packet loss (which there wasn't), it should only
> > > take about 1.2 seconds to achieve 10 Gbps. Only TCP reno is in
> > > this ballpark range.
> > >
> > > Now it's quite possible there's something basic I don't understand,
> > > such as some /proc/sys/net/ipv4/tcp_* or /sys/module/tcp_*/parameters/*
> > > parameter I've overlooked, in which case feel free to just refer me
> > > to any suitable documentation.
> > >
> > > I also checked the Changelog for 2.6.20.{8,9,10,11} to see if there
> > > might be any relevant recent bug fixes, but the only thing that seemed
> > > even remotely related was the 2.6.20.11 bug fix for the tcp_mem setting.
> > > Although this did affect me, I manually adjusted the tcp_mem settings
> > > before running these tests.
> > >
> > > [EMAIL PROTECTED] ~]# cat /proc/sys/net/ipv4/tcp_mem
393216 524288 786432
> > >
> > > The test setup was:
> > >
> > > +-------+ +-------+ +-------+
> > > | |eth2 eth2| |eth3 eth2| |
> > > | lang2 |-----10-GigE-----| lang1 |-----10-GigE-----| lang3 |
> > > | | | | | |
> > > +-------+ +-------+ +-------+
> > > 192.168.88.14 192.168.88.13/192.168.89.13 192.168.89.15
> > >
> > > All three systems are dual 2.8 GHz AMD Opteron Processor 254 systems
> > > with 4 GB memory and all running the 2.6.20.7 kernel. All the NICs
> > > are Myricom PCI-E 10-GigE NICs.
> > >
> > > The 80 ms delay was introduced by applying netem to lang1's eth3
> > > interface:
> > >
> > > [EMAIL PROTECTED] ~]# tc qdisc add dev eth3 root netem delay 80ms limit
20000
> > > [EMAIL PROTECTED] ~]# tc qdisc show
> > > qdisc pfifo_fast 0: dev eth2 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
1 1 1 1 1
> > > qdisc netem 8022: dev eth3 limit 20000 delay 80.0ms reorder 100%
> > >
> > > Experimentation determined that netem running on lang1 could handle
> > > about 8-8.5 Gbps without dropping packets.
> > >
> > > 8.5 Gbps UDP test:
> > >
> > > [EMAIL PROTECTED] ~]# nuttcp -u -Ri8.5g -w20m 192.168.89.15
> > > 10136.4844 MB / 10.01 sec = 8497.8205 Mbps 100 %TX 56 %RX 0 / 1297470
drop/pkt 0.00 %loss
> > >
> > > Increasing the rate to 9 Gbps would give some loss:
> > >
> > > [EMAIL PROTECTED] ~]# nuttcp -u -Ri9g -w20m 192.168.89.15
> > > 10219.1719 MB / 10.01 sec = 8560.2455 Mbps 100 %TX 58 %RX 65500 /
1373554 drop/pkt 4.77 %loss
> > >
> > > Based on this, the specification of a 60 MB TCP socket buffer size was
> > > used during the TCP tests to avoid overstressing the lang1 netem delay
> > > emulator (to avoid dropping any packets).
> > >
> > > Simple ping through the lang1 netem delay emulator:
> > >
> > > [EMAIL PROTECTED] ~]# ping -c 5 192.168.89.15
> > > PING 192.168.89.15 (192.168.89.15) 56(84) bytes of data.
> > > 64 bytes from 192.168.89.15: icmp_seq=1 ttl=63 time=80.4 ms
> > > 64 bytes from 192.168.89.15: icmp_seq=2 ttl=63 time=82.1 ms
> > > 64 bytes from 192.168.89.15: icmp_seq=3 ttl=63 time=82.1 ms
> > > 64 bytes from 192.168.89.15: icmp_seq=4 ttl=63 time=82.1 ms
> > > 64 bytes from 192.168.89.15: icmp_seq=5 ttl=63 time=82.1 ms
> > >
> > > --- 192.168.89.15 ping statistics ---
> > > 5 packets transmitted, 5 received, 0% packet loss, time 4014ms
> > > rtt min/avg/max/mdev = 80.453/81.804/82.173/0.722 ms
> > >
> > > And a bidirectional traceroute (using the "nuttcp -xt" option):
> > >
> > > [EMAIL PROTECTED] ~]# nuttcp -xt 192.168.89.15
> > > traceroute to 192.168.89.15 (192.168.89.15), 30 hops max, 40 byte packets
> > > 1 192.168.88.13 (192.168.88.13) 0.141 ms 0.125 ms 0.125 ms
> > > 2 192.168.89.15 (192.168.89.15) 82.112 ms 82.039 ms 82.541 ms
> > >
> > > traceroute to 192.168.88.14 (192.168.88.14), 30 hops max, 40 byte packets
> > > 1 192.168.89.13 (192.168.89.13) 81.101 ms 83.001 ms 82.999 ms
> > > 2 192.168.88.14 (192.168.88.14) 83.005 ms 82.985 ms 82.978 ms
> > >
> > > So is this a real bug in cubic (and bic), or do I just not understand
> > > something basic.
> > >
> > > -Bill
-
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
