>Synopsis: tcp.ts_recent set to 0 after receiving retransmission
>Category: kernel
>Environment:
System : OpenBSD 6.1
Details : OpenBSD 6.1-current (GENERIC.MP) #67: Mon Apr 17 15:22:46
MDT 2017
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
Architecture: OpenBSD.amd64
Machine : amd64
>Description:
On a congested TCP path between OpenBSD (receiver of data on the
connection) and illumos/OmniOS (sender), lost fast retransmission
segments cause the retransmission timer of the sender to keep growing
until it reaches maximum without ever going down. That's an illumos bug,
but the trigger is the ACK segments sent by OpenBSD always have TSecr=0
after some retransmissions are received; partial packet capture on the
receiver below:
7183 0.875078 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
7184 0.876087 sender → receiver 1090 [TCP Previous segment not
captured] 9879 → 34480 [ACK] Seq=4512769 Ack=1 Win=128872 Len=1024
TSval=2677655334 TSecr=1179635542
7185 0.876089 sender → receiver 1090 9879 → 34480 [ACK]
Seq=4513793 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
7186 0.876090 sender → receiver 1090 9879 → 34480 [ACK]
Seq=4514817 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
7187 0.876090 sender → receiver 1090 9879 → 34480 [ACK]
Seq=4515841 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
7188 0.876091 sender → receiver 1090 9879 → 34480 [ACK]
Seq=4516865 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
7189 0.876093 sender → receiver 1090 9879 → 34480 [ACK]
Seq=4517889 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
7190 0.876102 receiver → sender 66 [TCP Dup ACK 7183#1] 34480
→ 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
7191 0.876107 receiver → sender 66 [TCP Dup ACK 7183#2] 34480
→ 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
7192 0.876110 receiver → sender 66 [TCP Dup ACK 7183#3] 34480
→ 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
7193 0.876114 receiver → sender 66 [TCP Dup ACK 7183#4] 34480
→ 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
7194 0.876117 receiver → sender 66 [TCP Dup ACK 7183#5] 34480
→ 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
7195 0.876120 receiver → sender 66 [TCP Dup ACK 7183#6] 34480
→ 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
7196 0.876198 sender → receiver 1090 9879 → 34480 [ACK]
Seq=4518913 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
7197 0.876202 receiver → sender 66 [TCP Dup ACK 7183#7] 34480
→ 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
7198 0.876326 sender → receiver 1090 9879 → 34480 [ACK]
Seq=4519937 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
7199 0.876327 sender → receiver 1090 9879 → 34480 [ACK]
Seq=4520961 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
7200 0.876331 receiver → sender 66 [TCP Dup ACK 7183#8] 34480
→ 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
7201 0.876335 receiver → sender 66 [TCP Dup ACK 7183#9] 34480
→ 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
7202 0.877872 sender → receiver 1090 [TCP Previous segment not
captured] 9879 → 34480 [ACK] Seq=4529153 Ack=1 Win=128872 Len=1024
TSval=2677655334 TSecr=1179635542
7203 0.877876 receiver → sender 66 [TCP Dup ACK 7183#10] 34480
→ 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
7204 0.878002 sender → receiver 1090 9879 → 34480 [ACK]
Seq=4530177 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
7205 0.878003 sender → receiver 1090 [TCP Previous segment not
captured] 9879 → 34480 [ACK] Seq=4536321 Ack=1 Win=128872 Len=1024
TSval=2677655334 TSecr=1179635542
7206 0.878007 receiver → sender 66 [TCP Dup ACK 7183#11] 34480
→ 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
7207 0.878011 receiver → sender 66 [TCP Dup ACK 7183#12] 34480
→ 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
7208 1.296048 sender → receiver 1514 [TCP Retransmission] 9879
→ 34480 [ACK] Seq=4510721 Ack=1 Win=128872 Len=1448 TSval=2677655376
TSecr=1179635542
7209 1.296064 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4512169 Win=31808 Len=0 TSval=1179635543 TSecr=2677655376
7210 1.299226 sender → receiver 1514 [TCP Retransmission] 9879
→ 34480 [ACK] Seq=4512169 Ack=1 Win=128872 Len=1448 TSval=2677655376
TSecr=1179635543
7211 1.299228 sender → receiver 1514 [TCP Retransmission] 9879
→ 34480 [ACK] Seq=4513617 Ack=1 Win=128872 Len=1448 TSval=2677655376
TSecr=1179635543
7212 1.299275 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4521985 Win=23488 Len=0 TSval=1179635543 TSecr=2677655376
7213 1.299282 receiver → sender 66 [TCP Dup ACK 7212#1] 34480
→ 9879 [ACK] Seq=1 Ack=4521985 Win=23488 Len=0 TSval=1179635543 TSecr=0
7214 1.299315 receiver → sender 66 [TCP Window Update] 34480 →
9879 [ACK] Seq=1 Ack=4521985 Win=33280 Len=0 TSval=1179635543 TSecr=0
7215 1.302160 sender → receiver 1514 [TCP Retransmission] 9879
→ 34480 [ACK] Seq=4521985 Ack=1 Win=128872 Len=1448 TSval=2677655376
TSecr=1179635543
7216 1.302205 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4523433 Win=31808 Len=0 TSval=1179635543 TSecr=0
7217 1.302365 sender → receiver 1514 [TCP Retransmission] 9879
→ 34480 [ACK] Seq=4523433 Ack=1 Win=128872 Len=1448 TSval=2677655376
TSecr=1179635543
7218 1.302378 sender → receiver 1514 [TCP Retransmission] 9879
→ 34480 [ACK] Seq=4524881 Ack=1 Win=128872 Len=1448 TSval=2677655376
TSecr=1179635543
7219 1.302399 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4524881 Win=31808 Len=0 TSval=1179635543 TSecr=0
7220 1.302413 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4526329 Win=30400 Len=0 TSval=1179635543 TSecr=0
7221 1.302423 receiver → sender 66 [TCP Window Update] 34480 →
9879 [ACK] Seq=1 Ack=4526329 Win=33280 Len=0 TSval=1179635543 TSecr=0
7222 1.305086 sender → receiver 1514 [TCP Retransmission] 9879
→ 34480 [ACK] Seq=4526329 Ack=1 Win=128872 Len=1448 TSval=2677655376
TSecr=1179635543
7223 1.305095 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4527777 Win=31808 Len=0 TSval=1179635543 TSecr=0
7224 1.305220 sender → receiver 1514 [TCP Retransmission] 9879
→ 34480 [ACK] Seq=4527777 Ack=1 Win=128872 Len=1448 TSval=2677655376
TSecr=1179635543
7225 1.305228 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4531201 Win=29824 Len=0 TSval=1179635543 TSecr=0
7226 1.305248 receiver → sender 66 [TCP Window Update] 34480 →
9879 [ACK] Seq=1 Ack=4531201 Win=33280 Len=0 TSval=1179635543 TSecr=0
7227 1.305480 sender → receiver 1514 [TCP Spurious
Retransmission] 9879 → 34480 [ACK] Seq=4529225 Ack=1 Win=128872 Len=1448
TSval=2677655376 TSecr=1179635543
7228 1.305492 receiver → sender 66 [TCP Dup ACK 7225#1] 34480
→ 9879 [ACK] Seq=1 Ack=4531201 Win=33280 Len=0 TSval=1179635543 TSecr=0
7229 1.305605 sender → receiver 1514 [TCP Retransmission] 9879
→ 34480 [ACK] Seq=4530673 Ack=1 Win=128872 Len=1448 TSval=2677655376
TSecr=1179635543
7230 1.305606 sender → receiver 1514 [TCP Retransmission] 9879
→ 34480 [ACK] Seq=4532121 Ack=1 Win=128872 Len=1448 TSval=2677655376
TSecr=1179635543
7231 1.305607 sender → receiver 1514 [TCP Retransmission] 9879
→ 34480 [ACK] Seq=4533569 Ack=1 Win=128872 Len=1448 TSval=2677655376
TSecr=1179635543
7232 1.305616 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4532121 Win=32384 Len=0 TSval=1179635543 TSecr=0
7233 1.305620 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4533569 Win=30912 Len=0 TSval=1179635543 TSecr=0
7234 1.305624 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4535017 Win=29440 Len=0 TSval=1179635543 TSecr=0
7235 1.305638 receiver → sender 66 [TCP Window Update] 34480 →
9879 [ACK] Seq=1 Ack=4535017 Win=33280 Len=0 TSval=1179635543 TSecr=0
7236 1.308405 sender → receiver 1514 [TCP Retransmission] 9879
→ 34480 [ACK] Seq=4535017 Ack=1 Win=128872 Len=1448 TSval=2677655377
TSecr=1179635543
7237 1.308407 sender → receiver 1514 [TCP Retransmission] 9879
→ 34480 [ACK] Seq=4536465 Ack=1 Win=128872 Len=1448 TSval=2677655377
TSecr=1179635543
7238 1.308408 sender → receiver 1514 9879 → 34480 [ACK]
Seq=4537913 Ack=1 Win=128872 Len=1448 TSval=2677655377 TSecr=1179635543
7239 1.308409 sender → receiver 1514 9879 → 34480 [ACK]
Seq=4539361 Ack=1 Win=128872 Len=1448 TSval=2677655377 TSecr=1179635543
7240 1.308416 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4537345 Win=30976 Len=0 TSval=1179635543 TSecr=0
7241 1.308421 receiver → sender 66 [TCP ACKed unseen segment]
34480 → 9879 [ACK] Seq=1 Ack=4539361 Win=28928 Len=0 TSval=1179635543 TSecr=0
7242 1.308432 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4540809 Win=33280 Len=0 TSval=1179635543 TSecr=0
7243 1.308665 sender → receiver 1514 9879 → 34480 [ACK]
Seq=4540809 Ack=1 Win=128872 Len=1448 TSval=2677655377 TSecr=1179635543
7244 1.308666 sender → receiver 594 9879 → 34480 [ACK]
Seq=4542257 Ack=1 Win=128872 Len=528 TSval=2677655377 TSecr=1179635543
7245 1.308667 sender → receiver 770 9879 → 34480 [ACK]
Seq=4542785 Ack=1 Win=128872 Len=704 TSval=2677655377 TSecr=1179635543
7246 1.308668 sender → receiver 1090 9879 → 34480 [ACK]
Seq=4543489 Ack=1 Win=128872 Len=1024 TSval=2677655377 TSecr=1179635543
7247 1.308670 sender → receiver 1090 9879 → 34480 [ACK]
Seq=4544513 Ack=1 Win=128872 Len=1024 TSval=2677655377 TSecr=1179635543
7248 1.308676 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4542785 Win=31296 Len=0 TSval=1179635543 TSecr=0
7249 1.308680 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4544513 Win=29568 Len=0 TSval=1179635543 TSecr=0
7250 1.308689 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4545537 Win=33280 Len=0 TSval=1179635543 TSecr=0
7251 1.308790 sender → receiver 1090 9879 → 34480 [ACK]
Seq=4545537 Ack=1 Win=128872 Len=1024 TSval=2677655377 TSecr=1179635543
7252 1.308792 sender → receiver 1090 9879 → 34480 [ACK]
Seq=4546561 Ack=1 Win=128872 Len=1024 TSval=2677655377 TSecr=1179635543
7253 1.308793 sender → receiver 1090 9879 → 34480 [ACK]
Seq=4547585 Ack=1 Win=128872 Len=1024 TSval=2677655377 TSecr=1179635543
7254 1.308794 sender → receiver 1090 9879 → 34480 [ACK]
Seq=4548609 Ack=1 Win=128872 Len=1024 TSval=2677655377 TSecr=1179635543
7255 1.308800 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4547585 Win=31232 Len=0 TSval=1179635543 TSecr=0
7256 1.308805 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4549633 Win=29184 Len=0 TSval=1179635543 TSecr=0
7257 1.308812 receiver → sender 66 [TCP Window Update] 34480 →
9879 [ACK] Seq=1 Ack=4549633 Win=33280 Len=0 TSval=1179635543 TSecr=0
7258 1.311403 sender → receiver 1090 9879 → 34480 [ACK]
Seq=4549633 Ack=1 Win=128872 Len=1024 TSval=2677655377 TSecr=1179635543
7259 1.311404 sender → receiver 1090 9879 → 34480 [ACK]
Seq=4550657 Ack=1 Win=128872 Len=1024 TSval=2677655377 TSecr=1179635543
7260 1.311405 sender → receiver 1090 9879 → 34480 [ACK]
Seq=4551681 Ack=1 Win=128872 Len=1024 TSval=2677655377 TSecr=1179635543
7261 1.311411 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4551681 Win=31232 Len=0 TSval=1179635543 TSecr=0
7262 1.311420 receiver → sender 66 34480 → 9879 [ACK] Seq=1
Ack=4552705 Win=33280 Len=0 TSval=1179635543 TSecr=0
The first segment with TSecr=0 is 7216, but even after loss recovery all
further ACKs sent by the receiver have TSecr=0. I could not reproduce
TSecr staying at 0 on a connection between Linux and OpenBSD (though I
could see TSecr=0 ACKs there intermittently), so I'm not sure what
exactly it is that keeps ts_recent at zero (but the diff below makes it
not be set to zero at all).
>How-To-Repeat:
- tcp sender on illumos (OmniOS): nc -l 9879 </dev/null
- for more reliable repro, "disable" fast retransmissions on the sender:
ndd -set /dev/tcp tcp_dupack_fast_retransmit 1000
- introduce some packet loss on the connection path
- tcp receiver on OpenBSD: nc omnios 9879 | pv >/dev/null
- observe pauses of 0B/s transferred that grow in length during
connection lifetime
>Fix:
Diff below. It removes the TSecr=0 ACKs from Linux->OpenBSD connections
as well as fixes the OmniOS->OpenBSD stalls.
https://tools.ietf.org/html/rfc7323#section-4.3 states in the algorithm
description (2) that if SEG.TSval is not copied to TS.Recent, then it
should be ignored (ie. not set TS.Recent to 0); in fact RFC 1323 says
the same (but the algorithm changed a little). I don't have a copy of
the book referenced in the comment, but AFAICT RFC7323's algorithm makes
the mentioned fix unnecessary; see (d) in
https://tools.ietf.org/html/rfc7323#appendix-H
Index: tcp_input.c
===================================================================
RCS file: /cvs/src/sys/netinet/tcp_input.c,v
retrieving revision 1.341
diff -u -p -r1.341 tcp_input.c
--- tcp_input.c 19 Apr 2017 15:21:54 -0000 1.341
+++ tcp_input.c 20 Apr 2017 13:35:10 -0000
@@ -1374,15 +1374,10 @@ trimthenstep6:
/*
* If last ACK falls within this segment's sequence numbers,
* record its timestamp if it's more recent.
- * Cf fix from Braden, see Stevens p. 870
*/
if (opti.ts_present && TSTMP_GEQ(opti.ts_val, tp->ts_recent) &&
SEQ_LEQ(th->th_seq, tp->last_ack_sent)) {
- if (SEQ_LEQ(tp->last_ack_sent, th->th_seq + tlen +
- ((tiflags & (TH_SYN|TH_FIN)) != 0)))
- tp->ts_recent = opti.ts_val;
- else
- tp->ts_recent = 0;
+ tp->ts_recent = opti.ts_val;
tp->ts_recent_age = tcp_now;
}
--
Lauri Tirkkonen | lotheac @ IRCnet
