Hi Eric, Thanks a lot for the response, and sorry about the 3-times-email, was not sure whether majordomo accepted my subscription or not and hence the retx :)
So if a sequence happens as below 1. Client sends the first SYN with some TSval X 2. Server responds SYN-ACK with TSecr X 3. The SYN-ACK just gets dropped on the way back to the client 4. Client sends a SYN retry after N seconds with a new TSval Y 5. Server responds SYN-ACK with TSecr X And if there is some firewall in between in the amazon environment where the firewall expects to see the SYN-ACK with TSecr Y, then I guess it matches the problem I saw ? In my case clearly the SYN-ACKs never reached the client no matter how many times they were retransmitted. So this would mean that if there is such a wierd firewall in between, then one missing SYN-ACK can cause the tcp connection to eventually timeout ! This of course is just guesswork based on what we saw as the behaviour from tcpdump on server and client side when the timeouts were happening. Does this sound like a possibility - has anyone come across "interesting" firewalls like this ? And about your question: "Are you establishing many active sessions per minute to this particular target ?" - in my particular case there were not more than three linux client boxes sitting behind a NAT (sharing the same public IP) and talking to the same server. And each client box opens a tcp socket once in 30 seconds to the server. So the number of active sessions per 30 seconds is not more than 3 sessions. Now if the NAT device had some bug and ended up NAT-ing more than one client SYN packet to the same source port, then of course thats another theory for why this TSecr/TSval mismatch can happen (other than the SYN-ACK drop theory above). Rgds, Gopa. On Tue, May 26, 2015 at 11:23 PM, Eric Dumazet <eric.duma...@gmail.com> wrote: > On Tue, 2015-05-26 at 23:12 -0700, Eric Dumazet wrote: >> On Tue, 2015-05-26 at 22:47 -0700, Gopakumar Choorakkot Edakkunni wrote: >> > All, >> > >> > The original query I had posted is here : >> > http://stackoverflow.com/questions/30414350/tcp-syn-ack-tsecr-not-matching-tsval-in-syn >> > .. The summary is that once in a while, the TSval in SYN is not what >> > is getting echoed in TSecr, and looks like something on amazon aws >> > side is very strict about that and drops those packets. Any clues on >> > this - whether its a known issue/fixed elsewhere etc. would be of >> > great help. >> >> I guess that if you send SYN packets 3 times as your email did on >> netdev, that might cause some issues... >> >> More seriously, server has a SYN_RECV socket with same tuple, because of >> a SYN sent earlier : >> >> 8:36:00.593136 IP XX.YY.ZZ.VV.24548 > AA.BB.CC.DD.443: Flags [S], seq >> 1204544933, win 29200, options [mss 1320,sackOK,TS val 6032576 ecr >> 0,nop,wscale 7], length 0 >> >> 18:36:00.593171 IP AA.BB.CC.DD.443 > XX.YY.ZZ.VV.24548: Flags [S.], seq >> 986069863, ack 1204544934, win 14480, options [mss 1460,sackOK,TS val >> 180940028 ecr 6001497,nop,wscale 5], length 0 >> >> 18:36:00.992699 IP AA.BB.CC.DD.443 > XX.YY.ZZ.VV.24548: Flags [S.], seq >> 986069863, ack 1204544934, win 14480, options [mss 1460,sackOK,TS val >> 180940128 ecr 6001497,nop,wscale 5], length 0 >> >> >> From these traces, we can guess a SYN packet was sent about 31 seconds >> earlier. >> >> SYNACK rtx do not update the TSECR : Initial SYN TSval value (6001497) >> is mirrored. >> >> Are you establishing many active sessions per minute to this particular >> target ? >> > > Here is a packetdrill test to demonstrate behavior : > > // Test that SYNACK rtx tsecr is not changed (original SYN tsval) > > `../common/defaults.sh > ` > > // Create a socket. > 0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3 > 0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 > > 0.000 bind(3, ..., ...) = 0 > 0.000 listen(3, 1) = 0 > > // Establish a connection. > 0.100 < S 0:0(0) win 20000 <mss 1000,sackOK,TS val 100 ecr 0> > +0 > S. 0:0(0) ack 1 <mss 1460,sackOK,TS val 100 ecr 100> > > +0.100 < S 0:0(0) win 20000 <mss 1000,sackOK,TS val 199 ecr 0> > // check rtx tsecr is sill 100, not 199 > +0 > S. 0:0(0) ack 1 <mss 1460,sackOK,TS val 200 ecr 100> > > +0.100 < . 1:1(0) ack 1 win 20000 <nop,nop,TS val 300 ecr 200> > +0 accept(3, ..., ...) = 4 > > -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
