Re: TCP connection stops after high load.

2007-04-17 Thread Robert Iakobashvili
> Yes, it fixes. Thanks, I will submit it to -stable branch. David and John, Thanks for your caring and attention. -- Sincerely, Robert Iakobashvili, coroberti %x40 gmail %x2e com ... Navigare necesse est, vivere non est necesse

Re: TCP connection stops after high load.

2007-04-17 Thread David Miller
From: John Heffner <[EMAIL PROTECTED]> Date: Tue, 17 Apr 2007 15:47:58 -0400 > My only reservation in submitting this to -stable is that it will in > many cases increase the default tcp_mem values, which in turn can > increase the default tcp_rmem values, and therefore the window scale. > There

Re: TCP connection stops after high load.

2007-04-17 Thread John Heffner
David Miller wrote: From: "Robert Iakobashvili" <[EMAIL PROTECTED]> Date: Tue, 17 Apr 2007 10:58:04 +0300 David, On 4/16/07, David Miller <[EMAIL PROTECTED]> wrote: Commit: 53cdcc04c1e85d4e423b2822b66149b6f2e52c2c Author: John Heffner <[EMAIL PROTECTED]> Fri, 16 Mar 2007 15:04:03 -0700

Re: TCP connection stops after high load.

2007-04-17 Thread David Miller
From: "Robert Iakobashvili" <[EMAIL PROTECTED]> Date: Tue, 17 Apr 2007 10:58:04 +0300 > David, > > On 4/16/07, David Miller <[EMAIL PROTECTED]> wrote: > > > > Commit: 53cdcc04c1e85d4e423b2822b66149b6f2e52c2c > > > > Author: John Heffner <[EMAIL PROTECTED]> Fri, 16 Mar 2007 15:04:03 -0700 > > > >

Re: TCP connection stops after high load.

2007-04-17 Thread Robert Iakobashvili
David, On 4/16/07, David Miller <[EMAIL PROTECTED]> wrote: > > Commit: 53cdcc04c1e85d4e423b2822b66149b6f2e52c2c > > Author: John Heffner <[EMAIL PROTECTED]> Fri, 16 Mar 2007 15:04:03 -0700 > > > > [TCP]: Fix tcp_mem[] initialization. > > Change tcp_mem initialization function. The fra

Re: TCP connection stops after high load.

2007-04-16 Thread David Miller
From: John Heffner <[EMAIL PROTECTED]> Date: Mon, 16 Apr 2007 15:11:07 -0400 > I don't know if this qualifies as an unconditional bug. The commit > above was actually a bugfix so that the limits were not higher than > total memory on some systems, but had the side effect that it made them > ev

Re: TCP connection stops after high load.

2007-04-16 Thread David Miller
From: "Robert Iakobashvili" <[EMAIL PROTECTED]> Date: Mon, 16 Apr 2007 20:51:54 +0200 > > Commit: 53cdcc04c1e85d4e423b2822b66149b6f2e52c2c > > Author: John Heffner <[EMAIL PROTECTED]> Fri, 16 Mar 2007 15:04:03 -0700 > > > > [TCP]: Fix tcp_mem[] initialization. > > Change tcp_mem initiali

Re: TCP connection stops after high load.

2007-04-16 Thread John Heffner
Robert Iakobashvili wrote: Kernels 2.6.19 and 2.6.20 series are effectively broken right now. Don't you wish to patch them? I don't know if this qualifies as an unconditional bug. The commit above was actually a bugfix so that the limits were not higher than total memory on some systems, bu

Re: TCP connection stops after high load.

2007-04-16 Thread Robert Iakobashvili
>> Robert Iakobashvili wrote: >> > Vanilla 2.6.18.3 works for me perfectly, whereas 2.6.19.5 and >> > 2.6.20.6 do not. >> > >> > Looking into the tcp /proc entries of 2.6.18.3 versus 2.6.19.5 >> > tcp_rmem and tcp_wmem are the same, whereas tcp_mem are >> > much different: >> > >> > kernel

Re: TCP connection stops after high load.

2007-04-16 Thread John Heffner
Robert Iakobashvili wrote: Hi John, On 4/15/07, John Heffner <[EMAIL PROTECTED]> wrote: Robert Iakobashvili wrote: > Vanilla 2.6.18.3 works for me perfectly, whereas 2.6.19.5 and > 2.6.20.6 do not. > > Looking into the tcp /proc entries of 2.6.18.3 versus 2.6.19.5 > tcp_rmem and tcp_wmem are th

Re: TCP connection stops after high load.

2007-04-15 Thread Robert Iakobashvili
Hi John, On 4/15/07, John Heffner <[EMAIL PROTECTED]> wrote: Robert Iakobashvili wrote: > Vanilla 2.6.18.3 works for me perfectly, whereas 2.6.19.5 and > 2.6.20.6 do not. > > Looking into the tcp /proc entries of 2.6.18.3 versus 2.6.19.5 > tcp_rmem and tcp_wmem are the same, whereas tcp_mem are

Re: TCP connection stops after high load.

2007-04-15 Thread John Heffner
Robert Iakobashvili wrote: Vanilla 2.6.18.3 works for me perfectly, whereas 2.6.19.5 and 2.6.20.6 do not. Looking into the tcp /proc entries of 2.6.18.3 versus 2.6.19.5 tcp_rmem and tcp_wmem are the same, whereas tcp_mem are much different: kernel tcp_mem --

Re: TCP connection stops after high load.

2007-04-15 Thread Robert Iakobashvili
On 4/13/07, David Miller <[EMAIL PROTECTED]> wrote: From: "Robert Iakobashvili" <[EMAIL PROTECTED]> Date: Thu, 12 Apr 2007 23:11:14 +0200 > It works good with 2.6.11.8 and debian 2.6.18.3-i686 image. > > At the same Intel Pentium-4 PC with the same about kernel configuration > (make oldconfig u

Re: TCP connection stops after high load.

2007-04-15 Thread Robert Iakobashvili
On 4/13/07, David Miller <[EMAIL PROTECTED]> wrote: From: "Robert Iakobashvili" <[EMAIL PROTECTED]> Date: Thu, 12 Apr 2007 23:11:14 +0200 > It works good with 2.6.11.8 and debian 2.6.18.3-i686 image. > > At the same Intel Pentium-4 PC with the same about kernel configuration > (make oldconfig u

Re: TCP connection stops after high load.

2007-04-13 Thread David Miller
From: Eric Dumazet <[EMAIL PROTECTED]> Date: Sat, 14 Apr 2007 07:31:35 +0200 > When did tg3 model changed exactly ? June of 2006: commit 00b7050426da8e7e58c889c5c80a19920d2d41b3 Author: Michael Chan <[EMAIL PROTECTED]> Date: Sat Jun 17 21:58:45 2006 -0700 [TG3]: Convert to non-LLTX

Re: TCP connection stops after high load.

2007-04-13 Thread Eric Dumazet
Herbert Xu a écrit : Eric Dumazet <[EMAIL PROTECTED]> wrote: dev_queue_xmit_nit() is called before attempting to send packet to device. If device could not accept the packet (hard_start_xmit() returns an error), packet is requeued and retried later. each retry means call ev_queue_xmit_nit() ag

Re: TCP connection stops after high load.

2007-04-13 Thread David Miller
From: Herbert Xu <[EMAIL PROTECTED]> Date: Sat, 14 Apr 2007 14:21:44 +1000 > Eric Dumazet <[EMAIL PROTECTED]> wrote: > > > > dev_queue_xmit_nit() is called before attempting to send packet to device. > > > > If device could not accept the packet (hard_start_xmit() returns an error), > > packet

Re: TCP connection stops after high load.

2007-04-13 Thread Herbert Xu
Eric Dumazet <[EMAIL PROTECTED]> wrote: > > dev_queue_xmit_nit() is called before attempting to send packet to device. > > If device could not accept the packet (hard_start_xmit() returns an error), > packet is requeued and retried later. > each retry means call ev_queue_xmit_nit() again, so tcp

Re: TCP connection stops after high load.

2007-04-13 Thread Ben Greear
Evgeniy Polyakov wrote: On Thu, Apr 12, 2007 at 02:36:34PM -0700, Ben Greear ([EMAIL PROTECTED]) wrote: I am not sure if the problem is fixed or just harder to hit, but for now it looks good. Wasn't default congestion control algo changed between that kernel releases? With such small r

Re: TCP connection stops after high load.

2007-04-13 Thread Eric Dumazet
On Fri, 13 Apr 2007 18:10:12 +0200 Daniel Schaffrath <[EMAIL PROTECTED]> wrote: > > On 2007/04/12 , at 20:19, Eric Dumazet wrote: > > > > Warning : tcpdump can lie, telling you packets being transmited > > several time. > Maybe you have further pointers how come that tcpdump lies about > dupl

Re: TCP connection stops after high load.

2007-04-13 Thread Daniel Schaffrath
On 2007/04/12 , at 20:19, Eric Dumazet wrote: On Thu, 12 Apr 2007 10:59:19 -0700 Ben Greear <[EMAIL PROTECTED]> wrote: Here is a tcpdump of the connection in the stalled state. As you can see by the 'time' output, it's running at around 100,000 packets per second. tcpdump dropped the v

Re: TCP connection stops after high load.

2007-04-13 Thread Evgeniy Polyakov
On Thu, Apr 12, 2007 at 02:36:34PM -0700, Ben Greear ([EMAIL PROTECTED]) wrote: > I am not sure if the problem is fixed or just harder to hit, > but for now it looks good. Wasn't default congestion control algo changed between that kernel releases? With such small rtt like in your setup there coul

Re: TCP connection stops after high load.

2007-04-12 Thread Ben Greear
Eric Dumazet wrote: Hum, could you try to bind nic irqs on separate cpus ? I just started a run on 2.6.20.4, and so far (~20 minutes), it is behaving perfect..running at around 925Mbps in both directions. CWND averages about 600, bouncing from a low of 300 up to 800, but that could very well b

Re: TCP connection stops after high load.

2007-04-12 Thread David Miller
From: "Robert Iakobashvili" <[EMAIL PROTECTED]> Date: Thu, 12 Apr 2007 23:11:14 +0200 > It works good with 2.6.11.8 and debian 2.6.18.3-i686 image. > > At the same Intel Pentium-4 PC with the same about kernel configuration > (make oldconfig using Debian config-2.6.18.3-i686) the setup fails wit

Re: TCP connection stops after high load.

2007-04-12 Thread Eric Dumazet
Ben Greear a écrit : Eric Dumazet wrote: What "tc -s -d qdisc" "ifconfig -a" "cat /proc/interrupts" "cat /proc/net/sockstat" and "cat /proc/net/softnet_stat" are telling ? In this test, eth2 is talking to eth3, using something similar to this send-to-self patch: http://www.candelatech.com/os

Re: TCP connection stops after high load.

2007-04-12 Thread Ben Greear
Eric Dumazet wrote: What "tc -s -d qdisc" "ifconfig -a" "cat /proc/interrupts" "cat /proc/net/sockstat" and "cat /proc/net/softnet_stat" are telling ? In this test, eth2 is talking to eth3, using something similar to this send-to-self patch: http://www.candelatech.com/oss/sts.patch [EMAI

Re: TCP connection stops after high load.

2007-04-12 Thread Eric Dumazet
On Thu, 12 Apr 2007 10:59:19 -0700 Ben Greear <[EMAIL PROTECTED]> wrote: > > Here is a tcpdump of the connection in the stalled state. As you can see by > the 'time' output, it's running at around 100,000 packets per second. tcpdump > dropped the vast majority of these. Based on the network int

Re: TCP connection stops after high load.

2007-04-12 Thread Ben Greear
Andi Kleen wrote: Ben Greear <[EMAIL PROTECTED]> writes: I don't mind adding printks...and I've started reading through the code, but there is a lot of it, and indiscriminate printks will likely just hide the problem because it will slow down performance so much. You could add /proc/net/snmp c

Re: TCP connection stops after high load.

2007-04-12 Thread Andi Kleen
Ben Greear <[EMAIL PROTECTED]> writes: > > I don't mind adding printks...and I've started reading through the code, > but there is a lot of it, and indiscriminate printks will likely just > hide the problem because it will slow down performance so much. You could add /proc/net/snmp counters for i

Re: TCP connection stops after high load.

2007-04-11 Thread Ilpo Järvinen
On Wed, 11 Apr 2007, Ben Greear wrote: > The problem is that I set up a TCP connection with bi-directional traffic > of around 800Mbps, doing large (20k - 64k writes and reads) between two ports > on > the same machine (this 2.6.18.2 kernel is tainted with my full patch set, > but I also reproduce

Re: TCP connection stops after high load.

2007-04-11 Thread SANGTAE HA
I also noticed this happening with 2.6.18 kernel version, but this was not severe with linux 2.6.20.3. So, the short-term solution will be upgrading to the latest kernel of FC-6. A long black-out is mostly observed when a lot of packet losses happened in slow start. You can prevent this by apply

Re: TCP connection stops after high load.

2007-04-11 Thread Benjamin LaHaise
On Wed, Apr 11, 2007 at 02:06:31PM -0700, Ben Greear wrote: > For the dup acks, I see nothing *but* dup acks on the wire...going in > both directions interestingly, at greater than 100,000 packets per second. > > I don't mind adding printks...and I've started reading through the code, > but there

Re: TCP connection stops after high load.

2007-04-11 Thread David Miller
From: Ben Greear <[EMAIL PROTECTED]> Date: Wed, 11 Apr 2007 14:31:00 -0700 > I've spent solid weeks tracking down obscure races. I've spent solid weeks tracking down kernel stack corruption and scsi problems on sparc64, as well as attending to my network maintainer duties, what is your point? >

Re: TCP connection stops after high load.

2007-04-11 Thread Ben Greear
David Miller wrote: From: Ben Greear <[EMAIL PROTECTED]> Date: Wed, 11 Apr 2007 14:06:31 -0700 Does the CWND == 1 count as solid? Any idea how/why this would go to 1 in conjunction with the dup acks? For the dup acks, I see nothing *but* dup acks on the wire...going in both directions interes

Re: TCP connection stops after high load.

2007-04-11 Thread David Miller
From: Ben Greear <[EMAIL PROTECTED]> Date: Wed, 11 Apr 2007 14:06:31 -0700 > Does the CWND == 1 count as solid? Any idea how/why this would go > to 1 in conjunction with the dup acks? > > For the dup acks, I see nothing *but* dup acks on the wire...going in > both directions interestingly, at gr

Re: TCP connection stops after high load.

2007-04-11 Thread Ben Greear
David Miller wrote: From: Ben Greear <[EMAIL PROTECTED]> Date: Wed, 11 Apr 2007 13:26:36 -0700 Interestingly, I found this page mentioning a SACK problem in Linux: http://www-didc.lbl.gov/TCP-tuning/linux.html Don't read that page, it is the last place in the world your should take hints and

Re: TCP connection stops after high load.

2007-04-11 Thread David Miller
From: Ben Greear <[EMAIL PROTECTED]> Date: Wed, 11 Apr 2007 13:26:36 -0700 > Interestingly, I found this page mentioning a SACK problem in Linux: > http://www-didc.lbl.gov/TCP-tuning/linux.html Don't read that page, it is the last place in the world your should take hints and advice from, most of

Re: TCP connection stops after high load.

2007-04-11 Thread David Miller
From: Ben Greear <[EMAIL PROTECTED]> Date: Wed, 11 Apr 2007 11:50:18 -0700 > So, I would like to dig into this problem myself since no one else > is reporting this type of problem, but I am quite ignorant of the TCP > stack implementation. Based on the dup-acks I see on the wire, I assume > the T

Re: TCP connection stops after high load.

2007-04-11 Thread Ben Greear
Ben Greear wrote: Back in May of last year, I reported this problem, but worked around it at the time by changing the kernel memory settings in the networking stack. I reproduced the problem again today with the previously working kernel memory settings..which is not supprising since I just pape