Re: Packet timestamps (was: Re: Network performance degradation from 2.6.11.12 to 2.6.16.20)

2007-03-06 Thread Vladimir B. Savkin
On Tue, Mar 06, 2007 at 04:16:24PM +0100, Eric Dumazet wrote: > > It would be better to name the tunable "disable_timestamps", default 0 of > course I agree. If networking maintainers are interested, I surely can prepare a patch. But IMO some way to force TSC usage on x86_64 will be even b

Re: Packet timestamps (was: Re: Network performance degradation from 2.6.11.12 to 2.6.16.20)

2007-03-06 Thread Eric Dumazet
On Tuesday 06 March 2007 15:43, Vladimir B. Savkin wrote: > On Tue, Mar 06, 2007 at 03:38:44PM +0100, Eric Dumazet wrote: > > 2) "accurate_timestamps" is misleading. > > Should be "disable_timestamps" > > Not, if default is 1, as in my patch. Yes I saw this. I should write more words next time

Re: Packet timestamps (was: Re: Network performance degradation from 2.6.11.12 to 2.6.16.20)

2007-03-06 Thread Vladimir B. Savkin
On Tue, Mar 06, 2007 at 03:38:44PM +0100, Eric Dumazet wrote: > 2) "accurate_timestamps" is misleading. > Should be "disable_timestamps" Not, if default is 1, as in my patch. ~ :wq With best regards, Vladi

Re: Packet timestamps (was: Re: Network performance degradation from 2.6.11.12 to 2.6.16.20)

2007-03-06 Thread Eric Dumazet
On Tuesday 06 March 2007 14:25, Vladimir B. Savkin wrote: > }, > + { > + .ctl_name = NET_CORE_ACCURATE_TIMESTAMPS, > + .procname = "accurate_timestamps", > + .data = &sysctl_accurate_timestamps, > + .maxlen = s

Packet timestamps (was: Re: Network performance degradation from 2.6.11.12 to 2.6.16.20)

2007-03-06 Thread Vladimir B. Savkin
On Fri, Sep 22, 2006 at 09:51:09AM -0700, Rick Jones wrote: > >That came from named. It opens lots of sockets with SIOCGSTAMP. > >No idea what it needs that many for. > > IIRC ISC BIND named opens a socket for each IP it finds on the system. > Presumeably in this way it "knows" implicitly the des

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-22 Thread Rick Jones
That came from named. It opens lots of sockets with SIOCGSTAMP. No idea what it needs that many for. IIRC ISC BIND named opens a socket for each IP it finds on the system. Presumeably in this way it "knows" implicitly the destination IP without using platform-specific recvfrom/whatever extensi

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-22 Thread Andi Kleen
On Friday 22 September 2006 17:35, Alexey Kuznetsov wrote: > Hello! > > > I can't even find a reference to SIOCGSTAMP in the > > dhcp-2.0pl5 or dhcp3-3.0.3 sources shipped in Ubuntu. > > > > But I will note that tpacket_rcv() expects to always get > > valid timestamps in the SKB, it does a: > >

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-22 Thread Alexey Kuznetsov
Hello! > I can't even find a reference to SIOCGSTAMP in the > dhcp-2.0pl5 or dhcp3-3.0.3 sources shipped in Ubuntu. > > But I will note that tpacket_rcv() expects to always get > valid timestamps in the SKB, it does a: It is equally unlikely it uses mmapped packet socket (tpacket_rcv). I even i

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-19 Thread Andi Kleen
> It seems only natural to me that the real problem is the slow > clock source which needs to be resolved regardless of the > outcome of this discussion. I believe that updating the stamp > at socket enqueue time is the right thing to do but it shouldn't > be considered as a solution to the perfor

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-19 Thread Thomas Graf
* David Miller <[EMAIL PROTECTED]> 2006-09-18 14:22 > From: Alexey Kuznetsov <[EMAIL PROTECTED]> > Date: Tue, 19 Sep 2006 01:03:21 +0400 > > > 1. It even does not disable possibility to record timestamp inside > >driver, which Alan was afraid of. The sequence is: > > > > if (!skb->tstamp.

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-19 Thread David Miller
From: Alexey Kuznetsov <[EMAIL PROTECTED]> Date: Tue, 19 Sep 2006 02:00:38 +0400 > * I do not undestand what the hell dhcp needs timestamps for. I can't even find a reference to SIOCGSTAMP in the dhcp-2.0pl5 or dhcp3-3.0.3 sources shipped in Ubuntu. But I will note that tpacket_rcv() expects to

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-19 Thread Stephen Hemminger
The sky2 hardware (and others) can timestamp in hardware, but trying to keep device ticks and system clock in sync looked too nasty to contemplate actually using it. - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-19 Thread David Miller
From: David Lang <[EMAIL PROTECTED]> Date: Mon, 18 Sep 2006 14:57:04 -0700 (PDT) > yes tcpdump may be wrong in requesting timestamps (in most cases it > probably is, but in some cases it's doing exactly what the sysadmin > wants it to do), but I don't think that many sysadmins would expect > this

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-19 Thread David Miller
From: "Vladimir B. Savkin" <[EMAIL PROTECTED]> Date: Tue, 19 Sep 2006 02:03:31 +0400 > On Tue, Sep 19, 2006 at 02:00:38AM +0400, Alexey Kuznetsov wrote: > > * I do see get_offset_pmtmr() in top lines of profile. That's scary enough. > > I had it at the very top line. That is just rediculious. Y

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Andi Kleen
On Monday 18 September 2006 23:22, David Miller wrote: > Ok, ok, but don't we have queueing disciplines that need the timestamp > even on ingress? I grepped and I can't find any. The only non SIOCGTSTAMP users of the time stamp seem to be sunrpc and conntrack and I bet both can be converted over

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Andi Kleen
On Monday 18 September 2006 23:03, Alexey Kuznetsov wrote: > > > And do you have some other prefered way to solve this? Even if the timer > > was fast it would be still good to avoid it in the fast path when DHCPD > > is running. > > No. The way, which you suggested, seems to be the best. Ok. I

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread David Lang
On Tue, 19 Sep 2006, Alexey Kuznetsov wrote: Hello! Please think about it this way: suppose you haave a heavily loaded router and some network problem is to be diagnosed. You run tcpdump and suddenly router becomes overloaded (by switching to timestamp-it-all mode I am sorry. I cannot think

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Vladimir B. Savkin
On Tue, Sep 19, 2006 at 02:00:38AM +0400, Alexey Kuznetsov wrote: > Hello! > > > Please think about it this way: > > suppose you haave a heavily loaded router and some network problem is to > > be diagnosed. You run tcpdump and suddenly router becomes overloaded (by > > switching to timestamp-it-a

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Alexey Kuznetsov
Hello! > Please think about it this way: > suppose you haave a heavily loaded router and some network problem is to > be diagnosed. You run tcpdump and suddenly router becomes overloaded (by > switching to timestamp-it-all mode I am sorry. I cannot think that way. :-) Instead of attempts to scar

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Alexey Kuznetsov
Hello! > Ok, ok, but don't we have queueing disciplines that need the timestamp > even on ingress? I cannot find. ip_queue does. But it is just another user, not different of sockets. BTW in any case, any user of timestamp who sees 0, because skb was received before timestamping was enabled, ha

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread David Miller
From: Alexey Kuznetsov <[EMAIL PROTECTED]> Date: Tue, 19 Sep 2006 01:03:21 +0400 > 1. It even does not disable possibility to record timestamp inside >driver, which Alan was afraid of. The sequence is: > > if (!skb->tstamp.off_sec) > net_timestamp(skb); > > 2. Maybe, ne

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Vladimir B. Savkin
On Mon, Sep 18, 2006 at 06:50:22PM +0200, Andi Kleen wrote: > > I suppose in the worst case a sysctl like Vladimir asked for could be added, > but it would seem somewhat lame. > Please think about it this way: suppose you haave a heavily loaded router and some network problem is to be diagnosed.

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Vladimir B. Savkin
On Mon, Sep 18, 2006 at 01:27:57PM +0200, Andi Kleen wrote: > The codebase for timing (and lots of other things) is quite different > between 32bit and 64bit. You're really surprised it doesn't work if you do > such things? > It works, and after your remark above, I'm surprised. Dunno about slow

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Alexey Kuznetsov
Hello! > But that never happens right? Right. Well, not right. It happens. Simply because you get packet with newer timestamp after previous handler saw this packet and did some actions. I just do not see any bad consequences. > And do you have some other prefered way to solve this? Even if t

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Andi Kleen
On Monday 18 September 2006 18:28, Alexey Kuznetsov wrote: > Hello! > > > Hmm, not sure how that could happen. Also is it a real problem > > even if it could? > > As I said, the problem is _occasionally_ theoretical. > > This would happen f.e. if packet socket handler was installed > after IP ha

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Alexey Kuznetsov
Hello! > Hmm, not sure how that could happen. Also is it a real problem > even if it could? As I said, the problem is _occasionally_ theoretical. This would happen f.e. if packet socket handler was installed after IP handler. Then tcpdump would get packet after it is processed (acked/replied/for

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Andi Kleen
hat it results in "performance degradation", it just > should not be used at all, even such pariah as tcpdump wants to be fast. > > Actually, I have a question. Why the subject is > "Network performance degradation from 2.6.11.12 to 2.6.16.20"? > I do not see

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Alexey Kuznetsov
lock source. If it is so slow, that it results in "performance degradation", it just should not be used at all, even such pariah as tcpdump wants to be fast. Actually, I have a question. Why the subject is "Network performance degradation from 2.6.11.12 to 2.6.16.20"? I do not see

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Andi Kleen
On Monday 18 September 2006 17:19, Alan Cox wrote: > Ar Llu, 2006-09-18 am 16:29 +0200, ysgrifennodd Andi Kleen: > > The only delay this would add would be the queueing time from the NIC > > to the softirq. Do you really think that is that bad? > > If you are trying to do things like network recor

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Alan Cox
Ar Llu, 2006-09-18 am 16:29 +0200, ysgrifennodd Andi Kleen: > The only delay this would add would be the queueing time from the NIC > to the softirq. Do you really think that is that bad? If you are trying to do things like network record/playback then you want the minimal delay. There's a reason

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Andi Kleen
> > People who run tcpdump want "wire" timestamps as close as possible. > Yes, things get delayed with the IRQ path, DMA delays, IRQ > mitigation and whatnot, but it's an order of magnitude worse if > you delay to user read() since that introduces also the delay of > the packet copies to userspac

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread David Miller
From: Andi Kleen <[EMAIL PROTECTED]> Date: 18 Sep 2006 11:58:21 +0200 > For netdev: I'm more and more thinking we should just avoid the > problem completely and switch to "true end2end" timestamps. This > means don't time stamp when a packet is received, but only when it > is delivered to a socket

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Andi Kleen
"Vladimir B. Savkin" <[EMAIL PROTECTED]> writes: [you seem to send your emails in a strange way that doesn't keep me in cc. Please stop doing that.] > On Mon, Sep 18, 2006 at 11:58:21AM +0200, Andi Kleen wrote: > > > > The x86-64 timer subsystems currently doesn't have clocksources > > > > at all

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Vladimir B. Savkin
On Mon, Sep 18, 2006 at 11:58:21AM +0200, Andi Kleen wrote: > > > The x86-64 timer subsystems currently doesn't have clocksources > > > at all, but it supports TSC and some other timers. > > > > > until I hacked arch/i386/kernel/tsc.c > > Then you don't use x86-64. > Oh. I mean I made arch/i38

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Andi Kleen
"Vladimir B. Savkin" <[EMAIL PROTECTED]> writes: > On Mon, Sep 18, 2006 at 10:35:38AM +0200, Andi Kleen wrote: > > > I just found out that TSC clocksource is not implemented on x86-64. > > > Kernel version 2.6.18-rc7, is it true? > > > > The x86-64 timer subsystems currently doesn't have clocksou

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Vladimir B. Savkin
On Mon, Sep 18, 2006 at 10:35:38AM +0200, Andi Kleen wrote: > > I just found out that TSC clocksource is not implemented on x86-64. > > Kernel version 2.6.18-rc7, is it true? > > The x86-64 timer subsystems currently doesn't have clocksources > at all, but it supports TSC and some other timers. H

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-18 Thread Andi Kleen
"Vladimir B. Savkin" <[EMAIL PROTECTED]> writes: > On Mon, Jun 19, 2006 at 05:24:31PM +0200, Andi Kleen wrote: > > > > > If you use "pmtmr" try to reboot with kernel option "clock=tsc". > > > > That's dangerous advice - when the system choses not to use > > TSC it often has a reason. > > I just

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-09-16 Thread Vladimir B. Savkin
On Mon, Jun 19, 2006 at 05:24:31PM +0200, Andi Kleen wrote: > > > If you use "pmtmr" try to reboot with kernel option "clock=tsc". > > That's dangerous advice - when the system choses not to use > TSC it often has a reason. I just found out that TSC clocksource is not implemented on x86-64. Kern

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-07-10 Thread Jesper Dangaard Brouer
On Tue, 4 Jul 2006, Andi Kleen wrote: On Tuesday 04 July 2006 13:41, Jesper Dangaard Brouer wrote: Actually the change happens between kernel version 2.6.15 and 2.6.16. The timestamp optimizations are older. Don't remember the exact release, but earlier 2.6. What I'm saying is that, with

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-07-04 Thread Andi Kleen
On Tuesday 04 July 2006 13:41, Jesper Dangaard Brouer wrote: > > On Mon, 26 Jun 2006, Andi Kleen wrote: > > >> I encountered the same problem on a dual core opteron equipped with a > >> broadcom NIC (tg3) under 2.4. It could receive 1 Mpps when using TSC > >> as the clock source, but the time jum

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-07-04 Thread Jesper Dangaard Brouer
On Mon, 26 Jun 2006, Andi Kleen wrote: I encountered the same problem on a dual core opteron equipped with a broadcom NIC (tg3) under 2.4. It could receive 1 Mpps when using TSC as the clock source, but the time jumped back and forth, so I changed it to 'notsc', then the performance dropped dra

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-25 Thread Andi Kleen
> I encountered the same problem on a dual core opteron equipped with a > broadcom NIC (tg3) under 2.4. It could receive 1 Mpps when using TSC > as the clock source, but the time jumped back and forth, so I changed > it to 'notsc', then the performance dropped dramatically to around the > same val

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-25 Thread Bill Fink
On Sun, 25 Jun 2006, Harry Edmon wrote: > I understand the saying "beggars can't be choosers", but I have heard nothing > on > this issue since June 19th. Does anyone have any ideas on what is going on? > Is > there more information I can collect that would help diagnose this problem? > An

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-25 Thread Willy Tarreau
Hi Andi, On Mon, Jun 19, 2006 at 05:24:31PM +0200, Andi Kleen wrote: > > > If you use "pmtmr" try to reboot with kernel option "clock=tsc". > > That's dangerous advice - when the system choses not to use > TSC it often has a reason. > > > > > On my Opteron AMD system i normally can route 400 k

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-25 Thread Harry Edmon
I understand the saying "beggars can't be choosers", but I have heard nothing on this issue since June 19th. Does anyone have any ideas on what is going on? Is there more information I can collect that would help diagnose this problem? And again, thanks for any and all help! -- Dr. Harry Ed

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-19 Thread Herbert Xu
Harry Edmon <[EMAIL PROTECTED]> wrote: > > That did not help. I have 1 minute outputs from tcpdump under both 2.6.11.12 > and 2.6.16.20. You will see a large size difference between the files. > Since > the 2.6.11.12 one is 2 MBytes, I thought I would post them via the web > instead > of v

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-19 Thread Andi Kleen
On Monday 19 June 2006 19:34, Chris Friesen wrote: > Andi Kleen wrote: > > Incoming packets are only time stamped > > when someone asks for the timestamps. > > Doesn't that add scheduling latency to the timestamps? Or is is a flag > that gets set to trigger timestamping at packet arrival? It's a

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-19 Thread Jesper Dangaard Brouer
On Mon, 19 Jun 2006, Andi Kleen wrote: If you use "pmtmr" try to reboot with kernel option "clock=tsc". That's dangerous advice - when the system choses not to use TSC it often has a reason. Sorry, it was not a general advice, just something to try out. It really solved my network perform

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-19 Thread Chris Friesen
Andi Kleen wrote: Incoming packets are only time stamped when someone asks for the timestamps. Doesn't that add scheduling latency to the timestamps? Or is is a flag that gets set to trigger timestamping at packet arrival? Chris - To unsubscribe from this list: send the line "unsubscribe n

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-19 Thread Harry Edmon
Jesper Dangaard Brouer wrote: Harry Edmon <[EMAIL PROTECTED]> wrote: I have a system with a strange network performance degradation from 2.6.11.12 to most recent kernels including 2.6.16.20 and 2.6.17-rc6. The system is has Dual single core Xeons with hyperthreading on. Hi Harry Can y

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-19 Thread Andi Kleen
> If you use "pmtmr" try to reboot with kernel option "clock=tsc". That's dangerous advice - when the system choses not to use TSC it often has a reason. > > On my Opteron AMD system i normally can route 400 kpps, but with > timesource "pmtmr" i could only route around 83 kpps. (I found the t

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-19 Thread Jesper Dangaard Brouer
Harry Edmon <[EMAIL PROTECTED]> wrote: I have a system with a strange network performance degradation from 2.6.11.12 to most recent kernels including 2.6.16.20 and 2.6.17-rc6. The system is has Dual single core Xeons with hyperthreading on. Hi Harry Can you check which "high-res timesourc

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-19 Thread Harry Edmon
Stephen Hemminger wrote: Does this fix it? # sysctl -w net.ipv4.tcp_abc=0 That did not help. I have 1 minute outputs from tcpdump under both 2.6.11.12 and 2.6.16.20. You will see a large size difference between the files. Since the 2.6.11.12 one is 2 MBytes, I thought I would post them

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-18 Thread Harry Edmon
Stephen Hemminger wrote: Does this fix it? # sysctl -w net.ipv4.tcp_abc=0 Thanks for the suggestion. I will give it a try later tonight. Also Andrew - sorry for the incorrect placement of my follow-up comments. I do appreciate everyone's help in figuring this out. -- Dr. Harry Ed

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-17 Thread Stephen Hemminger
Andrew Morton wrote: On Sat, 17 Jun 2006 16:23:34 -0700 Harry Edmon <[EMAIL PROTECTED]> wrote: Andrew Morton wrote: On Fri, 16 Jun 2006 09:01:23 -0700 Harry Edmon <[EMAIL PROTECTED]> wrote: I have a system with a strange network performance degradation from 2.6.11.12 to most r

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-17 Thread Andrew Morton
On Sat, 17 Jun 2006 16:23:34 -0700 Harry Edmon <[EMAIL PROTECTED]> wrote: > Andrew Morton wrote: > > On Fri, 16 Jun 2006 09:01:23 -0700 > > Harry Edmon <[EMAIL PROTECTED]> wrote: > > > >> I have a system with a strange network performance degradation from > >> 2.6.11.12 to most recent kernels in

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-17 Thread Harry Edmon
I assume you are talking about using TCP_NODELAY as a socket option within the LDM software. I could give that a try. There is a lot of traffic on this node, on the order of 2000 packets in and out per second, so the tcpdump output will grow pretty fast. How long a tcpdump would be useful, a

Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

2006-06-17 Thread Andrew Morton
On Fri, 16 Jun 2006 09:01:23 -0700 Harry Edmon <[EMAIL PROTECTED]> wrote: > I have a system with a strange network performance degradation from > 2.6.11.12 to most recent kernels including 2.6.16.20 and 2.6.17-rc6. > The system is has Dual single core Xeons with hyperthreading on. The > app