Some interesting comments from an interesting source.
BTW. I'd appreciate it if everyone noticed how friendly and helpful Linus
is and Not find some thing to rag on linux about with this. The aim of
this thread is to improve FreeBSD in any way we see as being a real
improvement. Don't cc him back into the topic. I'm sure he has enough to
worry about. This is FYI only ok?
---------- Forwarded message ----------
Date: Fri, 25 Jun 1999 00:51:10 -0700 (PDT)
From: Linus Torvalds <[EMAIL PROTECTED]>
To: Sean Eric Fagan <[EMAIL PROTECTED]>
Cc: Julian Elischer <[EMAIL PROTECTED]>
Subject: Re: Microsoft performance (was: ...)
On Thu, 24 Jun 1999, Sean Eric Fagan wrote:
>
> Not sure if you've seen this stuff before; thought you might find it
> interesting if not. (This is not intended to be anything other than, "NT
> performs surprisingly well" :).)
Indeed (Julian cc'd).
We had this taught to us the hard way with the mindcraft benchmark.
NT scales VERY well for netbench. It became pretty obvious pretty quickly
that it's one of their basic benchmarks, and it's what they have optimized
NT and IIS for. Fair enough - netbench may not be the best benchmark out
there, but there isn't anything better.
Julian, feel free to forward my comments to anybody, for whatever they are
worth.
> Ok well here are some real numbers for you..
> Win NT 4processors 1GB ram + raid array + IIS
> webbench... 4000 transactions per second...
>
> FreeBSD.. Identical hardware..
> 1450 transactions per seccond
> Linux: 2000 per second
> Solaris86 6000 per second
The major problem with webbench is apache, although the OS _does_ matter.
The webbench overhead is largely the socket timeouts and you have to do
accept() right scalability-wise. I suspect NT just doesn't do the timewait
stuff at all, but I could be wrong.
We ended up doing the accept() thing in Linux (waking up just one thread),
even though I personally think it doesn't have that big an impact in real
life. It was easy enough to do.
Once you get over that, the numbers seem to be just purely about scaling
the fs and network, but I never got interested enough to check more into
it. The netbench load was more interesting in that area, and we ended up
having a benchmark that didn't have the apache overhead so it was a
"purer" benchmark anyway. AND it was what NT did really well on, so..
The apache guys are definitely looking at a new threading model, you might
try to approach Dean Gaudet and ask him what he's up to. That, together
with the scalaibility stuff, should hopefully be enough. NT didn't do
_that_ well on this one.
And btw, if you're feeling bad about the numbers above: C'T in Germany
did a similar kind of benchmark which was more UNIX-oriented, with perl
CGI scripts etc, and NT did really badly on it. Another case of NT being
optimized for one thing, and one thing only.
Oh, except that if you do the "MS CGI" (whetever they call they built-in
modules), NT actually does that pretty well - built in to IIS. So it's not
even "dynamic content" - it's really about what _kind_ of CGI you do.
> With Netbench:
> NT blows us away.
Netbench is the big thing for NT. And the numbers you quote are the "bad"
NT numbers (ie bad enough that Linux actually was able to inch ahead of
NT). The _good_ NT numbers are three times as good, but they are done with
(a) NT running FATFS, (b) w98 clients exclusively.
> (we're talking an order of magnitude faster)
> I'm not going ot give real numbers as I don't have them readily at hand
> but they are something like 12MB/Sec for FreeBSD vs 90 MB/sec for NT and
> 120MB/sec for linux. Matt has some patches that raise the 12 to 35 and
> kirk has some changes that may raise the numbers to 70 or more,
> and John has some patches that may add more again, but it's all theory,
> and some of the patches have had less results than we expected.
Qutoe frankly, to get close to NT, you have to scale really well to four
CPU's. We've had the core "deep SMP people" working on this for the last
month, and we _finally_ have a system that seems to scale pretty well:
well enough that we think we can beat the _good_ NT numbers too. Or at
least come so close as to not matter any more.
When I say scale really well, I mean it. You need to (a) make sure your
disk never does anything at all, and (b) make sure that you get almost
perfect scaling for cached reads and writes. That's what NT comes pretty
close to doing, it seems.
The new Linux code has a per-page lock on the page cache, and pretty much
nothing else. And I don't think you can beat NT with anything less.
> With Uniprocessor things are a lot more equal.
> but we still suck on netbench.
If you suck on netbench even on UP, the most likely reason is just that
you're doing too much IO. You can make a noticeable portion of the load go
away by trying to keep dirty state in memory, and making sure that
unlink() removes the dirty state without ever writing it out.
That doesn't get rid of the problem (the netbench load is large enough
that it doesn't fit in 1GB - each part of it would fit, but if you
actually do the clients concurrently their combined working set is more
than you can cache).
The way to cheat on netbench _seems_ to be to actually serialize the
client accesses - doing a few clients at a time, and letting the clients
remove their own crud and avoiding the writeouts because the files got
unlinked. That's what it looked like from our analysis, anyway. That way
you can cache the whole working set because you do it piece-meal.
HOWEVER - NT doesn't actually seem to cheat that way, so we never pursued
that as a venue either (I'd feel really bad if the only way I could beat
NT was through cheating). I just thought I'd bring it up as an example of
how bad a benchmark it really is.
> This is due to the exact form of netbench which is exactly nonoptimal for
> FreeBSD.
netbench really is a horrible benchmark. I don't know what the FreeBSD
tuning parameters are, but if you can avoid flushing dirty buffers that
will help. Try it (it will make interactive performance suck, but hey,
it's a benchmark, not real life).
> Also becaosue of the GKL (Giant Kernel Lock) (see Solaris's results)
That's the big one. We're seeing a factor of 2.5 on a four-way. It's not a
four-time increase, but it seems to become memory bus limited at that
point.
I'll warn you: we were able to get rid of the big kernel lock around the
filesystem reasonably quickly, but that was mainly because we had all the
infrastructure in place (that's what Linux-2.2 was all about - in
Linux-2.2 the kernel lock is still heavily used, but it's not really a
fundamental requirement any more). Even so, it wasn't exactly painless. We
had to change the page cache quite extensively, even though I originally
tried to design the page cache exactly for something like this.
The networking layer also needs to scale. I refused to put as much effort
into that, because I don't think that 4 100Mbps ethernet cards are all
that realistic - it's much more realistic to have a single gigabit card
and a switch for this kind of server, so I feel that for the networking
most of the scalability is in the end going to be limited by the IO.
On the filesystem side, I decided that the cache really has to scale
perfectly.
> Basically there are some applications and benchmarks for which FreeBSD
> will really suck. We're working on them but some things are just a result
> of how we do things.
>
> So don't assume that NT figures must be bad..
> we have too many weaknesses in our own code to throw stones.
I was certainyl surprised by how well NT does. NT does reasonably well in
another area too: writing back data through a shared file mapping was one
of the few benchmarks NT actually _won_ against both Solaris and Linux on
lmbench. We fixed it in the meantime, but the point being that there _are_
things that NT does well.
However, I still feel pretty confident in that the "NT does this really
well" category does seem to be a matter of a few special things that they
have worked on a lot rather than anything "good across the board due to a
kick-ass design" kind of goodness.
So I do feel pretty good, and I do believe that I'll be able to release a
new version of Linux by the end of the year that actually outperforms NT
on the things it seems to shine on. I'm waiting with interest to see what
the next benchmark the MS people are going to find that they do better on.
They've obviously started to look for benchmarks to do that they do well
on..
> It'd be intersting to see how FreeBSD 1.1.5 would have performed on the
> same tests. Sometimes we've gained in general performance but lost in
> some specific cases.
I don't think you'll find that you'd do very differently on either
NetBench or WebBench, unless you've really been working on the parts that
they test. But I'd be happy to share what we found out when we were trying
to figure out exactly how NT was able to do so much better..
Linus
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
