I am sorry for not making things more clear. scp was faster then NFS
only when NFS server was being hammered by the client nodes. This
happens only when number of client nodes increase above a certain
number. In my test I was reading files (1-2 MB) from an NFS server from
180 nodes. All the nodes were reading different files from different
folders and there was no caching. There was a huge io wait time on the
server CPUs.
The setup is like this:
Server exports /data
Clients mount /data and read files from /data/node1, /data/node2 and so on.
While the server is busy, if I try to do
node1:/tmp# scp largefile1 server:/data/node1
node1:/tmp# cp largefile2 /data/node1
Now in this situation scp is faster then the cp. This lead me to believe
that NFS is the bottleneck when it hammered by requests from too many nodes.
Amrik
Bogdan Costescu wrote:
On Wed, 25 Apr 2007, Amrik Singh wrote:
I agree that Jumbo Frames would not be a great help with the root
file system but we hope to get a better performance from other NFS
servers.
I don't quite follow you here: the link that was sent earlier had
shown some kernel level autoconfig problem. If this is indeed still
the case with newer kernels, this should only affect the root FS -
before mounting the other NFS exports, you should have the chance to
perform a proper initialization of the network, which would give you
the possibility to use a larger MTU.
As all the machines on the same subnet have to be using the jumbo
frames,
Why ? The MTU specifies a _maximum_ value; even when using 1500, not
all packets are exactly 1500 bytes. The larger MTU _allows_ larger
values, but doesn't _force_ them.
Even though NFS is extremely slow, copying files over scp is still
very fast between a client and server.
If by this you mean that NFS is slower than scp, then this should be
exactly the other way around, because scp needs CPU time for its
encryption. I would have said that you have some network problems and
use NFS over UDP which leads to high retransmission rates; scp would
adapt better to the network problems due to using TCP... but you just
mention that you tried switching between TCP and UDP.
We have tried all different ways to tune the NFS for a better
performance (increasing NFS deamons on the servers, changing rsize &
wsize, using TCP vs UDP, using async vs sync, noatime, timeo).
Could it be that you tried so hard to tune it that you just used too
many settings that don't play well together in your particular
situation ? I've found the NFS client and server in the kernels of the
recent years to perform reasonably well in their default
configuration; although they could be further optimized, a NFS
transfer in these conditions would always beat a scp one.
initrd=bzImage ramdisk=40960
dhclient could be located in the ramdisk. Actually even a whole root
FS could be located in the ramdisk, if there are not too many nodes
booting at the same time and leading to UDP packet loss.
... and you already have a rather large ramdisk. Have you created it
yourself (and therefore know what's in it and how to add some more) ?
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
