My two cents: 2009/9/9 Rahul Nabar <rpna...@gmail.com>
> Our new cluster aims to have around 300 compute nodes. I was wondering > what is the largest setup people have tested NFS with? Any tips or > comments? There seems no way for me to say if it will scale well or > not. > > I have been warned of performance hits but how bad will they be? > Infiniband is touted as a solution but the economics don't work out. > My question is this: > > Assume each of my compute nodes have gigabit ethernet AND I specify > the switch such that it can handle full line capacity on all ports. > Will there still be performance hits as I start adding compute nodes? > Yes. > Why? Because final NFS server bandwidth will be the bandwidth of the most limited device, be it disk, network interface or switch. Even if you have a switch capable of fill line capacity for all 300 nodes, you must put a insanely fast interface in your NFS server and a giant pool of disks to have a decent bandwidth if all nodes access NFS at the same time. But depending on the way people run applications in your cluster, only a small set of nodes will access NFS at the same time and a Ethernet 10Gb with tens of disks will be enough. > Or is it unrealistic to configure a switching setup with full > line capacities on 300 ports? > This will be good for MPI but it will not help much your NFS server problem. > If not NFS then Lustre etc options do exist. But the more I read about > those the more I am convinced that those open another big can of > worms. Besides, if NFS would work I do not want to switch. > > -- > Rahul > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf >
_______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
