Henning Fehrmann wrote: > On Wed, Jul 02, 2008 at 09:19:50AM +0100, Tim Cutts wrote: >> On 2 Jul 2008, at 8:26 am, Carsten Aulbert wrote: >> >>> OK, we have 1342 nodes which act as servers as well as clients. Every >>> node exports a single local directory and all other nodes can mount this. >>> >>> What we do now to optimize the available bandwidth and IOs is spread >>> millions of files according to a hash algorithm to all nodes (multiple >>> copies as well) and then run a few 1000 jobs opening one file from one >>> box then one file from the other box and so on. With a short autofs >>> timeout that ought to work. Typically it is possible that a single >>> process opens about 10-15 files per second, i.e. making 10-15 mounts per >>> second. With 4 parallel process per node that's 40-60 mounts/second. >>> With a timeout of 5 seconds we should roughly have 200-300 concurrent >>> mounts (on average, no idea abut the variance). >> Please tell me you're not serious! The overheads of just performing the NFS >> mounts are going to kill you, never mind all the network traffic going >> all over the place. >> >> Since you've distributed the files to the local disks of the nodes, surely >> the right way to perform this work is to schedule the computations so that >> each node works on the data on its own local disk, and doesn't have to talk >> networked storage at all? Or don't you know in advance which files a >> particular job is going to need? > > Yes, this is the problem. The amount of files is too big to store it > everywhere (few TByte and 50 million files). Mounting a view NFS server does > not provide > the bandwidth. > On the other hand, the coreswitch should be able to handle the flows non > blocking. We think that nfs mounts are the fastest possibility to > distribute the demanded files to the nodes. > > Henning
Sounds like you need a parallel filesystem of some sort. Have you looked at that option? I know, they cost $$$$. -- Prentice _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
