hmmm... 200 nodes writing to the same file. That is a hard problem. In all my testing of global FS's I haven't found one that is capable of doing this while delivering good performance. One might think that that MPI-IO would deliver performance while writing to the same file (on something like lustre) but in my experience, MPI-IO is more about functionality not performance.
In any code that I write that needs lot of bandwidth, I always write an n-m io routine. That is, your n processor task can read the previous m checkpoint-chunks (produced from an earlier m processor job). Then, when writing out the checkpoint or output file, you get each process to open its own individual file and dump its data to it. This gives you maximum bandwidth and stops meta-data thrashing on your cluster FS. It is also quite easy to write single-cpu tools which concatenate the files together... Alternatively, you can write a simple client-side FUSE file system which sort of joins multiple NFS mounts together into a single FS. In this way, you can stripe your IO over multiple NFS mounts... very similar to the cluster file system that was present in the Digital/Compaq SC machines. In this fashon, your file in the FUSE FS looks consistent and coherent while in the underlying nfs directories you see your file split up into bits (file.1 file.2 file.3 file.4 etc for a 4 nfs mount system). A simple way to get your bandwidth up (especially if your nfs mounts are coming in over different gig-e nics) but still gives REALLY crap bandwidth when trying to have multiple threads writing to the same file... Try Lustre :)
.Our big cluster is 2000+ plus nodes and we only have some 270TB (I say only cause we'e getting another 100TB+ SAN by year end) and we are able move a lot of production through that cluster and that's what it is all about. We do have numbers cluster wide. As we move to the new model, we have to deal with 200 nodes trying to write to one large file, so we need to explore ways of accomplishing this without affecting our production environment. Any ideas?
-- Dr Stuart Midgley [EMAIL PROTECTED] _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
