> On 09/13/2012 07:52 AM, hol...@th.physik.uni-frankfurt.de wrote: > > [...] > >> If I set up a single machine to hammer the fileserver with IOzone I see >> something like 50,000 IOPS but if all four machines are hammering the >> filesystem concurrently we got it up to 180,000 IOPS. > > I wouldn't recommend IOzone for this sort of testing. Its not a very > good load generator, and it has a tendency to report things which are > not actually seen at the hardware level. I'd noticed this some years > ago, when running some of our benchmark testing on these units, that an > entire IOzone benchmark completed with very few activity lights going on > the disks. Which suggested that the test was happily entirely cached, > and I was running completely within cache.
Hmm, It has thus far behaved in a very predictable manner. I am always using it in o_direct mode with the -I flag and/or -o o_sync mode. It has always given me results consistant with dd. > > Use fio. I will take a look. > > Second, are the disks behind the NFS/ZFS server solid state, ram disk, > or spinning rust? I have 20x 15k sas drives in 10 mirrored sets. An STEC IOPS SSD as read cache and an 8GB STEC ZEUS battery backed RAM device for my SLOG/ZIL. Also lots and lots of mem. > > Are you asking why a single machine cannot fill your QDR bandwidth? That is exactly my question. > > I'd recommend running traces on the individual machines to see where > things are getting lost. One you have the traces, post em, and see if > people can help. https://th.physik.uni-frankfurt.de/~holway/strace-trunkd.log https://th.physik.uni-frankfurt.de/~holway/strace.log I have chopped out all the data from the strace in strace-trunkd.log Not sure this output is completely useful :) Thanks, Andrew > > -- > Joseph Landman, Ph.D > Founder and CEO > Scalable Informatics Inc. > email: land...@scalableinformatics.com > web : http://scalableinformatics.com > http://scalableinformatics.com/sicluster > phone: +1 734 786 8423 x121 > fax : +1 866 888 3112 > cell : +1 734 612 4615 > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
