On Sep 13, 2012, at 2:34 PM, Joe Landman wrote: > On 09/13/2012 07:52 AM, [email protected] wrote: > > [...] > >> If I set up a single machine to hammer the fileserver with IOzone >> I see >> something like 50,000 IOPS but if all four machines are hammering the >> filesystem concurrently we got it up to 180,000 IOPS. > > I wouldn't recommend IOzone for this sort of testing. Its not a very > good load generator, and it has a tendency to report things which are > not actually seen at the hardware level. I'd noticed this some years > ago, when running some of our benchmark testing on these units, > that an > entire IOzone benchmark completed with very few activity lights > going on > the disks. Which suggested that the test was happily entirely cached, > and I was running completely within cache. > > Use fio. > > Second, are the disks behind the NFS/ZFS server solid state, ram disk, > or spinning rust? > >> Can anyone tell me what might be the bottleneck on the single >> machines? >> Why can I not get 180,000 IOPS when running on a single machine. > > 50k IOPs x 16k/IOP = 819 MB/s
10 gbit TCP limit i'd guess > > 180k IOPs x 16k/IOP = 2949 MB/s (close to pragmatic limits on QDR) > 40 gbit QDR yeah the actual data goes through RDMA here and maybe through TCP using mysql. > Some observations ... these don't sound like disk subsystems. A > 15k RPM > drive will give you ~300 IOPs. To get 50k IOPs, you would need 167 > disk > drives per machine, operating in a best performance case scenario > (RAID0 > or JBOD). To get 180k IOPs, you'd need 600x 15k RPM disks. I am > guessing you don't have that. > > Are you asking why a single machine cannot fill your QDR bandwidth? > > I'd recommend running traces on the individual machines to see where > things are getting lost. One you have the traces, post em, and see if > people can help. > > -- > Joseph Landman, Ph.D > Founder and CEO > Scalable Informatics Inc. > email: [email protected] > web : http://scalableinformatics.com > http://scalableinformatics.com/sicluster > phone: +1 734 786 8423 x121 > fax : +1 866 888 3112 > cell : +1 734 612 4615 > > _______________________________________________ > Beowulf mailing list, [email protected] sponsored by Penguin > Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, [email protected] sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
