Hi Patrick, I have been quite vocal in the past against the merit of high packet rate, > but I have learned to appreciate it. There is a set of applications that can > benefit from it, especially at scale. Actually, packet rate is much more > important outside of HPC (where application throughput is what money buys). >
The 'especially at scale' bit seems to me to be the critical issue - weighing the price/performance as the ratio of small-scale to large-scale runs changes, assuming that an adapter with better large-scale performance has a significant cost differential. If only we knew what that ratio would be ahead of time, this would be easier. :-) However, I would pay attention to a different problem with many-core > machines. Each user-space process uses a dedicated set of NIC resources, and > this can be a problem with 48 cores per node (it affects all vendors, even > if they swear otherwise). You may want to consider multiple NICs, unless you > know that only a subset of the cores are communicating through the network > (hybrid MPI/Open-MP model for example) or that the multiplexing overhead is > not a big deal for you. Well, clearly we hope to move more towards hybrid methods -all that's old is new again?- but, again, it's *currently* hard to quantify the variables involved. Time to transition, performance differences, user effort, etc. But getting back to a technical vein, is the multiplexing an issue due to atomic locks on mapped memory pages? Or just because each copy reserves its own independent buffers? What are the critical issues? > You need PCIe Gen2 x16 to saturate a 32 Gb/s QDR link. There is no such NIC > on the market AFAIK (only Gen1 x16 or Gen2 x8). But even then, you won't > have any PCIe bandwidth left to drive a second port on the same NIC. There > may be other rationales for a second port, but bandwidth is not one of them. > I thought PCIe Gen2 x 8 @ 500 Mhz gives 8GB/s? I know there are 250 and 500 Mhz variants in addition to the lane sizes, so while a 250 Mhz x8 link wouldn't provide enough bandwidth to a dual-port card, the 500 Mhz one should. But I'm woefully out of date on my hardware knowledge, it seems. Of course, EDR (eight data-rate) IB is on the roadmap for 2011, so if we're in no rush that could help, too. Cheers, - Brian
_______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
