Hi Andrew, andrew holway wrote:
I'm trying to find out about the effects of virtualisation on high performance interconnects. Effects on latency and bandwidth.
Virtualization has virtually (pun intended) no effect on OS-bypass (user-level talking directly to the virtualized hardware) operations.
However, it does for all non-OS-bypass operations. The most important of these is memory registration, because it's not a one-time cost as the Ohio State papers have always portrayed it. For applications that do not effectively reuse buffers, or when you cannot hijack the native malloc/munmap/sbrk symbols, or when you cannot guarantee that the memory is going to stay around (OS buffer cache for example), the memory registration (and deregistration !) is going to happen in the critical path.
If the memory registration is ~150us for 1 page, then adding 50us of hypervisor/IOMMU overhead seems relatively cheap. If the memory registration is ~2us for 1 page, then it's bad. To give you an idea of the impact on latency/bandwidth, the communication time (DMA + wire + DMA) for a 4K page is in the order of 10us at 10Gb/s.
Patrick _______________________________________________ Beowulf mailing list, [email protected] To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
