> -----Original Message-----
> From: Patrick Geoffray [mailto:[EMAIL PROTECTED]
> Sent: Thursday, April 19, 2007 6:47 PM
> To: Steffen Persvold; Beowulf Mailing List
> Subject: Re: [Beowulf] 1.2 us IB latency?
>
> [EMAIL PROTECTED] wrote:
> >> Back then we were struggling with PIO transfers and how they were
> >> treated in the CPU/North bridge (write combining and all that). I
> >> believe this might still be an issue, correct ?
>
> WC is well implemented on Opteron, it will aggregate consecutive PIO
> writes at 16, 32 and 64 Bytes smoothly. On Intel processors, this is
> more painful: WC is only 64 Bytes. If you flush the WC buffer with
less
> than 64 bytes in it, you will see multiple 8-byte PIO writes, and not
> always in order.
Yeah, that rings a bell... :)
So I'm guessing, both Myrinet MX and Qlogic Infinipath (confirmed) is
using PIO for "small" messages. Are we sure that Mellanox ConnectX
doesn't ? It seems they would have to in order to get the 1.2us numbers.
There's nothing that stops them from doing :
verbs_post_rdma_write() {
...
if (msg_size < MAX_PIO_TRESHOLD) {
copybuffertoremotewithpio();
} else {
setupdmaengine();
}
...
}
Or something of that order.. However, they claim that it's "fully
offloaded", so I'm not sure..
>
> > cases we can manipulate the mtrrs after boot to fix this. Getting
> > formal support for PAT in the Linux kernel is the long-term fix for
> > this.
>
> It's interesting to note that most current OSes have native PAT
support,
> except Linux. Even Windows does it well :-)
>
Hmm, I seem to remember having PAT support working fine with SCI on
Linux a couple of years ago. We started using PAT on x86_64 because of
the nightmare with MTRR and memory holes/overlapping regions (BIOSes
never seemed to get it right) especially on boxes with >4GB memory
(which became more and more common with the introduction of x86_64).
Cheers,
Steffen Persvold
Technical Director Americas
tel. 508-281-7100 x401
fax. 508-281-7171
http://www.scali.com/
Higher Performance Computing
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
