Re: [Beowulf] DMA Memory Mapping Question

2007-02-21 Thread Greg Lindahl
On Wed, Feb 21, 2007 at 11:06:50PM -0500, Patrick Geoffray wrote: > Time for dreaming about an MPI ABI :-) Ssh! It isn't April Fools yet! -- greg ___ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe)

Re: [Beowulf] DMA Memory Mapping Question

2007-02-21 Thread Chris Samuel
On Thu, 22 Feb 2007, Patrick Geoffray wrote: > Hi Chris, G'day Patrick! > Chris Samuel wrote: > > We occasionally get users who manage to use up all the DMA memory that is > > addressable by the Myrinet card through the Power5 hypervisor. > > The IOMMU limit set by the hypervisor varies dependin

Re: [Beowulf] DMA Memory Mapping Question

2007-02-21 Thread Patrick Geoffray
Hi Chris, Chris Samuel wrote: We occasionally get users who manage to use up all the DMA memory that is addressable by the Myrinet card through the Power5 hypervisor. The IOMMU limit set by the hypervisor varies depending on the machine, the hypervisor version and the phase of the moon. Somet

Re: [Beowulf] DMA Memory Mapping Question

2007-02-21 Thread Chris Samuel
On Thu, 22 Feb 2007, Scott Atchley wrote: Hello Scott! > Isn't this in hex? If so, it would be 4096 bytes. I do not use GM   > much and I do not know what this is. I just loaded GM on one node and   > with no GM processes running except the mapper, I have a similar   > entry (at a different addre

Re: [Beowulf] DMA Memory Mapping Question

2007-02-21 Thread Chris Samuel
On Thu, 22 Feb 2007, Chris Samuel wrote: > Through various firmware and driver tweaks (thanks to both IBM and Myrinet) > we've gotten that limit up to almost 1GB and then we use an undocumented > environment variable (GMPI_MAX_LOCKED_MBYTE) to say only use 248MB of that > per process (as we've got

Re: [Beowulf] DMA Memory Mapping Question

2007-02-21 Thread Scott Atchley
On Feb 21, 2007, at 7:45 PM, Chris Samuel wrote: Hi folks, We've got an IBM Power5 cluster running SLES9 and using the GM drivers. We occasionally get users who manage to use up all the DMA memory that is addressable by the Myrinet card through the Power5 hypervisor. Through various fir

[Beowulf] DMA Memory Mapping Question

2007-02-21 Thread Chris Samuel
Hi folks, We've got an IBM Power5 cluster running SLES9 and using the GM drivers. We occasionally get users who manage to use up all the DMA memory that is addressable by the Myrinet card through the Power5 hypervisor. Through various firmware and driver tweaks (thanks to both IBM and Myrinet)