> The problem is that when I run with nodes=64:ppn=2 and only use -np 64 > with my openmpi (v 1.0.2 gcc compiled), it still runs fine, but when I > run with -np 65 I get megabytes of error messages and the job never > completes. The errors all look like this: > > mca_mpool_openib_register: ibv_reg_mr(0x2a96641000,1060864) > failed with error: Cannot allocate memory
.. which means you need to raise that memlock limit higher than your current 8192, which is only 8 megabytes. Most MPIs on IB lock lots of memory, more if you use more processes, and more for apps that send large messages. -- greg _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
