Yeah, I saw that too, which is part of why I was confused.
It appears to me as if, at the very least, the kernel driver and OFED have two different recommendations. The CentOS kernel driver (from CentOS 7.6 at least, but likely longer) seems to set log_mtts_per_seg to 3 on all of my nodes. You can see from my earlier reply that this driver already takes total memory into account. OFED on the other hand (I was looking at a VM with OFED 4.5 on it — have to confirm the host version, etc.) seems to set that value to 0 by default. So I’m not a ton less confused, but reasonably confident the right move on the CentOS kernel driver is not to set that value as the defaults are sane. -- ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novos...@rutgers.edu<mailto:novos...@rutgers.edu> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `' On Jul 9, 2019, at 21:25, Jonathan Engwall <engwalljonathanther...@gmail.com<mailto:engwalljonathanther...@gmail.com>> wrote: Hello, They make a recommendation: https://community.mellanox.com/s/article/howto-increase-memory-size-used-by-mellanox-adapters interesting stuff. Jonathan Engwall On Tue, Jul 9, 2019 at 1:13 PM Ryan Novosielski <novos...@rutgers.edu<mailto:novos...@rutgers.edu>> wrote: Hi all, There seems to be a whole lot of misinformation out there about the appropriate setting for the log_mtts_per_seg parameter in the mlx4 driver. Some folks suggest that it’s different between the RHEL/kernel.org<http://kernel.org> provided driver and Mellanox OFED, some suggest it’s been fixed such that the default can assign 2x the total memory on a node since at least RHEL6.6 (and so one would assume even longer ago in OFED), and some places seem to still be carrying it forward because “maybe it matters.” Is anyone here sure of the current state? I’ll probably read the source code if not, but I’d like to spare myself the hassle. -- ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novos...@rutgers.edu<mailto:novos...@rutgers.edu> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `' _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org<mailto:Beowulf@beowulf.org> sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
_______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
