No worries, No, we don’t have any OS level settings, only “allowed_devices.conf” which just has /dev/random, /dev/tty and stuff like that.
But I think this could be the culprit, check out man page for cgroup.conf AllowedRAMSpace=100 I would just leave these four: CgroupAutomount=yes ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes Vlad. From: slurm-users <slurm-users-boun...@lists.schedmd.com> On Behalf Of Boris Yazlovitsky Sent: Thursday, June 22, 2023 5:40 PM To: Slurm User Community List <slurm-users@lists.schedmd.com> Subject: Re: [slurm-users] [EXT] --mem is not limiting the job's memory APL external email warning: Verify sender slurm-users-boun...@lists.schedmd.com<mailto:slurm-users-boun...@lists.schedmd.com> before clicking links or attachments thank you Vlad - looks like we have the same yes's Do you remember if you had to make any settings on the OS level or in the kernel to make it work? -b On Thu, Jun 22, 2023 at 5:31 PM Ozeryan, Vladimir <vladimir.ozer...@jhuapl.edu<mailto:vladimir.ozer...@jhuapl.edu>> wrote: Hello, We have the following configured and it seems to be working ok. CgroupAutomount=yes ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes Vlad. From: slurm-users <slurm-users-boun...@lists.schedmd.com<mailto:slurm-users-boun...@lists.schedmd.com>> On Behalf Of Boris Yazlovitsky Sent: Thursday, June 22, 2023 4:50 PM To: Slurm User Community List <slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>> Subject: Re: [slurm-users] [EXT] --mem is not limiting the job's memory APL external email warning: Verify sender slurm-users-boun...@lists.schedmd.com<mailto:slurm-users-boun...@lists.schedmd.com> before clicking links or attachments Hello Vladimir, thank you for your response. this is the cgroups.conf file: CgroupAutomount=yes ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes ConstrainSwapSpace=yes MaxRAMPercent=90 AllowedSwapSpace=0 AllowedRAMSpace=100 MemorySwappiness=0 MaxSwapPercent=0 /etc/default/grub: GRUB_DEFAULT=0 GRUB_TIMEOUT_STYLE=hidden GRUB_TIMEOUT=0 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT="" GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0 cgroup_enable=memory swapaccount=1" what other cgroup settings need to be set? && thank you! -b On Thu, Jun 22, 2023 at 4:02 PM Ozeryan, Vladimir <vladimir.ozer...@jhuapl.edu<mailto:vladimir.ozer...@jhuapl.edu>> wrote: --mem=5G. Should allocate 5G of memory per node. Are your cgroups configured? From: slurm-users <slurm-users-boun...@lists.schedmd.com<mailto:slurm-users-boun...@lists.schedmd.com>> On Behalf Of Boris Yazlovitsky Sent: Thursday, June 22, 2023 3:28 PM To: slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> Subject: [EXT] [slurm-users] --mem is not limiting the job's memory APL external email warning: Verify sender slurm-users-boun...@lists.schedmd.com<mailto:slurm-users-boun...@lists.schedmd.com> before clicking links or attachments Running slurm 22.03.02 on Ubunutu 22.04 server. Jobs submitted with --mem=5g are able to allocate an unlimited amount of memory. how to limit on the job submission level how much memory it can grab? thanks, and best regards! Boris
