Hello Tina and Hermann, Thanks for your suggestions.
@Tina, I have SelectType=select/cons_res and SelectTypeParameters=CR_CPU_Memory in slurm.conf I added TaskAffinity=no, MaxRAMPercent=100 and MaxSwapPercent=0 to cgroup.conf. I will see what happens. Best! On Tue, Oct 5, 2021 at 1:33 AM Tina Friedrich <tina.friedr...@it.ox.ac.uk> wrote: > Hi Rodrigo, > > we do pretty much what you do - constrain via cgroups - and it works > fine. So I know it's possible. (I don't think I've ever twiddled the > VSizeFactor.) > > I think you also need > > SelectType=select/cons_res (or cons_tres) > SelectTypeParameters=CR_Core_Memory > > in your slurm.conf; have you got that? > > My cgroup.conf is this: > > CgroupMountpoint="/sys/fs/cgroup" > CgroupAutomount=yes > CgroupReleaseAgentDir="/etc/slurm/cgroup" > ConstrainCores=yes > TaskAffinity=no > ConstrainRAMSpace=yes > ConstrainSwapSpace=yes > ConstrainDevices=yes > AllowedDevicesFile="/etc/slurm/cgroup_allowed_devices_file.conf" > AllowedRamSpace=100 > AllowedSwapSpace=0 > MaxRAMPercent=100 > MaxSwapPercent=0 > MinRAMSpace=30 > > Tina > > On 05/10/2021 09:15, Hermann Schwärzler wrote: > > Hi Rodrigo, > > > > a possible solution is using > > > > VSizeFactor=100 > > > > in slurm.conf. > > > > With this settings, programs that try to allocate more memory than > > requested in the job's settings will fail. > > > > Be aware that this puts a limit on *virtual* memory, not on RSS. This > > might or might not be what you want as a lot of programs tend to > > allocate (a lot) more virtual memory than they really use (RSS). > > > > Regards, > > Hermann > > > > On 10/5/21 12:46 AM, Rodrigo Santibáñez wrote: > >> Hello Slurm Users, > >> > >> I'm having a hard time configuring slurm to kill jobs when they use > >> more memory than requested. Also, I can't make jobs use only RAM, and > >> some of them starts to use SWAP. > >> > >> I don't know what I'm missing. > >> > >> Thanks for your help > >> > >> slurmd -V > >> slurm 20.02.6 > >> > >> slurm.conf > >> TaskPlugin=task/affinity,task/cgroup > >> ProctrackType=proctrack/cgroup > >> > >> cgroup.conf > >> AllowedRAMSpace=100.0 > >> AllowedSwapSpace=0.0 > >> ConstrainRAMSpace=yes > >> ConstrainSwapSpace=yes > >> MemorySwappiness=0 > >> CgroupAutomount=yes > >> ConstrainCores=yes > > > > -- > Tina Friedrich, Advanced Research Computing Snr HPC Systems Administrator > > Research Computing and Support Services > IT Services, University of Oxford > http://www.arc.ox.ac.uk http://www.it.ox.ac.uk > >
