Hi Bill, Sorry for the late reply. As I greped for pam_limits.so, I see [root@rocks7 ~]# grep -r pam_limits.so /etc/pam.d/ /etc/pam.d/sudo:session required pam_limits.so /etc/pam.d/runuser:session required pam_limits.so /etc/pam.d/sudo-i:session required pam_limits.so /etc/pam.d/system-auth-ac:session required pam_limits.so /etc/pam.d/fingerprint-auth-ac:session required pam_limits.so /etc/pam.d/smartcard-auth-ac:session required pam_limits.so /etc/pam.d/password-auth-ac:session required pam_limits.so [root@rocks7 ~]# grep -r UsePAM /etc/slurm/ /etc/slurm/slurm.conf:#UsePAM=
So, UsePAM has not been set. So, slurm shouldn't limit anything. Is that correct? however, I see that slurm limits the virtual memory size [hamid@rocks7 case1_source2]$ cat slurm_script.sh #!/bin/bash #SBATCH --job-name=hvacSteadyFoam #SBATCH --output=hvacSteadyFoam.log #SBATCH --ntasks=32 #SBATCH --time=100:00:00 #SBATCH --mem=64000M ulimit -a mpirun hvacSteadyFoam -parallel [hamid@rocks7 case1_source2]$ sbatch slurm_script.sh Submitted batch job 55 [hamid@rocks7 case1_source2]$ ssh compute-0-3 Warning: untrusted X11 forwarding setup failed: xauth key data not generated Last login: Sun Apr 15 23:11:15 2018 from rocks7.local Rocks Compute Node Rocks 7.0 (Manzanita) Profile built 19:21 11-Apr-2018 Kickstarted 19:37 11-Apr-2018 [hamid@compute-0-3 ~]$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 256712 max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 4096 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [hamid@compute-0-3 ~]$ exit logout Connection to compute-0-3 closed. [hamid@rocks7 case1_source2]$ cat hvacSteadyFoam.log core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 256712 max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) 65536000 open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 4096 virtual memory (kbytes, -v) 72089600 file locks (-x) unlimited [hamid@rocks7 case1_source2]$ Regards, Mahmood On Mon, Apr 16, 2018 at 12:02 AM, Bill Barth <bba...@tacc.utexas.edu> wrote: > Specifying --mem to Slurm only tells it to find a node that has that much, > not to enforce a limit as far as I know. That node has that much so it finds > it. You probably want to enable UsePAM and setup the pam.d slurm files and > /etc/security/limits.conf to keep users under the 64000MB physical memory > that the node has (minus some padding for the OS, etc.). IS UsePAM enabled in > your slurm.conf, maybe that’s doing it. > > Best, > Bill. > > -- > Bill Barth, Ph.D., Director, HPC > bba...@tacc.utexas.edu | Phone: (512) 232-7069 > Office: ROC 1.435 | Fax: (512) 475-9445 >
