Suksmandhira,
That qos specifies a walltime, cpu, and memory limit. From the job script, it
appears you are within the cpu limit. But, the job script does not specify
walltime nor memory and your squeue output is not showing those values (or cpu)
for the job.
'scontrol show job=JOBID' will show it all values. Added flags=DenyOnLimit to
the qos will reject the job when it is over the limit of a QOS, hopefully so
there are not jobs that will never run sitting in queue.
-b
On 11/7/19 9:37 PM, Sukman wrote:
Hi all,
I am currently having a problem in limiting the number of CPU used for running
a job.
I tried to limit the CPU to just only 2 from the maximum 56.
But, when I run the job, using only 1 CPU, the QOS has been reached already.
When I set the CPU to 56, the job runs finely.
Does anyone have any suggestion regarding this problem?
Following is the details of the problem.
My node has 56 cores (2sockets x 28cores).
I configured already slurm.conf by enabling the qos/limit enforcement.
#slurm.conf
AccountingStorageEnforce=qos,limits
For QOS itself, I just tried applying a simple limit-CPU number to be 2.
#QOS
sacctmgr show qos where Name=normal_compute
format=Name,Priority,UsageFactor,MaxWall,MaxTRESPU
Name Priority UsageFactor MaxWall MaxTRESPU
---------- ---------- ----------- ----------- -------------
normal_co+ 10 1.000000 00:01:00 cpu=2,mem=1G
I then applied the QOS to a specific user, sukman.
#QOS-defined user
sacctmgr list association where User=sukman format=User,QOS,
User QOS
---------- --------------------
sukman normal_compute
Then, I tried to run a simple bash command, hostname, by just using 1 node, 1
task, and 1 CPU
#!/bin/bash
#SBATCH --job-name=hostname
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --nodelist=cn110
srun hostname
However, the QOS has been reached already.
squeue
JOBID PARTITION NAME USER ST TIME NODES
NODELIST(REASON)
68 defq hostname sukman PD 0:00 1
(QOSMaxCpuPerUserLimit)
When I change the CPU limit to the max cores number in a server, 56 cores
sacctmgr show qos where Name=normal_compute
format=Name,Priority,UsageFactor,MaxWall,MaxTRESPU
Name Priority UsageFactor MaxWall MaxTRESPU
---------- ---------- ----------- ----------- -------------
normal_co+ 10 1.000000 00:01:00 cpu=56,mem=1G
the script runs perfectly.
cat slurm-68.out
cn110
--------------------------------
Suksmandhira H
ITB Indonesia
