We have the same issue see:
* https://bugs.schedmd.com/show_bug.cgi?id=8527
* temporary fix we switched back to DefMemPerCpu
regards
On 26/03/2020 16:42, Wayne Hendricks wrote:
When using 20.02/cons_tres and defining DefMemPerGPU, jobs submitted
that request GPUs without defining “—mem” will not run more than one job
per node. I can see where it is allocating the correct amount of memory
for the job per GPUs requested, but no other jobs will run on the node.
If a value for “—mem” is defined, other jobs will share the node. Is
this the expected behavior? I understand that when jobs do not request
memory it is assumed that the job is running on the whole node, but here
when we are asking for GPUs there is a default memory set with
DefMemPerGPU and it seems this is not being taken into account. Let me
know if there is a reason for this behavior or if there is another way
to set the default job memory.
Config:
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory,CR_CORE_DEFAULT_DIST_BLOCK
PartitionName=p100 Nodes=ucs480 OverSubscribe=FORCE:4 DefCpuPerGPU=20
DefMemPerGPU=125000 Default=YES MaxTime=INFINITE State=UP
Node and job state when two jobs submitted with each requesting half the
GPUs (no —mem specified):
CfgTRES=cpu=80,mem=500000M,billing=80
AllocTRES=cpu=40,mem=250000M
Job state:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
872 p100 test-s6 wayne.he PD 0:00 1 (Resources)
871 p100 test-s5 wayne.he R 0:03 1 ucs480
--
--
Bas van der Vlies
| Operations, Support & Development | SURFsara | Science Park 140 | 1098 XG
Amsterdam
| T +31 (0) 20 800 1300 | bas.vandervl...@surfsara.nl | www.surfsara.nl |
