I'm either misunderstanding how to configure the limit "MaxCPUsPerNode" or how it behaves. My desired end-state is that if a user submits a job to a partition that requests more resources (CPUs) than available on any node in that partition, the job will be immediately rejected, rather than pending with state "Resources" as it does now.
In the on-site cluster it works as expected. I'm running into this problem in our cloud clusters where I've got power management set up. When I tear out all of the cloud/power management configurations the limit works as expected. We're using the backfill scheduler and do have QOS's configured (though none of the QOS's in play have a similar limit applied). The relevant bits from the cloud-enabled configuration are: SelectType=select/cons_res SelectTypeParameters=CR_Core PartitionName=campus Default=yes DefaultTime=3-0 MaxTime=7-0 Nodes=nodef[0-69] PreemptMode=off Priority=10000 MaxCPUsPerNode=4 MaxMemPerNode=32000 State=UP PartitionName=largenode Default=no DefaultTime=1-0 MaxTime=7-0 Nodes=nodeg[0-9] PreemptMode=off Priority=10000 State=UP PartitionName=gpu Default=no DefaultTime=1-0 MaxTime=7-0 Nodes=nodek[0-9] PreemptMode=off Priority=10000 State=UP NodeName=nodef[0-69] CPUS=4 RealMemory=32768 Weight=40 State=CLOUD NodeName=nodeg[0-9] CPUS=8 RealMemory=262144 Weight=40 State=CLOUD NodeName=nodek[0-9] gres=gpu:V100-SXM2-16GB:1 CPUS=4 RealMemory=131072 Weight=40 State=CLOUD Submitting a job that exceeds that limit (e.g. `sbatch -c 12 ...`) results in a job JobId=27072660 JobName=wrap UserId=me(12345) GroupId=g_me(12345) MCS_label=N/A Priority=100012209 Nice=0 Account=hpc QOS=normal JobState=PENDING Reason=Resources Dependency=(null) Requeue=0 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=3-00:00:00 TimeMin=N/A SubmitTime=2018-11-26T07:50:27 EligibleTime=2018-11-26T07:50:27 AccrueTime=2018-11-26T07:50:27 StartTime=Unknown EndTime=Unknown Deadline=N/A PreemptTime=None SuspendTime=None SecsPreSuspend=0 LastSchedEval=2018-11-26T07:53:14 Partition=campus AllocNode:Sid=cluster-login:13257 ReqNodeList=(null) ExcNodeList=(null) NodeList=(null) NumNodes=1 NumCPUs=12 NumTasks=1 CPUs/Task=12 ReqB:S:C:T=0:0:*:* TRES=cpu=12,node=1,billing=12 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=12 MinMemoryNode=0 MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Controller log notes: slurmctld: _build_node_list: No nodes satisfy JobId=27072660 requirements in partition campus slurmctld: _slurm_rpc_submit_batch_job: JobId=27072660 InitPrio=100012209 usec=1037 So there's either the possibility that there's a bug in Slurm, or I'm misunderstanding how this limit is supposed to work. Thanks for looking at this- any suggestions greatly appreciated. Michael
