Hi Bjørn-Helge, that's very similar to what we did as well in order to avoid confusion with Core vs. Threads vs. CPU counts when Hyperthreading is kept enabled in the BIOS.
Adding CPUs=<core_count> (not <thread_count>) will tell Slurm to only schedule physical cores. We have SelectType=select/cons_res SelectTypeParameters=CR_Core_Memory and NodeName=DEFAULT CPUs=48 Sockets=2 CoresPerSocket=24 ThreadsPerCore=2 This is for compute nodes that have 2 sockets, 2 x 24 physical cores with hyperthreading enabled in the BIOS. (Although, in general, we do not encourage our users to make use of hyperthreading, we have decided to leave it enabled in the BIOS as there are some corner cases that are known to benefit from hyperthreading.) With this setting Slurm does also show the total physical core counts instead of the thread counts and also treats the --mem-per-cpu option as "--mem-per-core" which is in our case what most of our users expect. As to the number of tasks spawned with `--cpus-per-task=1´, I think this is intended behavior. The following sentence from the srun manpage is probably relevant: -c, --cpus-per-task=<ncpus> If -c is specified without -n, as many tasks will be allocated per node as possible while satisfying the -c restriction. In our configuration, we allow multiple jobs to run for the same user on a node (ExclusiveUser=yes) and we get $ srun -c 1 echo foo | wc -l 1 $ However, in case of CPUs=<thread_count> instead of CPUs=<core_count>, I guess, this would have been 2 lines of output, because the smallest unit to schedule for a job is 1 physical core which allows 2 tasks to run with hyperthreading enabled. In case of exclusive node allocation for jobs (i.e. no node sharing allowed) Slurm would give all cores of a node to the job which allows even more tasks to be spawned: $ srun --exclusive -c 1 echo foo | wc -l 48 $ 48 lines correspond exactly to the number of physical cores on the node. Again, with CPUs=<thread_count> instead of CPUs=<core_count>, I would expect 2 x 48 = 96 lines of output, but I did not test that. Best regards Jürgen * Bjørn-Helge Mevik <b.h.me...@usit.uio.no> [220325 08:49]: > For what it's worth, we have a similar setup, with one crucial > difference: we are handing out physical cores to jobs, not hyperthreads, > and we are *not* seeing this behaviour: > > $ srun --cpus-per-task=1 -t 10 --mem-per-cpu=1g -A nn9999k -q devel echo foo > srun: job 5371678 queued and waiting for resources > srun: job 5371678 has been allocated resources > foo > $ srun --cpus-per-task=3 -t 10 --mem-per-cpu=1g -A nn9999k -q devel echo foo > srun: job 5371680 queued and waiting for resources > srun: job 5371680 has been allocated resources > foo > > We have > > SelectType=select/cons_tres > SelectTypeParameters=CR_CPU_Memory > > and node definitions like > > NodeName=DEFAULT CPUs=40 Sockets=2 CoresPerSocket=20 ThreadsPerCore=2 > RealMemory=182784 Gres=localscratch:330G Weight=1000 > > (so we set CPUs to the number of *physical cores*, not *hyperthreads*). > > -- > Regards, > Bjørn-Helge Mevik, dr. scient, > Department for Research Computing, University of Oslo > -- Jürgen Salk Scientific Software & Compute Services (SSCS) Kommunikations- und Informationszentrum (kiz) Universität Ulm Telefon: +49 (0)731 50-22478 Telefax: +49 (0)731 50-22471
