Re: [slurm-users] 19.05 and GPUs vs GRES

2019-09-05 Thread Christopher Samuel
On 9/5/19 3:49 PM, Bill Broadley wrote: I have a user with a particularly flexible code that would like to run a single MPI job across multiple nodes, some with 8 GPUs each, some with 2 GPUs. Perhaps they could just specify a number of tasks with cpus per task, mem per task and GPUs per task

Re: [slurm-users] 19.05 and GPUs vs GRES

2019-09-05 Thread Christopher Samuel
On 8/13/19 10:44 PM, Barbara Krašovec wrote: We still have the gres configuration, users have their workload scripted and some still use sbatch with gres. Both options work. I missed this before Barbara, sorry - that's really good to know that the options aren't mutually exclusive, thank you!

Re: [slurm-users] 19.05 and GPUs vs GRES

2019-09-05 Thread Bill Broadley
Anyone know if the new GPU support allows having a different number of GPUs per node? I found: https://www.ch.cam.ac.uk/computing/slurm-usage Which mentions "SLURM does not support having varying numbers of GPUs per node in a job yet." I have a user with a particularly flexible code that would

Re: [slurm-users] 19.05 and GPUs vs GRES

2019-08-13 Thread Barbara Krašovec
We have SLURM 19.05. and implemented the cons_tres scheduling type. It does work only by specifying the --gpus-per-node when submitting the job. And there are many more options. I found this presentation to be quite informative: https://slurm.schedmd.com/SLUG18/cons_tres.pdf We still have the gr

Re: [slurm-users] 19.05 and GPUs vs GRES

2019-08-13 Thread Chris Samuel
On 13/8/19 1:36 pm, Christopher Benjamin Coffey wrote: Thanks for that Chris!:) No worries Chris! Sounds like other than the new requests for gpu specifics, things should just work when upgrading to 19.05 as slurm is likely backwards compatible with the previous setup gres stuff. Yup, we

Re: [slurm-users] 19.05 and GPUs vs GRES

2019-08-13 Thread Christopher Benjamin Coffey
Thanks for that Chris! :) Sounds like other than the new requests for gpu specifics, things should just work when upgrading to 19.05 as slurm is likely backwards compatible with the previous setup gres stuff. Best, Chris — Christopher Coffey High-Performance Computing Northern Arizona Universi

Re: [slurm-users] 19.05 and GPUs vs GRES

2019-08-12 Thread Chris Samuel
On Monday, 12 August 2019 11:42:48 AM PDT Christopher Benjamin Coffey wrote: > Excuse me if this has been explained somewhere, I did some searching. With > 19.05, is there any reason to have gres.conf on the GPU nodes? Is slurm > smart enough to enumerate the /dev/nvidia* devices? We are moving t

[slurm-users] 19.05 and GPUs vs GRES

2019-08-12 Thread Christopher Benjamin Coffey
Hi, Excuse me if this has been explained somewhere, I did some searching. With 19.05, is there any reason to have gres.conf on the GPU nodes? Is slurm smart enough to enumerate the /dev/nvidia* devices? We are moving to 19.05 shortly, any gotchas with GRES and GPUs? Also, I'm guessing now, the