Re: [slurm-users] Is it safe to convert cons_res to cons_tres on a running system?

2020-02-20 Thread Chris Samuel
On 20/2/20 2:16 pm, Nathan R Crawford wrote:   I interpret this as, in general, changing SelectType will nuke existing jobs, but that since cons_tres uses the same state format as cons_res, it should work. We got caught with just this on our GPU nodes (though it was fixed before I got to se

[slurm-users] Is it safe to convert cons_res to cons_tres on a running system?

2020-02-20 Thread Nathan R Crawford
Hi All, I have 19.05.4 and want to change SelectType from select/cons_res to select/cons_tres without losing running or pending jobs. The documentation is a bit conflicting. >From the man page: SelectType Identifies the type of resource selection algorithm to be used. Changing this value can

Re: [slurm-users] Slurm Upgrade from 17.02

2020-02-20 Thread Steven Senator (slurm-dev-list)
When upgrading to 18.08 it is prudent to add following lines into your /etc/my.cnf as per https://slurm.schedmd.com/accounting.html https://slurm.schedmd.com/SLUG19/High_Throughput_Computing.pdf (slide #6) [mysqld] innodb_buffer_pool_size=1G innodb_log_file_size=64M innodb_lock_wait_timeout=90

[slurm-users] DefMemPerGPU is ignored by scheduler and is only used after the job has started

2020-02-20 Thread Martijn Kruiten
Hi all, After upgrading to Slurm 19, we switched over to the new GPU functionality using cons_tres. However, we noticed that DefMemPerGPU is not accounted for by the scheduler. I've made a bug report here: https://bugs.schedmd.com/show_bug.cgi?id=8527 Do more people experience this? Kind regard

Re: [slurm-users] Slurm Upgrade from 17.02

2020-02-20 Thread Ricardo Gregorio
Thank you Ole/Chris/Marcus. Your input was much appreciated Ole, I was(am) basing my upgrade plan using the documentation found on the link you had sent me. In fact your wiki is always my first stop when learning/tshooting SLURM issues, even before SLURM docs pages. Excellent work, well don

Re: [slurm-users] Inconsistent cpu bindings with cpu-bind=none

2020-02-20 Thread Boden, Marcus Vincent
Hey John, thanks for the workaround. After some more testing, I've noticed that this does not occur when using Intel MPI 2018.4, only with the 2019 versions. The slurmd logs show that slurm does not set any binding in that case. Best, Marcus From: slurm-use