Re: [slurm-users] slurm node weights

2019-09-08 Thread Ole Holm Nielsen
You should be able to assign node weights to accommodate your prioritization wishes. I've summarized this setting in my Slurm Wiki page: https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#node-weight I hope this helps. /Ole On 9/5/19 5:48 PM, Douglas Duckworth wrote: Hello We added some

Re: [slurm-users] slurm node weights

2019-09-05 Thread Marcus Boden
Hello Doug, tp quote the slurm.conf page: It would be preferable to allocate smaller memory nodes rather than larger memory nodes if either will satisfy a job's requirements. So I guess the idea is, that if a smaller node satisfies all requirements, why 'waste' a bigger one for it? It makes sense

Re: [slurm-users] slurm node weights

2019-09-05 Thread Merlin Hartley
I believe this is so that small jobs will naturally go on older, slower nodes first - leaving the bigger,better ones for jobs that actually need them. Merlin -- Merlin Hartley IT Support Engineer MRC Mitochondrial Biology Unit University of Cambridge Cambridge, CB2 0XY United Kingdom > On 5 Sep

Re: [slurm-users] slurm node weights

2019-09-05 Thread Brian Andrus
The intention there is to pack jobs on the smallest node that can handle the job. This way jobs that only need 1 cpu don't take it from a 64-core node unless it has to, leaving that one available for that 64-core job. It really boils down to what you want to happen, which will vary with eac

[slurm-users] slurm node weights

2019-09-05 Thread Douglas Duckworth
Hello We added some newer Epyc nodes, with NVMe scratch, to our cluster and so want jobs to run on these over others. So we added "Weight=100" to the older nodes and left the new ones blank. So indeed, ceteris paribus, srun reveals that the faster nodes will accept jobs over older ones. We h

Re: [slurm-users] Slurm node weights

2019-07-27 Thread Chris Samuel
On Thursday, 25 July 2019 10:02:05 AM PDT Ryan Novosielski wrote: > My understanding is that the topology plug-in will overrule this, and that > may or may not be a problem depending on your environment. Note you can set: TopologyParam=TopoOptional so that its only invoked if a job specifically

Re: [slurm-users] Slurm node weights

2019-07-27 Thread Chris Samuel
On Thursday, 25 July 2019 4:53:28 AM PDT David Baker wrote: > Thank you for the replies. We're running an early version of Slurm 18.08 and > it does appear that the node weights are being ignored re the bug. I would recommend trying to get to 18.08.7, the current 18.08 release if you can. Ther

Re: [slurm-users] Slurm node weights

2019-07-25 Thread Ryan Novosielski
My understanding is that the topology plug-in will overrule this, and that may or may not be a problem depending on your environment. I had a ticket in to SchedMD about this, because it looked like our nodes were getting allocated in the exact reverse order. I suspected this was because our high

Re: [slurm-users] Slurm node weights

2019-07-25 Thread Sarlo, Jeffrey S
users-boun...@lists.schedmd.com] On Behalf Of David Baker Sent: Thursday, July 25, 2019 8:30 AM To: Slurm User Community List Subject: Re: [slurm-users] Slurm node weights Hi Jeff, Thank you for these details. so far we have never implemented any Slurm fixes. I suspect the node weights feature is quite imp

Re: [slurm-users] Slurm node weights

2019-07-25 Thread David Baker
the system to be at risk. Or alternatively, do we need to arrange downtime, etc? Best regards, David From: slurm-users on behalf of Sarlo, Jeffrey S Sent: 25 July 2019 13:04 To: Slurm User Community List Subject: Re: [slurm-users] Slurm node weights Th

Re: [slurm-users] Slurm node weights

2019-07-25 Thread Sarlo, Jeffrey S
Community List Subject: Re: [slurm-users] Slurm node weights Hello, Thank you for the replies. We're running an early version of Slurm 18.08 and it does appear that the node weights are being ignored re the bug. We're experimenting with Slurm 19*, however we don't expect to

Re: [slurm-users] Slurm node weights

2019-07-25 Thread David Baker
anyone know if there any fix or alternative strategy that might help us to achieve the same result? Best regards, David From: slurm-users on behalf of Sarlo, Jeffrey S Sent: 25 July 2019 12:26 To: Slurm User Community List Subject: Re: [slurm-users] Slu

Re: [slurm-users] Slurm node weights

2019-07-25 Thread Sarlo, Jeffrey S
Subject: Re: [slurm-users] Slurm node weights Hello, As an update I note that I have tried restarting the slurmctld, however that doesn't help. Best regards, David From: slurm-users on behalf of David Baker Sent: 25 July 2019 11:47:35 To: slurm-

Re: [slurm-users] Slurm node weights

2019-07-25 Thread Sean Crosby
Hi David, What does: scontrol show node orange01 scontrol show node orange02 show? Just to see if there's a default node weight hanging around, and if your weight changes have been picked up. Sean -- Sean Crosby Senior DevOpsHPC Engineer and HPC Team Lead | Research Platform Services Research

Re: [slurm-users] Slurm node weights

2019-07-25 Thread Viviano, Brad
Did you try assigning a weight to orange[02-03]? I've found with Slurm it's better to be exact in your slurm.conf and don't relay on the defaults. This is what I use on my cluster: NodeName=r1n[01-32] weight=10 CoresPerSocket=16 Sockets=2 ThreadsPerCore=1 RealMemory=257000 NodeName=r1n[33-64

Re: [slurm-users] Slurm node weights

2019-07-25 Thread David Baker
Hello, As an update I note that I have tried restarting the slurmctld, however that doesn't help. Best regards, David From: slurm-users on behalf of David Baker Sent: 25 July 2019 11:47:35 To: slurm-users@lists.schedmd.com Subject: [slurm-users]

[slurm-users] Slurm node weights

2019-07-25 Thread David Baker
Hello, I'm experimenting with node weights and I'm very puzzled by what I see. Looking at the documentation I gathered that jobs will be allocated to the nodes with the lowest weight which satisfies their requirements. I have 3 nodes in a partition and I have defined the nodes like so.. Node