Re: [slurm-users] jobs stuck in ReqNodeNotAvail,

2017-11-30 Thread Christian Anthon
I now realised I probably need some kind of job preemption to make things work the way I want them to. I'll take a look at how slurm does that. Cheers, Christian. On 30-11-2017 13:29, Chris Samuel wrote: On Thursday, 30 November 2017 9:40:53 PM AEDT Christian Anthon wrote: The queue has a t

Re: [slurm-users] jobs stuck in ReqNodeNotAvail,

2017-11-30 Thread Chris Samuel
On Thursday, 30 November 2017 9:40:53 PM AEDT Christian Anthon wrote: > The queue has a ton of of single-core jobs and somebody submits a high > priority multi-core job, will the mulit-core job not run before all > single-core jobs are done or will slurm free up a node? I can see you are weightin

Re: [slurm-users] jobs stuck in ReqNodeNotAvail,

2017-11-30 Thread Christian Anthon
Okay, how is slurm handling the following situation: The queue has a ton of of single-core jobs and somebody submits a high priority multi-core job, will the mulit-core job not run before all single-core jobs are done or will slurm free up a node? Cheers, Christian. On 30-11-2017 07:57, Ch

Re: [slurm-users] jobs stuck in ReqNodeNotAvail,

2017-11-29 Thread Chris Samuel
On Thursday, 30 November 2017 2:21:36 AM AEDT Christian Anthon wrote: > The nodes are fully allocated in terms of memory, but not all cpu > resources are consumed I suspect that's your problem, the job wants 16 cores on a single node and 32GB of RAM free. If you've got no RAM free it's not goi

Re: [slurm-users] jobs stuck in ReqNodeNotAvail,

2017-11-29 Thread Christian Anthon
Thanks, I believe the user must have resubmitted the job, hence the updated id. Cheers, Christian JobId=6986 JobName=Morgens UserId=ferro(2166) GroupId=ferro(22166) MCS_label=N/A Priority=1031 Nice=0 Account=rth QOS=normal JobState=PENDING Reason=ReqNodeNotAvail,_UnavailableNodes: Depen

Re: [slurm-users] jobs stuck in ReqNodeNotAvail,

2017-11-29 Thread Merlin Hartley
damn autocorrect - I meant: # scontrol show job 6982 -- Merlin Hartley Computer Officer MRC Mitochondrial Biology Unit Cambridge, CB2 0XY United Kingdom > On 29 Nov 2017, at 16:08, Merlin Hartley > wrote: > > Can you give us the output of > # control show job 6982 > > Could be an issue wi

Re: [slurm-users] jobs stuck in ReqNodeNotAvail,

2017-11-29 Thread Merlin Hartley
Can you give us the output of # control show job 6982 Could be an issue with requesting too many CPUs or something… Merlin -- Merlin Hartley Computer Officer MRC Mitochondrial Biology Unit Cambridge, CB2 0XY United Kingdom > On 29 Nov 2017, at 15:21, Christian Anthon wrote: > > Hi, > > I ha

[slurm-users] jobs stuck in ReqNodeNotAvail,

2017-11-29 Thread Christian Anthon
Hi, I have a problem with a newly setup slurm-17.02.7-1.el6.x86_64 that jobs seems to be stuck in ReqNodeNotAvail:   6982 panic  Morgens    ferro PD   0:00 1 (ReqNodeNotAvail, UnavailableNodes:)   6981 panic SPEC    ferro PD   0:00 1 (ReqNodeNotAva