Re: [slurm-users] Jobs waiting while plenty of cpu and memory available

2019-07-10 Thread Edward Ned Harvey (slurm)
> From: slurm-users On Behalf Of > Andy Georges > Sent: Wednesday, July 10, 2019 3:57 AM > > EnforcePartLimits=YES Hmmm It's already yes... I assume it's not case-sensitive...

Re: [slurm-users] Jobs waiting while plenty of cpu and memory available

2019-07-10 Thread Andy Georges
Hi, > So here's something funny. One user submitted a job that requested 60 cpu's > and 40M of memory. Our largest nodes in that partition have 72 cpu's and > 256G of memory. So when a user requests 400G of ram, what would be good > behavior? I would like to see slurm reject the job, "job i

Re: [slurm-users] Jobs waiting while plenty of cpu and memory available

2019-07-09 Thread Burian, John
limits is a User Education Opportunity. John From: slurm-users on behalf of "Thomas M. Payerle" Reply-To: Slurm User Community List Date: Tuesday, July 9, 2019 at 10:23 AM To: Slurm User Community List Subject: Re: [slurm-users] Jobs waiting while plenty of cpu and memory ava

Re: [slurm-users] Jobs waiting while plenty of cpu and memory available

2019-07-09 Thread Edward Ned Harvey (slurm)
> From: slurm-users On Behalf Of > Thomas M. Payerle > Sent: Tuesday, July 9, 2019 10:23 AM > > Do you have backfill enabled?  This can help in many cases. Yup - I checked for backfill yesterday. It's backfill. > If the job with highest priority is quite wide, Slurm will reserve resources > f

Re: [slurm-users] Jobs waiting while plenty of cpu and memory available

2019-07-09 Thread Thomas M. Payerle
You can use squeue to see the priority of jobs. I believe it normally shows jobs in order of priority, even though does not display priority. If you want to see actual priority, you need to request it in the format field. I typically use squeue -o "%.18i %.12a %.6P %.8u %.2t %.8m %.4D %.4C %12l

Re: [slurm-users] Jobs waiting while plenty of cpu and memory available

2019-07-09 Thread Edward Ned Harvey (slurm)
> From: slurm-users On Behalf Of > Ole Holm Nielsen > Sent: Tuesday, July 9, 2019 2:36 AM > > When some jobs are pending with Reason=Priority this means that other > jobs with a higher priority are waiting for the same resources (CPUs) to > become available, and they will have Pending=Resources i

Re: [slurm-users] Jobs waiting while plenty of cpu and memory available

2019-07-08 Thread Ole Holm Nielsen
Hi Edward, The squeue command tells you about job status. You can get extra information using format options (see the squeue man-page). I like to set this environment variable for squeue: export SQUEUE_FORMAT="%.18i %.9P %.6q %.8j %.8u %.8a %.10T %.9Q %.10M %.10V %.9l %.6D %.6C %m %R" Wh

[slurm-users] Jobs waiting while plenty of cpu and memory available

2019-07-08 Thread Edward Ned Harvey (slurm)
I have a cluster, where I submit a bunch (600) jobs, but the cluster only runs about 20 at a time. By using pestat, I can see there are a bunch of systems with plenty of available cpu and memory. Hostname Partition Node Num_CPU CPUload Memsize Freemem Sta