It could probably be implemented in the job submit Lua script if your jobs are 
being rejected at submit time. 
https://slurm.schedmd.com/job_submit_plugins.html that would need to be 
implemented by the cluster admins, and then modified in a way that would 
address your specific scenario.
________________________________
From: slurm-users <slurm-users-boun...@lists.schedmd.com> on behalf of Ebbert, 
Mark T. <mark.ebb...@uky.edu>
Sent: Thursday, August 3, 2023 4:37 AM
To: slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com>
Subject: Re: [slurm-users] Automatically converting jobs to regular priority 
when high-priority is exhausted

WARNING: This message has originated from an External Source. This may be a 
phishing expedition that can result in unauthorized access to our IT System. 
Please use proper judgment and caution when opening attachments, clicking 
links, or responding to this email.

Hello,



Just wanted to give another go at this question.



If no one has input, is there somewhere I can submit feature requests? I didn’t 
see a way to do that on the schedmd website.



____________________________________________



Mark T. W. Ebbert, Ph.D.

Assistant Professor

University of Kentucky

Sanders-Brown Center on Aging

545 Lee T. Todd Jr. Building

Lexington, KY 40536

859.218.0125<tel:859-218-0125>

ebbertlab.com





From: Ebbert, Mark T. <mark.ebb...@uky.edu>
Date: Wednesday, July 26, 2023 at 6:49 PM
To: slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com>
Subject: Automatically converting jobs to regular priority when high-priority 
is exhausted

Hello,



I’m a SLURM user (not administrator) so I may not use the correct terms, but 
hopefully I can get my question across.



Ultimate question: Is there a way to have SLURM automatically convert job 
submissions from a “high-priority” account to regular priority when that 
account’s submissions have been maxed out? If not, is there a way to request 
such a feature?



More background: Essentially, my lab as two accounts that we can use on our 
campus cluster. One is given higher priority in the queue because we directly 
funded a certain number of compute nodes, while the other has regular priority. 
Once we exhaust the number of job units allotted to our high-priority account, 
any new jobs submitted from that account get straight-up rejected from the 
scheduler. We don’t want to have to track how many units we’ve used from a 
certain account and then switch to a different account. We’d like to always be 
able to submit under a single account and let SLURM first use our allotted 
high-priority units, but once those are used up, simply set the rest of the 
jobs as regular priority (and not reject them).



We use Nextflow to manage our job submissions, so in the middle of a large and 
complicated run, all of our jobs start getting rejected. Going through our 
scripts, etc. and changing accounts is a waste of time and error prone. It’s 
gotten to the point where we’re only submitting jobs under our regular account 
so we don’t have to deal with our jobs getting rejected, which defeats the 
purpose of buying into the high-priority to begin with.



Thanks for your help.



____________________________________________



Mark T. W. Ebbert, Ph.D.

Assistant Professor

University of Kentucky

Sanders-Brown Center on Aging

545 Lee T. Todd Jr. Building

Lexington, KY 40536

859.218.0125<tel:859-218-0125>

ebbertlab.com


Reply via email to