I'd probably write a shell function that would calculate the time required, and
add it as a command-line parameter to sbatch. We do a similar thing for easier
interactive shells in our /etc/profile.d folder on the login node:
function hpcshell() {
srun --partition=interactive $@ --pty bash -i
}
So something like (untested):
weekendjob () {
WEEKRESEND=$(scontrol show res | head -n1 | awk '{print $3}' | cut -f2 -d= |
xargs -I {} date +%s --date "{}" )
NOW=$(date +%s)
HOURS=$(((WEEKRESEND - NOW) / 3600))
sbatch --time=${HOURS}:00:00 $@
}
and:
weekendjob myjob.sh
could work, with the possible downside that anyone using this early in the
reservation period would potentially reserve resources for the entire weekend.
From: slurm-users <slurm-users-boun...@lists.schedmd.com> on behalf of Jeremy
Fix <jeremy....@centralesupelec.fr>
Date: Tuesday, March 30, 2021 at 2:24 AM
To: Florian Zillner <fzill...@lenovo.com>, slurm-users@lists.schedmd.com
<slurm-users@lists.schedmd.com>
Subject: Re: [slurm-users] [External] Autoset job TimeLimit to fit in a
reservation
External Email Warning
This email originated from outside the university. Please use caution when
opening attachments, clicking links, or responding to requests.
________________________________
Hi Florian, and first of all , thanks for this single line command which is
already quite a lot; To give it a bit more sense, I will compute the remaining
time until the reservation ends, so just changing $2 in $3 in your command
# WEEKRESEND=$(scontrol show res | head -n1 | awk '{print $3}' | cut -f2 -d= |
xargs -I {} date +%s --date "{}" )
# NOW=$(date +%s)
# echo "$(((WEEKRESEND - NOW) / 3600)) hours left until reservation ends"
178 hours left until reservation ends
Trying to give some sense, here is the use case. We use slurm for allocating
resources for teaching purposes. We want to allow the students to allocate as
they wish during the week ends, starting friday evening up to monday early
morning. For that we think about creating a reservation that is regularly
scheduled every week and assigned to a partition with part of the nodes we want
the students to be able to work with. Then comes the reason why we need to
set the timelimit. The partition have their timelimit and we set it to, say,
2 days and a half for a job submitted on friday evening to be allowed to run
until monday morning. Now, suppose a user starts a job on sunday evening
without specifying the timelimit of its jobs, the default will be the
partition's one (2 days and a half) and if there is a reservation scheduled on
monday morning, for a practical say, if I'm not wrong, slurm will not allow the
allocation because it will consider that sunday evening + the default partition
timelimit will overlap with a planned reservation, which makes sense. The
first answer could be to ask the user to specify the timelimit on its job so
that it's allocation does not overlap with the next reservation but we thought
about automatically setting that, hence my question.
So, in our use case, it's not a problem to kill a job by the end of the
reservation even if the job is not completed because we may need the resources
for a practical on monday morning.
Now, back to your proposal; I was thinking about putting that line in a job
prolog but .... maybe that's was you meant by "putting the cart before the
horse", I may run into troubles : for the prolog to be executed, the job
has to the allocated but It will not with the default timelimit of the
partition because slurm may not allow it (given a practical may be reserved on
monday morning) . Do you think there is any other way than asking the user to
set the timelimit himself in its srun/sbatch command ?
Best;
Jeremy.
On 29/03/2021 22:09, Florian Zillner wrote:
Hi,
well, I think you're putting the cart before the horse, but anyway, you could
write a script that extracts the next reservation and does some simple math to
display the time in hours or else to the user. It's the users job to set the
time their job needs to finish. Auto-squeezing a job that takes 2 days to
complete into a remaining 2 hour window until the reservation starts doesn't
make any sense to me.
# NEXTRES=$(scontrol show res | head -n1 | awk '{print $2}' | cut -f2 -d= |
xargs -I {} date +%s --date "{}" )
# NOW=$(date +%s)
# echo "$(((NEXTRES - NOW) / 3600)) hours left until reservation begins"
178 hours left until reservation begins
Cheers,
Florian
________________________________
From: slurm-users
<slurm-users-boun...@lists.schedmd.com><mailto:slurm-users-boun...@lists.schedmd.com>
on behalf of Jeremy Fix
<jeremy....@centralesupelec.fr><mailto:jeremy....@centralesupelec.fr>
Sent: Monday, 29 March 2021 10:48
To: slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>
<slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com>
Subject: [External] [slurm-users] Autoset job TimeLimit to fit in a reservation
Hi,
I'm wondering if there is any built-in option to autoset a job TimeLimit
to fit within a defined reservation.
For now, it seems to me that the timelimit must be explicitely provided,
in a agreement with the deadline of the reservation, by a user when
invoking the srun or sbatch command while I would find it comfortable to
let slurm calculate the remaining time of the reservation on which
srun/sbatch is executed and fill in the TimeLimit accordingly;
Best;
Jeremy.
