[slurm-users] Re: Implementing a "soft" wall clock limit

2025-06-24 Thread Prentice Bisbal via slurm-users
I'm  the one to blame for taking this conversation off target. Sorry about that! Unfortunately, people are hard, and I don't think any amount of technology will ever fix that. 😉 Thanks for explaining your situation, it's certainly different than what most of us see. I would say you need to p

[slurm-users] Re: Implementing a "soft" wall clock limit

2025-06-20 Thread Christopher Samuel via slurm-users
On 6/20/25 2:36 pm, Smith, Sebastian via slurm-users wrote: I forced my users to specify time limits and they quickly adapted: You can also set a default (and maximum) time limit per partition, our default time limits are set to 10 minutes, QOS's have limits in between the partition default

[slurm-users] Re: Implementing a "soft" wall clock limit

2025-06-20 Thread Smith, Sebastian via slurm-users
-users] Re: Implementing a "soft" wall clock limit there is one reason to have a hard time limit. garbage collection. if your institution is like mine, visiting academics often come and go and many full-timers are "forgetful" of what they startup. at some point someone ha

[slurm-users] Re: Implementing a "soft" wall clock limit

2025-06-17 Thread Michael DiDomenico via slurm-users
there is one reason to have a hard time limit. garbage collection. if your institution is like mine, visiting academics often come and go and many full-timers are "forgetful" of what they startup. at some point someone has to clean all that up On Tue, Jun 17, 2025 at 8:18 AM Davide DelVento via

[slurm-users] Re: Implementing a "soft" wall clock limit

2025-06-17 Thread Davide DelVento via slurm-users
This conversation is drifting a bit away from my initial questions and covering various other related topics. In fact I do agree with almost everything written in the last few messages. However, that is somewhat orthogonal to my initial request, which I now understand has the answer "not possible w

[slurm-users] Re: Implementing a "soft" wall clock limit

2025-06-16 Thread Loris Bennett via slurm-users
Hi Prentice, Prentice Bisbal via slurm-users writes: > I think the idea of having a generous default timelimit is the wrong way to > go. In fact, I think any defaults for jobs are a bad way to go. The majority > of your > users will just use that default time limit, and backfill scheduling w

[slurm-users] Re: Implementing a "soft" wall clock limit

2025-06-16 Thread Prentice Bisbal via slurm-users
I think the idea of having a generous default timelimit is the wrong way to go. In fact, I think any defaults for jobs are a bad way to go.  The majority of your users will just use that default time limit, and backfill scheduling will remain useless to you. Instead, I recommend you use your j

[slurm-users] Re: Implementing a "soft" wall clock limit

2025-06-12 Thread Davide DelVento via slurm-users
Sounds good, thanks for confirming it. Let me sleep on it wrt the "too many" QOS, or think if I should ditch this idea. If I'll implement it, I'll post in this conversation details on how I did it. Cheers On Thu, Jun 12, 2025 at 6:59 AM Ansgar Esztermann-Kirchner < aesz...@mpinat.mpg.de> wrote: >

[slurm-users] Re: Implementing a "soft" wall clock limit

2025-06-12 Thread Ansgar Esztermann-Kirchner via slurm-users
On Thu, Jun 12, 2025 at 04:52:24AM -0600, Davide DelVento wrote: > Hi Ansgar, > > This is indeed what I was looking for: I was not aware of PreemptExemptTime. > > From my cursory glance at the documentation, it seems > that PreemptExemptTime is QOS-based and not job based though. Is that > correc

[slurm-users] Re: Implementing a "soft" wall clock limit

2025-06-12 Thread Davide DelVento via slurm-users
Hi Ansgar, This is indeed what I was looking for: I was not aware of PreemptExemptTime. >From my cursory glance at the documentation, it seems that PreemptExemptTime is QOS-based and not job based though. Is that correct? Or could it be set per-job, perhaps on a prolog/submit lua script? I'm thin

[slurm-users] Re: Implementing a "soft" wall clock limit

2025-06-12 Thread Ansgar Esztermann-Kirchner via slurm-users
Hi Davide, I think it should be possible to emulate this via preemption: if you set PreemptMode to CANCEL, a preempted job will behave just as if it reached the end of its wall time. Then, you can use PreemptExemptTime as your soft wall time limit -- the job will not be preempted before PreemptExe

[slurm-users] Re: Implementing a "soft" wall clock limit

2025-06-11 Thread Loris Bennett via slurm-users
Hi Davide, Davide DelVento writes: > Thanks Loris, > > Am I correct if reading in between the lines you're saying: rather than going > on with my "soft" limit idea, just use the regular hard limits, being > generous with > the default and providing user education instead? In fact that is an >

[slurm-users] Re: Implementing a "soft" wall clock limit

2025-06-11 Thread Davide DelVento via slurm-users
Thanks Loris, Am I correct if reading in between the lines you're saying: rather than going on with my "soft" limit idea, just use the regular hard limits, being generous with the default and providing user education instead? In fact that is an alternative approach that I am considering too. On W

[slurm-users] Re: Implementing a "soft" wall clock limit

2025-06-11 Thread Loris Bennett via slurm-users
Hi Davide, Davide DelVento via slurm-users writes: > In the institution where I work, so far we have managed to live > without mandatory wallclock limits (a policy decided well before I > joined the organization), and that has been possible because the > cluster was not very much utilized. > > N