QOS Group TRES limits apply to associations. If I recall correctly, an association is a (user,account,partition,cluster)
On Fri, Oct 21, 2022 at 9:46 AM Matthew R. Baney <mba...@umd.edu> wrote: > > Hello, > > I have noticed that jobs submitted to non-preemptable partitions (PreemptType > = preempt/partition_prio and PreemptMode = REQUEUE) under accounts with > GrpTRES limits will become pending with AssocGrpGRES as the reason when the > account is up against the relevant limit, even when there are other running > jobs on preemptable partitions under the same account and when the pending > jobs have higher priority. The GRES in consideration are GPUs. > > It seems like the scheduler is checking to see if the pending jobs are within > the GRES limit for the account before considering if any of the other jobs in > the account are running on preemptable partitions. In some specific observed > cases, even preempting a single job running in a preemptable partition would > allow the non-preemptable partition job to run (based on GRES freed up from > preemption). > > Is it possible to reverse the order in which these checks are evaluated? > > Best, > Matthew > > -- > Matthew Baney > UMIACS Technical Staff > mba...@umd.edu | (301) 405-6756 > University of Maryland Institute for Advanced Computer Studies > 3154 Brendan Iribe Center > 8125 Paint Branch Dr. > College Park, MD 20742
