Thanks Ole for pointing to necessary documents. So Do you mean to say I need to configure the "billing" parameter instead of "GrpTRESMins" .
>> [testuser100@login04 3072]$ scontrol show assoc users=testuser100 accounts=testfac3_imf flags=assoc|grep GrpTRESMins GrpTRESMins=cpu=0(88),mem=N(380538),energy=N(0),node=N(9),billing=N(176),fs/disk=N(0),vmem=N(0),pages=N(0),gres/gpu=N(0) >> Regards Hemanta On Wed, Oct 19, 2022 at 2:01 PM <slurm-users-requ...@lists.schedmd.com> wrote: > Send slurm-users mailing list submissions to > slurm-users@lists.schedmd.com > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users > or, via email, send a message with subject or body 'help' to > slurm-users-requ...@lists.schedmd.com > > You can reach the person managing the list at > slurm-users-ow...@lists.schedmd.com > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of slurm-users digest..." > > > Today's Topics: > > 1. Re: slurm-users Digest, Vol 60, Issue 19 (Hemanta Sahu) > 2. Re: slurm-users Digest, Vol 60, Issue 19 (Ole Holm Nielsen) > 3. GPU utilization of running jobs (Vecerka Daniel) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 18 Oct 2022 19:39:39 +0530 > From: Hemanta Sahu <hemantaku.s...@gmail.com> > To: slurm-users@lists.schedmd.com > Subject: Re: [slurm-users] slurm-users Digest, Vol 60, Issue 19 > Message-ID: > <CAH5Hmwf4jRXYFYbwwBe=+ > artrzwo22ycuy_akgch06w_opo...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi Ole, > > I confirm that the Slurm database has been configured and " > AccountingStorageEnforce" parameters have been set. > > >> > [admin2@login01 ~]$ scontrol show config | grep AccountingStorageEnforce > AccountingStorageEnforce = associations,limits,qos,safe > >> > > My Question : If I have multiple users under a slurm Account and I > want to limit user xxx to max 1000 CPU core-minutes and user yyy to > max 2000 CPU core-minutes for all past,present and future jobs , what > would be the best way to achieve this ? > > > Thanks > > Hemanta > > > On Tue, Oct 18, 2022 at 5:31 PM <slurm-users-requ...@lists.schedmd.com> > wrote: > > > Send slurm-users mailing list submissions to > > slurm-users@lists.schedmd.com > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users > > or, via email, send a message with subject or body 'help' to > > slurm-users-requ...@lists.schedmd.com > > > > You can reach the person managing the list at > > slurm-users-ow...@lists.schedmd.com > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of slurm-users digest..." > > > > > > Today's Topics: > > > > 1. How to implement resource restriction for diffrent slurm > > users under same slurm account (Hemanta Sahu) > > 2. Re: How to implement resource restriction for diffrent slurm > > users under same slurm account (Ole Holm Nielsen) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Mon, 17 Oct 2022 20:21:59 +0530 > > From: Hemanta Sahu <hemantaku.s...@gmail.com> > > To: slurm-users@lists.schedmd.com > > Subject: [slurm-users] How to implement resource restriction for > > diffrent slurm users under same slurm account > > Message-ID: > > <CAH5HmweLsQ7uUkH=D= > > 6xqbzrxlxsgdvpftsbt6t5snqbqqc...@mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > Dear All, > > > > I want to implement resource restriction for different slurm users > > under > > the same slurm account by setting "GrpTRESMins" Flag. For testing > purposes > > I set the "GrpTRESMins=cpu=0" and submitted the Job. > > > > I expect the job submission should fail but it is not happening. The > jobs > > are still going to Q and running. > > > > Kindly help me if I am missing anything obvious. Command outputs given > > below for reference. > > > > > > >> > > [testfac3@login04 export_bin]$ sacctmgr modify user name=testuser100 > > Account=testfac3_imf set GrpTRESMins=cpu=0 > > Modified user associations... > > C = param-shakti A = testfac3_imf U = testuser100 > > Would you like to commit changes? (You have 30 seconds to decide) > > (N/y): y > > > > [testuser100@login04 ~]$ sacctmgr show assoc where Account=testfac3_imf > > user=testuser100 format=Account%15,User%15,GrpTRESMins,QOS%30 > > Account User GrpTRESMins > > QOS > > --------------- --------------- ------------- > > ------------------------------ > > testfac3_imf testuser100 cpu=0 > > testfac3_imf > > > > [testuser100@login04 testuser100]$ sacctmgr show qos sacctmgr show qos > > testfac3_imf > > > > > format=Name%20,MaxWall,Flags%20,GrpTRESMins%20,MaxSubmitJobsPerUser,MaxSubmitJobsPeraccount,GrpTRESRunMin,Priority > > Name MaxWall Flags > GrpTRESMins > > MaxSubmitPU MaxSubmitPA GrpTRESRunMin Priority > > -------------------- ----------- -------------------- > -------------------- > > ----------- ----------- ------------- ---------- > > testfac3_imf 3-00:00:00 DenyOnLimit,NoDecay > cpu=210000000 > > 100 500 10000 > > [testuser100@login04 testuser100]$ > > > > [testuser100@login04 testuser100]$ scontrol show job 949622|grep > JobState > > JobState=COMPLETED Reason=None Dependency=(null) > > [testuser100@login04 testuser100]$ > > > > > > [testuser100@login04 testuser100]$ cat testjob.sh > > #!/bin/bash > > #SBATCH -J testjob # name of the job > > #SBATCH -p standard # name of the partition: available > > options "standard,standard-low,gpu,gpu-low,hm" > > #SBATCH -n 2 # no of processes > > #SBATCH -q testfac3_imf > > #SBATCH -A testfac3_imf > > #SBATCH -t 01:00:00 # walltime in HH:MM:SS, Max value > > 72:00:00 > > #list of modules you want to use, for example > > module load compiler/intel-mpi/mpi-2020-v4 compiler/intel/2020.4.304 > > > > #name of the executable > > exe="uname -n" > > > > #run the application > > mpirun -n $SLURM_NTASKS $exe > > > > [testuser100@login04 testuser100]$ sbatch testjob.sh > > Submitted batch job 949622 > > > > [testuser100@login04 testuser100]$ squeue > > JOBID PARTITION NAME USER ST TIME NODES > > NODELIST(REASON) > > 949622 standard testjob- testuser R 0:04 2 > > cn[304-305] > > >> > > > > Thanks in advance > > > > Best Regards > > Hemanta > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: < > > > http://lists.schedmd.com/pipermail/slurm-users/attachments/20221017/7b137ddb/attachment-0001.htm > > > > > > > ------------------------------ > > > > Message: 2 > > Date: Tue, 18 Oct 2022 07:41:29 +0200 > > From: Ole Holm Nielsen <ole.h.niel...@fysik.dtu.dk> > > To: <slurm-users@lists.schedmd.com> > > Subject: Re: [slurm-users] How to implement resource restriction for > > diffrent slurm users under same slurm account > > Message-ID: <e780481e-9314-9a3e-790b-40c08e3f6...@fysik.dtu.dk> > > Content-Type: text/plain; charset="UTF-8"; format=flowed > > > > On 10/17/22 16:51, Hemanta Sahu wrote: > > > ? ? ?I want to implement resource restriction for different slurm > users > > > under > > > the same slurm account by setting ?"GrpTRESMins" Flag. For testing > > > purposes I set the "GrpTRESMins=cpu=0" ?and submitted the Job. > > > > > > ? I expect the job submission should fail but it is not happening. The > > > jobs are still going to Q and running. > > > > > > Kindly help me if I am missing anything obvious. Command outputs given > > > below for reference. > > > > Job submission should not fail due to resource limits. > > > > Read the slurm.conf manual page to make sure you have set this parameter > > correctly, for example: > > > > $ scontrol show config | grep AccountingStorageEnforce > > AccountingStorageEnforce = associations,limits,qos,safe > > > > You should also read this documentation: > > https://slurm.schedmd.com/resource_limits.html > > > > I assume that you have configured a Slurm database? > > > > /Ole > > > > > > > > End of slurm-users Digest, Vol 60, Issue 19 > > ******************************************* > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://lists.schedmd.com/pipermail/slurm-users/attachments/20221018/662f720c/attachment-0001.htm > > > > ------------------------------ > > Message: 2 > Date: Wed, 19 Oct 2022 08:22:22 +0200 > From: Ole Holm Nielsen <ole.h.niel...@fysik.dtu.dk> > To: <slurm-users@lists.schedmd.com> > Subject: Re: [slurm-users] slurm-users Digest, Vol 60, Issue 19 > Message-ID: <76e0cc7c-8ec0-140f-beef-d82498ad6...@fysik.dtu.dk> > Content-Type: text/plain; charset="UTF-8"; format=flowed > > Hi Hemanta, > > On 10/18/22 16:09, Hemanta Sahu wrote: > > Hi Ole, > > > > I confirm that the Slurm database has been configured and " > > AccountingStorageEnforce"?parameters have been set. > > > > >> > > [admin2@login01 ~]$ ?scontrol show config | grep > AccountingStorageEnforce > > AccountingStorageEnforce = associations,limits,qos,safe > > >> > > > > My Question : If I have multiple users under a slurm Account and I want > to > > limit user xxx to max 1000 CPU core-minutes and user yyy to max 2000 > CPUcore-minutes for all past,present > > and future jobs , what would be the best way to achieve this ? > > Slurm works with a "Associations", see some information in > https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_accounting/#associations > > Under each account in the hierarchy you must define the different users > that you want. They must have different associations (under the same > account), and you can give them different limits. See this nice tutorial: > https://slurm.schedmd.com/slurm_ug_2012/SUG_Oct2012_DBD.pdf > > You can only put limits on future jobs, not on past and running jobs. If > you want to make limits for all of history, you need to setup "banking", > see https://slurm.schedmd.com/SLUG19/Banking.pdf > > /Ole > > > > > > ------------------------------ > >
