Re: [slurm-users] How to look for free nodes of a certain constraint efficiently

[Carsten Beyer]

Hi Matt,




[root@ma1 slurm]# sinfo -t idle -o "%P %.5a %.10l %.6D %.6t %N %b"
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST ACTIVE_FEATURES

prepost    up   12:00:00      0    n/a  (null)
shared    up 7-00:00:00      0    n/a  (null)
gpu    up   12:00:00      5   idle mg[106-109,111] k80,256G
gpu    up   12:00:00      2   idle mg[201,203] m40,512G
gpu    up   12:00:00      1   idle mg207 v100,512G
miklip    up 2-00:00:00      2   idle m[21431,21588] 256G

compute2    up    8:00:00     20   idle m[21167-21168,21368-21385] 128G

Cheers,
Carsten

--
Carsten Beyer
Abteilung Systeme

Deutsches Klimarechenzentrum GmbH (DKRZ)
Bundesstraße 45a * D-20146 Hamburg * Germany

Phone:  +49 40 460094-221
Fax:    +49 40 460094-270
Email:be...@dkrz.de
URL:http://www.dkrz.de

Geschäftsführer: Prof. Dr. Thomas Ludwig
Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784

you may have a look to sinfo/squeue command with the --format / -o output options, e.g.: compute    up    8:00:00     44   idle m[10474-10475,10594-10596,10598,10600-10608,10924-10925,10927,10934-10935,10990-11013] 64G compute2    up    8:00:00     30   idle m[20058,20110-20112,20233,20373-20375,20513,20521,20706,20709,20828-20831,20936-20941,20943-20944,21047,21053-21055,21733-21734] 64G Am 14.10.2021 um 14:44 schrieb Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]:

All,





I asked a SLURM guru here and they suggested using --nodelist/-w a la:

  squeue -a -w nodea[001-100],nodeb[001-100],... -t r





Thanks,

Matt



--

Matt Thompson, SSAI, Ld Scientific Programmer/Analyst

NASA GSFC,    Global Modeling and Assimilation Office

Code 610.1,  8800 Greenbelt Rd,  Greenbelt,  MD 20771

Phone: 301-614-6712                 Fax: 301-614-6246

_http://science.gsfc.nasa.gov/sed/bio/matthew.thompson_

I work on a cluster that uses SLURM which has various types of nodes that are are controlled via --constraint flags in sbatch. Now, I started thinking "How can I figure out how many jobs are running/pending/etc on a certain type of node?". I first thought obviously "squeue --constraint=foo", but...nope. No --constraint flag with squeue. Okay. Constraints are just Features by another name, but...you can't seem to just squeue a feature either. where you pass in all the nodes of a certain type. And, yep, that works! But that also means I have to know what nodes are what type. I could obviously do a one-time parsing of "scontrol show nodes" and see what each chunk is and be done with it...but dangit I'm lazy and SLURM has so many programs and options there might just be something and I haven't read the right manpage! :) So I was wondering if anyone out there knows of a cool/elegant/efficient way of doing this? PS: I still might write a bash script where I've listed what the node names are of constraint and realize I might have to update it once every year or two. Now time to look at what parser SLURM uses for nodelist. Can you use regexes and use *, etc? Or just use nodea[001-100]? Time to find out!

smime.p7s
Description: S/MIME Cryptographic Signature