[slurm-users] Re: Building Slurm debian package vs building from source
Hi Arnuld It is most important to keep the Slurm version the same across the board. As you are mentioning the "deb" package I am assuming all of your nodes are of a debian-based distribution that should be close enough for each other. However, Debian based distros are not as "binary compatible" as RHEL based distros (Say, RHEL, Alma, Rocky, CentOS, Oracle, Fedora etc.), thus even though they all use "deb" package, it would be better to avoid sharing deb across different distros. If all of your distros have a similar package version for the dependencies (say, at least on glibc level), except for different way to name a package (e.g. apache2 - httpd), that would potentially allow you to run the same slurm on other distros. In this case, you may work around them by using the DEBIAN/control Depends field to list all of the potential names for each dependency. Static linking packages or using a conda-like environment may help you more if those distros are more different and require a rebuild per distro. Otherwise, it would probably make more sense to just build them on each and every node based on the feature they need (say, ROCm or nvml makes no sense on a node without such devices). More complex structure does indeed require more maintenance work. I got quite tired of it and decided to just ship with RHEL-family OS for all computer nodes and let those who are more familiar with whatever distro to start one up with singularity or docker by themselves. Sincerely, S. Zhang 2024年5月22日(水) 17:11 Arnuld via slurm-users : > We have several nodes, most of which have different Linux distributions > (distro for short). Controller has a different distro as well. The only > common thing between controller and all the does is that all of them ar > x86_64. > > I can install Slurm using package manager on all the machines but this > will not work because controller will have a different version of Slurm > compared to the nodes (21.08 vs 23.11) > > If I build from source then I see two solutions: > - build a deb package > - build a custom package (./configure, make, make install) > > Building a debian package on the controller and then distributing the > binaries on nodes won't work either because that binary will start looking > for the shared libraries that it was built for and those don't exist on the > nodes. > > So the only solution I have is to build a static binary using a custom > package. Am I correct or is there another solution here? > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Building Slurm debian package vs building from source
Hi Arnuld, What I would probably do is to build one for each distro and install them either directly into /usr/local or using deb package. The DEBIAN/control is used by apt to manage a couple of things, such as indexing so apt search shows what this package is for, which package it could replace, which packages are recommended to be installed with it, and which packages need to be installed before this can work. For those machines with a certain brand of GPU, you would need a slurm that is configured and compiled with such option ON, and such device driver in the DEBIAN/control to allow apt to check the driver on the machine meets the requirement of your deb package. You can forget about the second part if you are not using deb packages and just compile - run the slurm on the client machine. The last thing he mentioned is about the slurm versions. A slurm client of lower version (say 23.02) should be able to talk to a slurmctld of higher version (say 23.11) just fine, though the reverse do not apply. For dependency management it is of such complexity that maintaining a distribution of Linux is quite some work - I knew it as I am a maintainer of a Linux distro that uses dpkg packages, but without a debian root and uses a different cli tool etc. In fact I am more worried about how the users would benefit from such a mixture of execution environments - a misstep in configuration or a user submitting job without specifying enough info on what they asks for would probably make the user's job works or does not work purely by chance of which node it got executed, and which environment the job's executables are built against. It probably need a couple of "similar" nodes to allow users benefiting from the job queue to send their job to the place where available. Good luck with your setup Sincerely, S. Zhang On 2024/05/23 13:04, Arnuld via slurm-users wrote: > Not that I recommend it much, but you can build them for each > environment and install the ones needed in each. Oh cool, I will download the latest version 23.11.7 and build debian packages on every machine then > A simple example is when you have nodes with and without GPUs. > You can build slurmd packages without for those nodes and with for the > ones that have them. I do have non-gpu machines. I guess I need to learn to modify the debian Control files for this > Generally, so long as versions are compatible, they can work together. > You will need to be aware of differences for jobs and configs, but it is > possible. you mean the versions of the dependencies are compatible? It is true for most (like munge) but might not be true for others like (yaml or http-parser). I need to check on that. On Thu, May 23, 2024 at 1:07 AM Brian Andrus via slurm-users wrote: Not that I recommend it much, but you can build them for each environment and install the ones needed in each. A simple example is when you have nodes with and without GPUs. You can build slurmd packages without for those nodes and with for the ones that have them. Generally, so long as versions are compatible, they can work together. You will need to be aware of differences for jobs and configs, but it is possible. Brian Andrus On 5/22/2024 12:45 AM, Arnuld via slurm-users wrote: > We have several nodes, most of which have different Linux > distributions (distro for short). Controller has a different distro as > well. The only common thing between controller and all the does is > that all of them ar x86_64. > > I can install Slurm using package manager on all the machines but this > will not work because controller will have a different version of > Slurm compared to the nodes (21.08 vs 23.11) > > If I build from source then I see two solutions: > - build a deb package > - build a custom package (./configure, make, make install) > > Building a debian package on the controller and then distributing the > binaries on nodes won't work either because that binary will start > looking for the shared libraries that it was built for and those don't > exist on the nodes. > > So the only solution I have is to build a static binary using a custom > package. Am I correct or is there another solution here? > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Can Not Use A Single GPU for Multiple Jobs
Arnuld, You may be looking for the srun parameter or configuration option of "--oversubscribe" for CPU as that is the limiting factor now. S. Zhang On 2024/06/21 2:48, Brian Andrus via slurm-users wrote: Well, if I am reading this right, it makes sense. Every job will need at least 1 core just to run and if there are only 4 cores on the machine, one would expect a max of 4 jobs to run. Brian Andrus On 6/20/2024 5:24 AM, Arnuld via slurm-users wrote: I have a machine with a quad-core CPU and an Nvidia GPU with 3500+ cores. I want to run around 10 jobs in parallel on the GPU (mostly are CUDA based jobs). PROBLEM: Each job asks for only 100 shards (runs usually for a minute or so), then I should be able to run 3500/100 = 35 jobs in parallel but slurm runs only 4 jobs in parallel keeping the rest in the queue. I have this in slurm.conf and gres.conf: # GPU GresTypes=gpu,shard # COMPUTE NODES PartitionName=pzero Nodes=ALL Default=YES MaxTime=INFINITE State=UP` PartitionName=pgpu Nodes=hostgpu MaxTime=INFINITE State=UP NodeName=hostgpu NodeAddr=x.x.x.x Gres=gpu:gtx_1080_ti:1,shard:3500 CPUs=4 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=64255 State=UNKNOWN -- Name=gpu Type=gtx_1080_ti File=/dev/nvidia0 Count=1 Name=shard Count=3500 File=/dev/nvidia0 -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Using more cores/CPUs that requested with
If you are letting systemd taking most things over, you got systemd-cgtop that work better than top for your case. There is also systemd-cgls for non-interactive listing. Also mind to check if you are using cgroup2? A mount to check your cgroup would suffice. As cgroup is likely not supposed to be used in newer deployments of Slurm. 2025年3月26日(水) 17:14 Gestió Servidors via slurm-users < slurm-users@lists.schedmd.com>: > Hello, > > > > Thanks for your answers. I will try now!! One more question: is there any > way to check if Cgroups restrictions is working fine during a “running” job > or during SLURM scheduling process? > > > > Thanks again! > > > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Using more cores/CPUs that requested with
Ugh I think I did not catch up with the docs. I started with a system that defaults to cgroup v1 but the Slurm doc for that plugin is NOT available at that time. Thus I converted everything to cgroup v2. It appears that they are both supported and that documentation issue is more on the dev side than admin side. Thanks for pointing that out. I misinterpreted the "coming soon" part of cgroup v1 plugin and the "legacy" naming for "do not use". It should be fine. 2025年3月27日(木) 0:48 Williams, Jenny Avis : > “ … As cgroup is likely not supposed to be used in newer deployments of > Slurm.” > > > > I am curious about this statement. Would someone expand on this, to either > support or counter it? > > > > Jenny Williams > > UNC Chapel Hill > > > > > > *From:* Shunran Zhang via slurm-users > *Sent:* Wednesday, March 26, 2025 10:52 AM > *To:* Gestió Servidors > *Cc:* Slurm User Community List > *Subject:* [slurm-users] Re: Using more cores/CPUs that requested with > > > > If you are letting systemd taking most things over, you got systemd-cgtop > that work better than top for your case. There is also systemd-cgls for > non-interactive listing. > > > > Also mind to check if you are using cgroup2? A mount to check your cgroup > would suffice. As cgroup is likely not supposed to be used in newer > deployments of Slurm. > > > > > > 2025年3月26日(水) 17:14 Gestió Servidors via slurm-users < > slurm-users@lists.schedmd.com>: > > Hello, > > > > Thanks for your answers. I will try now!! One more question: is there any > way to check if Cgroups restrictions is working fine during a “running” job > or during SLURM scheduling process? > > > > Thanks again! > > > > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com > > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Run only one time on a node
Assuming all node need to run the same task once... How about -n num_of_nodes --ntasks-per-node=1 ? Otherwise if it is more deployment related I would use ansible to do that. S. Zhang On 2025/02/19 2:37, John Hearns via slurm-users wrote: I am running single node tests on a cluster. I can select named nodes using the -2 flag with sbatch. However - if I want to submit perhaps 20 test jobs is there any smart way to run only one time on a node? I know I could touch a file with the hostname and test for that file. I am just wondering if there is a smarter way to do this.\ I should point out that the tests take a few minutes to run, so if a node finishes a test run it could become idle and run the tests again. -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
