[slurm-users] sbatch and --nodes

[Michael DiDomenico via slurm-users]

its friday and i'm either doing something silly or have a misconfig
somewhere, i can't figure out which

when i run

sbatch --nodes=1 --cpus-per-task=1 --array=1-100 --output
test_%A_%a.txt --wrap 'uname -n'

sbatch doesn't seem to be adhering to the --nodes param.  when i look
at my output files it's spreading them across more nodes.  in the
simple case above it's 50/50, but if i through a random sleep in,
it'll be more.  and if i expand the array it'll use even more nodes.
i'm using con/tres and have cr_core_memory,cr_one_core_per_task set

-- 
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com

[slurm-users] Re: Job not starting

[Michael DiDomenico via slurm-users]

you don't need to be a subscriber to search bugs.schedmd.com

On Tue, Dec 10, 2024 at 9:44 AM Davide DelVento via slurm-users
 wrote:
>
> Good sleuthing.
>
> It would be nice if Slurm would say something like 
> Reason=Priority_Lower_Than_Job_ so people will immediately find the 
> culprit in such situations. Has anybody with a SchedMD subscription ever 
> asked something like that, or is there some reasons for which it'd be 
> impossible (or too hard) information to gather programmatically?
>
> On Tue, Dec 10, 2024 at 1:09 AM Diego Zuccato via slurm-users 
>  wrote:
>>
>> Found the problem: another job was blocking access to the reservation.
>> The strangest thing is that the node (gpu03) has always been reserved
>> for a project, the blocking job did not explicitly request it (and even
>> if it did, it would have been denied access) but its state was:
>> JobState=PENDING Reason=ReqNodeNotAvail,_UnavailableNodes:gpu03
>> Dependency=(null)
>>
>> Paint me surprised...
>>
>> Diego
>>
>> Il 07/12/2024 10:03, Diego Zuccato via slurm-users ha scritto:
>> > Ciao Davide.
>> >
>> > Il 06/12/2024 16:42, Davide DelVento ha scritto:
>> >
>> >> I find it extremely hard to understand situations like this. I wish
>> >> Slurm were more clear on how it reported what it is doing, but I
>> >> digress...
>> > I agree. A "scontrol explain" command could be really useful to pinpont
>> > the cause :)
>> >
>> >> I suspect that there are other job(s) which have higher priority than
>> >> this one which are supposed to run on that node but cannot start
>> >> because maybe this/these high-priority jobs(s) need(s) several nodes
>> >> and the other nodes are not available at the moment?
>> > That partition is a single node, and it's IDLE. If another job needed
>> > it, it would be in PLANNED state (IIRC).
>> >
>> >> Pure speculation, obviously, since I have no idea what the rest of
>> >> your cluster looks like, and what the rest of the workflow is, but the
>> >> clue/ hint is
>> >>
>> >>  > JobState=PENDING Reason=Priority Dependency=(null)
>> >>
>> >> You are pending because something else has higher priority. Going back
>> >> to my first sentence, I wish Slurm would say which one other job
>> >> (maybe there are more than one, but one would suffice for this
>> >> investigation) is trumping this job priority so one could more
>> >> clearly understand what is going on, without sleuthing.
>> > Couldn't agree more :) Scheduler is quite opaque in its decisions. :(
>> >
>> > Actually the job that the user submitted is not starting and has
>> > Reason=PartitionConfig . But QoS 'debug' (the one I'm using for testing)
>> > does have higher priority (1000) than QoS 'long' (10, IIRC).
>> >
>> > Diego
>> >
>> >> On Fri, Dec 6, 2024 at 7:36 AM Diego Zuccato via slurm-users > >> us...@lists.schedmd.com > wrote:
>> >>
>> >> Hello all.
>> >> An user reported that a job wasn't starting, so I tried to replicate
>> >> the
>> >> request and I get:
>> >> -8<--
>> >> [root@ophfe1 root.old]# scontrol show job 113936
>> >> JobId=113936 JobName=test.sh
>> >>  UserId=root(0) GroupId=root(0) MCS_label=N/A
>> >>  Priority=1 Nice=0 Account=root QOS=long
>> >>  JobState=PENDING Reason=Priority Dependency=(null)
>> >>  Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
>> >>  RunTime=00:00:00 TimeLimit=2-00:00:00 TimeMin=N/A
>> >>  SubmitTime=2024-12-06T13:19:36 EligibleTime=2024-12-06T13:19:36
>> >>  AccrueTime=2024-12-06T13:19:36
>> >>  StartTime=Unknown EndTime=Unknown Deadline=N/A
>> >>  SuspendTime=None SecsPreSuspend=0
>> >> LastSchedEval=2024-12-06T13:21:32
>> >> Scheduler=Backfill:*
>> >>  Partition=m3 AllocNode:Sid=ophfe1:855189
>> >>  ReqNodeList=(null) ExcNodeList=(null)
>> >>  NodeList=
>> >>  NumNodes=1-1 NumCPUs=96 NumTasks=96 CPUs/Task=1
>> >> ReqB:S:C:T=0:0:*:*
>> >>  TRES=cpu=96,mem=95000M,node=1,billing=1296
>> >>  Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
>> >>  MinCPUsNode=1 MinMemoryNode=95000M MinTmpDiskNode=0
>> >>  Features=(null) DelayBoot=00:00:00
>> >>  OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
>> >>  Command=/home/root.old/test.sh
>> >>  WorkDir=/home/root.old
>> >>  StdErr=/home/root.old/%N-%J.err
>> >>  StdIn=/dev/null
>> >>  StdOut=/home/root.old/%N-%J.out
>> >>  Power=
>> >>
>> >>
>> >> [root@ophfe1 root.old]# scontrol sho partition m3
>> >> PartitionName=m3
>> >>  AllowGroups=ALL DenyAccounts=formazione AllowQos=ALL
>> >>  AllocNodes=ALL Default=NO QoS=N/A
>> >>  DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0
>> >> Hidden=NO
>> >>  MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO
>> >> MaxCPUsPerNode=UNLIMITED
>> >>  Nodes=mtx20
>> >>  Priority

[slurm-users] Re: Job running slower when using Slurm

[Michael DiDomenico via slurm-users]

without knowing anything about your environment, its reasonable to
suspect that maybe your openmp program is multi-threaded, but slurm is
constraining your job to a single core.  evidence of this should show
up when running top on the node, watching the cpu% used for the
program

On Wed, Apr 23, 2025 at 1:28 PM Jeffrey Layton via slurm-users
 wrote:
>
> Good morning,
>
> I'm running an NPB test, bt.C that is OpenMP and built using NV HPC SDK 
> (version 25.1). I run it on a compute node by ssh-ing to the node. It runs in 
> about 19.6 seconds.
>
> Then I run the code using a simple job:
>
> Command to submit job: sbatch --nodes=1 run-npb-omp
>
> The script run-npb-omp is the following:
>
> #!/bin/bash
>
> cd /home/.../NPB3.4-OMP/bin
>
> ./bt.C.x
>
>
> When I use Slurm, the job takes 482 seconds.
>
> Nothing really appears in the logs. It doesn't do any IO. No data is copied 
> anywhere. I'm king of at a loss to figure out why. Any suggestions of where 
> to look?
>
> Thanks!
>
> Jeff
>
>
>
> --
> slurm-users mailing list -- slurm-users@lists.schedmd.com
> To unsubscribe send an email to slurm-users-le...@lists.schedmd.com

-- 
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com

[slurm-users] Re: Job running slower when using Slurm

[Michael DiDomenico via slurm-users]

the program probably says 32 threads, because it's just looking at the
box, not what slurm cgroups allow (assuming your using them) for cpu

i think for an openmp program (not openmpi) you definitely want the
first command with --cpus-per-task=32

are you measuring the runtime inside the program or outside it?  if
the later the 10sec addition in time could be the slurm setup/node
allocation

On Wed, Apr 23, 2025 at 2:41 PM Jeffrey Layton  wrote:
>
> I tried using ntasks and cpus-per-task to get all 32 cores. So I added 
> --ntasks=# --cpus-per-task=N  to th sbatch command  so that it now looks like:
>
> sbatch --nodes=1 --ntasks=1 --cpus-per-task=32