Thanks Julie! Figured I was missing something. -Randy
On Mon, Sep 17, 2018 at 8:52 PM Julie Bernauer <jberna...@nvidia.com> wrote: > Hi Randy, > > This is expected on an HT machine, like on the one described below. If > you run lstopo, you see: > L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 > PU L#10 (P#5) > PU L#11 (P#45) > Slurm uses the logical cores so 10 and 11 gives you "physical" cores 5 and > 45. > > Julie > > > > ------------------------------ > *From:* slurm-users <slurm-users-boun...@lists.schedmd.com> on behalf of > Randall Radmer <rad...@gmail.com> > *Sent:* Wednesday, September 12, 2018 10:14 AM > *To:* slurm-users@lists.schedmd.com > *Subject:* [slurm-users] Using GRES to manage GPUs, but unable to assign > specific CPUs to specific GPUs > > I’m using GRES to manage eight GPUs in a node on a new Slurm cluster and > am trying to bind specific CPUs to specific GPUs, but it’s not working as I > expected. > > I am able to request a specific number of GPUs, but the CPU assignment > seems wrong. > > I assume I’m missing something obvious, but just can't find it. Any > suggestion for how to fix it, or how to better investigate the problem, > would be much appreciated. > > Example srun requesting one GPU follows: > $ srun -p dgx1 --gres=gpu:1 --pty $SHELL > [node-01:~]$ nvidia-smi --query-gpu=index,name --format=csv > index, name > 0, Tesla V100-SXM2-16GB > [node-01:~]$ cat /sys/fs/cgroup/cpuset/slurm/uid_*/job_*/cpuset.cpus > 5,45 > > Similar example requesting eight GPUs follows: > $ srun -p dgx1 --gres=gpu:8 --pty $SHELL > [node-01:~]$ nvidia-smi --query-gpu=index,name --format=csv > index, name > 0, Tesla V100-SXM2-16GB > 1, Tesla V100-SXM2-16GB > 2, Tesla V100-SXM2-16GB > 3, Tesla V100-SXM2-16GB > 4, Tesla V100-SXM2-16GB > 5, Tesla V100-SXM2-16GB > 6, Tesla V100-SXM2-16GB > 7, Tesla V100-SXM2-16GB > [node-01:~]$ cat /sys/fs/cgroup/cpuset/slurm/uid_*/job_*/cpuset.cpus > 5,45 > > The machines are all Ubuntu 16.04 and Slurm version is 17.11.9-2. > > The /etc/slurm/gres.conf file follows: > [node-01:~]$ less /etc/slurm/gres.conf > Name=gpu Type=V100 File=/dev/nvidia0 Cores=10-11 > Name=gpu Type=V100 File=/dev/nvidia1 Cores=12-13 > Name=gpu Type=V100 File=/dev/nvidia2 Cores=14-15 > Name=gpu Type=V100 File=/dev/nvidia3 Cores=16-17 > Name=gpu Type=V100 File=/dev/nvidia4 Cores=18-19 > Name=gpu Type=V100 File=/dev/nvidia5 Cores=20-21 > Name=gpu Type=V100 File=/dev/nvidia6 Cores=22-23 > Name=gpu Type=V100 File=/dev/nvidia7 Cores=24-25 > > The /etc/slurm/slurm.conf file on all machines in the cluster follows > (with minor cleanup): > ClusterName=testcluster > ControlMachine=slurm-master > SlurmUser=slurm > SlurmctldPort=6817 > SlurmdPort=6818 > AuthType=auth/munge > SlurmdSpoolDir=/var/spool/slurm/d > SwitchType=switch/none > MpiDefault=none > SlurmctldPidFile=/var/run/slurmctld.pid > SlurmdPidFile=/var/run/slurmd.pid > ProctrackType=proctrack/cgroup > PluginDir=/usr/lib/slurm > ReturnToService=2 > Prolog=/etc/slurm/slurm.prolog > PrologSlurmctld=/etc/slurm/slurm.ctld.prolog > Epilog=/etc/slurm/slurm.epilog > EpilogSlurmctld=/etc/slurm/slurm.ctld.epilog > TaskProlog=/etc/slurm/slurm.task.prolog > TaskPlugin=task/affinity,task/cgroup > SlurmctldTimeout=300 > SlurmdTimeout=300 > InactiveLimit=0 > MinJobAge=300 > KillWait=20 > Waittime=0 > SchedulerType=sched/backfill > SelectType=select/cons_res > SelectTypeParameters=CR_Core_Memory > FastSchedule=0 > DebugFlags=CPU_Bind,gres > SlurmctldDebug=debug5 > SlurmctldLogFile=/var/log/slurm/slurmctld.log > SlurmdDebug=3 > SlurmdLogFile=/var/log/slurm/slurmd.log > JobCompType=jobcomp/filetxt > JobCompLoc=/data/slurm/job_completions.log > AccountingStorageType=accounting_storage/slurmdbd > AccountingStorageLoc=/data/slurm/accounting_storage.log > AccountingStorageEnforce=associations,limits,qos > AccountingStorageTRES=gres/gpu,gres/gpu:V100 > PreemptMode=SUSPEND,GANG > PrologFlags=Serial,Alloc > RebootProgram="/sbin/shutdown -r 3" > PreemptType=preempt/partition_prio > CacheGroups=0 > DefMemPerCPU=2048 > GresTypes=gpu > NodeName=node-01 State=UNKNOWN \ > Sockets=2 CoresPerSocket=20 ThreadsPerCore=2 \ > Gres=gpu:V100:8 > PartitionName=all Nodes=node-01 \ > Default=YES MaxTime=4:0:0 DefaultTime=4:0:0 State=UP > > > Thanks, > Randy > > ------------------------------ > This email message is for the sole use of the intended recipient(s) and > may contain confidential information. Any unauthorized review, use, > disclosure or distribution is prohibited. If you are not the intended > recipient, please contact the sender by reply email and destroy all copies > of the original message. > ------------------------------ >
