> NodeName=dgx03-10] RealMemory=2063937 Boards=1 SocketsPerBoard=2 CoresPerSocket=56 ThreadsPerCore=2 MemSpecLimit=30017 Gres=gpu:H100:8 Feature=location=local
Just in case, that line shows you are missing a bracket in the node name. Are you *actually* missing the bracket? On Tue, Nov 25, 2025 at 9:11 AM Lee via slurm-users < [email protected]> wrote: > Hello, > > Sorry for the delayed response, SC25 interfered with my schedule. > > *Answers* : > 1. Yes, dgx09 and all the others boot the same software images. > > 2. dgx09 and the other nodes mount a shared file system where Slurm is > installed, so /cm/shared/apps/slurm/23.02.6/lib64/slurm/gpu_nvml.so is > the same for every node. I assume the library that is used for > autodetection lives there. I also found a shared library > /usr/lib/x86_64-linux-gnu/libnvml_injection.so.1.0 > (within the software image). I checked the md5sum and it is the same on > both dgx09 and a non-broken node. > > 3. `scontrol show config` is the same on dgx09 and a non-broken DGX. > > 4. The only meaningful difference between `scontrol show node` for dgx09 > and dgx08 (a working node) is : > > < Gres=gpu:*h100*:8(S:0-1) > --- > > Gres=gpu:*H100*:8(S:0-1) > > 5. Yes, we've restarted slurmd and slurmctld several times, the behavior > persists. Of note, when I run `scontrol reconfigure`, the phantom > allocated GPUs (see AllocTRES in original post) are cleared. > > > *Important Update :* > 1. We recently had another GPU tray replaced and now that DGX is > experiencing the same behavior as dgx09. I am more convinced that there is > something subtle with how the hardware is being detected by Slurm. > > Best regards, > Lee > > > On Mon, Nov 17, 2025 at 4:06 PM Timony, Mick < > [email protected]> wrote: > >> Hi Lee, >> >> I manage a BCM cluster as well. Does DGX09 have the same disk image and >> libraries in place? Could the NVidia NVML library, used to auto-detect the >> GPU's, be a diff version and causing the case differences? >> >> If you compare the output of scontrol show node dgx09 and another DGX >> node, do they look the same? Does scontrol show config look different on >> DGX09 and other nodes? >> >> Have you restarted the Slurm controllers (slurmctld) and restarted slurmd >> on the compute nodes? >> >> Kind regards >> >> -- >> Mick Timony >> Senior DevOps Engineer >> LASER, Longwood, & O2 Cluster Admin >> Harvard Medical School >> -- >> ------------------------------ >> *From:* Lee via slurm-users <[email protected]> >> *Sent:* Friday, November 14, 2025 7:17 AM >> *To:* John Hearns <[email protected]> >> *Cc:* [email protected] <[email protected]> >> *Subject:* [slurm-users] Re: Invalid generic resource (gres) >> specification after RMA >> >> Hello, >> >> Thank you for the suggestion. >> >> I ran lspci on dgx09 and a working DGX and the output was identical. >> >> nvidia-smi shows all 8 GPUs and looks the same as the output from a >> working DGX : >> >> root@dgx09:~# nvidia-smi >> Fri Nov 14 07:11:05 2025 >> >> +---------------------------------------------------------------------------------------+ >> | NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA >> Version: 12.2 | >> >> |-----------------------------------------+----------------------+----------------------+ >> | GPU Name Persistence-M | Bus-Id Disp.A | >> Volatile Uncorr. ECC | >> | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | >> GPU-Util Compute M. | >> | | | >> MIG M. | >> >> |=========================================+======================+======================| >> | 0 NVIDIA H100 80GB HBM3 On | 00000000:1B:00.0 Off | >> 0 | >> | N/A 29C P0 69W / 700W | 4MiB / 81559MiB | >> 0% Default | >> | | | >> Disabled | >> >> +-----------------------------------------+----------------------+----------------------+ >> | 1 NVIDIA H100 80GB HBM3 On | 00000000:43:00.0 Off | >> 0 | >> | N/A 30C P0 71W / 700W | 4MiB / 81559MiB | >> 0% Default | >> | | | >> Disabled | >> >> +-----------------------------------------+----------------------+----------------------+ >> | 2 NVIDIA H100 80GB HBM3 On | 00000000:52:00.0 Off | >> 0 | >> | N/A 33C P0 71W / 700W | 4MiB / 81559MiB | >> 0% Default | >> | | | >> Disabled | >> >> +-----------------------------------------+----------------------+----------------------+ >> | 3 NVIDIA H100 80GB HBM3 On | 00000000:61:00.0 Off | >> 0 | >> | N/A 31C P0 73W / 700W | 4MiB / 81559MiB | >> 0% Default | >> | | | >> Disabled | >> >> +-----------------------------------------+----------------------+----------------------+ >> | 4 NVIDIA H100 80GB HBM3 On | 00000000:9D:00.0 Off | >> 0 | >> | N/A 29C P0 68W / 700W | 4MiB / 81559MiB | >> 0% Default | >> | | | >> Disabled | >> >> +-----------------------------------------+----------------------+----------------------+ >> | 5 NVIDIA H100 80GB HBM3 On | 00000000:C3:00.0 Off | >> 0 | >> | N/A 28C P0 69W / 700W | 4MiB / 81559MiB | >> 0% Default | >> | | | >> Disabled | >> >> +-----------------------------------------+----------------------+----------------------+ >> | 6 NVIDIA H100 80GB HBM3 On | 00000000:D1:00.0 Off | >> 0 | >> | N/A 30C P0 70W / 700W | 4MiB / 81559MiB | >> 0% Default | >> | | | >> Disabled | >> >> +-----------------------------------------+----------------------+----------------------+ >> | 7 NVIDIA H100 80GB HBM3 On | 00000000:DF:00.0 Off | >> 0 | >> | N/A 32C P0 69W / 700W | 4MiB / 81559MiB | >> 0% Default | >> | | | >> Disabled | >> >> +-----------------------------------------+----------------------+----------------------+ >> >> >> +---------------------------------------------------------------------------------------+ >> | Processes: >> | >> | GPU GI CI PID Type Process name >> GPU Memory | >> | ID ID >> Usage | >> >> |=======================================================================================| >> | No running processes found >> | >> >> +---------------------------------------------------------------------------------------+ >> >> >> Best regards, >> Lee >> >> On Fri, Nov 14, 2025 at 3:53 AM John Hearns <[email protected]> wrote: >> >> I work for AMD... >> diagnostics I woud run are lspci nvidia-smi >> >> On Thu, 13 Nov 2025 at 19:18, Lee via slurm-users < >> [email protected]> wrote: >> >> Good afternoon, >> >> I have a cluster that is managed by Base Command Manager (v10) and it has >> several Nvidia DGXs. dgx09 is a problem child. The entire node was RMA'd >> and now it no longer behaves the same as my other DGXs. I think the below >> symptoms are caused by a single underlying issue. >> >> *Symptoms : * >> 1. When I look at our 8 non-MIG DGXs, via `scontrol show node=dgxXY | >> grep Gres`, 7/8 DGXs report "Gres=gpu:*H100*:8(S:0-1)" while dgx09 >> reports "Gres=gpu:*h100*:8(S:0-1)" >> >> 2. When I submit a job to this node, I get : >> >> $ srun --reservation=g09_test --gres=gpu:2 --pty bash >> srun: error: Unable to create step for job 105035: Invalid generic >> resource (gres) specification >> >> ### No job is running on the node, yet AllocTRES shows consumed >> resources... >> $ scontrol show node=dgx09 | grep -i AllocTRES >> *AllocTRES=gres/gpu=2* >> >> ### dgx09 : /var/log/slurmd contains no information >> ### slurmctld shows : >> root@h01:# grep 105035 /var/log/slurmctld >> [2025-11-13T07:44:56.380] sched: _slurm_rpc_allocate_resources >> JobId=105035 NodeList=dgx09 usec=3420 >> [2025-11-13T07:44:56.482] _job_complete: JobId=105035 WTERMSIG 1 >> [2025-11-13T07:44:56.483] _job_complete: JobId=105035 done >> >> >> *Configuration : * >> 1. gres.conf : >> # This section of this file was automatically generated by cmd. Do not >> edit manually! >> # BEGIN AUTOGENERATED SECTION -- DO NOT REMOVE >> AutoDetect=NVML >> NodeName=dgx[01,02] Name=gpu Type=1g.20gb Count=32 AutoDetect=NVML >> NodeName=dgx[03-10] Name=gpu Type=h100 Count=8 AutoDetect=NVML >> # END AUTOGENERATED SECTION -- DO NOT REMOVE >> >> 2. grep NodeName slurm.conf >> root@h01:# grep NodeName slurm.conf >> NodeName=dgx[01,02] RealMemory=2063937 Boards=1 SocketsPerBoard=2 >> CoresPerSocket=56 ThreadsPerCore=2 MemSpecLimit=30017 Gres=gpu:1g.20gb:32 >> Feature=location=local >> NodeName=dgx03-10] RealMemory=2063937 Boards=1 SocketsPerBoard=2 >> CoresPerSocket=56 ThreadsPerCore=2 MemSpecLimit=30017 Gres=gpu:H100:8 >> Feature=location=local >> >> 3. What slurmd detects on dgx09 >> >> root@dgx09:~# slurmd -C >> NodeName=dgx09 CPUs=224 Boards=1 SocketsPerBoard=2 CoresPerSocket=56 >> ThreadsPerCore=2 RealMemory=2063937 >> UpTime=8-00:39:10 >> >> root@dgx09:~# slurmd -G >> slurmd: gpu/nvml: _get_system_gpu_list_nvml: 8 GPU system device(s) >> detected >> slurmd: Gres Name=gpu Type=h100 Count=1 Index=0 ID=7696487 >> File=/dev/nvidia0 Cores=0-55 CoreCnt=224 Links=-1,0,0,0,0,0,0,0 >> Flags=HAS_FILE,HAS_TYPE,ENV_NVML >> slurmd: Gres Name=gpu Type=h100 Count=1 Index=1 ID=7696487 >> File=/dev/nvidia1 Cores=0-55 CoreCnt=224 Links=0,-1,0,0,0,0,0,0 >> Flags=HAS_FILE,HAS_TYPE,ENV_NVML >> slurmd: Gres Name=gpu Type=h100 Count=1 Index=2 ID=7696487 >> File=/dev/nvidia2 Cores=0-55 CoreCnt=224 Links=0,0,-1,0,0,0,0,0 >> Flags=HAS_FILE,HAS_TYPE,ENV_NVML >> slurmd: Gres Name=gpu Type=h100 Count=1 Index=3 ID=7696487 >> File=/dev/nvidia3 Cores=0-55 CoreCnt=224 Links=0,0,0,-1,0,0,0,0 >> Flags=HAS_FILE,HAS_TYPE,ENV_NVML >> slurmd: Gres Name=gpu Type=h100 Count=1 Index=4 ID=7696487 >> File=/dev/nvidia4 Cores=56-111 CoreCnt=224 Links=0,0,0,0,-1,0,0,0 >> Flags=HAS_FILE,HAS_TYPE,ENV_NVML >> slurmd: Gres Name=gpu Type=h100 Count=1 Index=5 ID=7696487 >> File=/dev/nvidia5 Cores=56-111 CoreCnt=224 Links=0,0,0,0,0,-1,0,0 >> Flags=HAS_FILE,HAS_TYPE,ENV_NVML >> slurmd: Gres Name=gpu Type=h100 Count=1 Index=6 ID=7696487 >> File=/dev/nvidia6 Cores=56-111 CoreCnt=224 Links=0,0,0,0,0,0,-1,0 >> Flags=HAS_FILE,HAS_TYPE,ENV_NVML >> slurmd: Gres Name=gpu Type=h100 Count=1 Index=7 ID=7696487 >> File=/dev/nvidia7 Cores=56-111 CoreCnt=224 Links=0,0,0,0,0,0,0,-1 >> Flags=HAS_FILE,HAS_TYPE,ENV_NVML >> >> >> *Questions : * >> 1. As far as I can tell, dgx09 is identical to all my non-MIG DGX nodes >> in terms of configuration and hardware. Why does scontrol report it having >> 'h100' with a lower case 'h' unlike the other dgxs which report with an >> upper case 'H'? >> >> 2. Why is dgx09 not accepting GPU jobs and afterwards it artificially >> thinks that there are GPUs allocated even though no jobs are on the node? >> >> 3. Are there additional tests / configurations that I can do to probe the >> differences between dgx09 and all my other nodes? >> >> Best regards, >> Lee >> >> -- >> slurm-users mailing list -- [email protected] >> To unsubscribe send an email to [email protected] >> >> > -- > slurm-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] >
-- slurm-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
