Re: [slurm-users] How to tell SLURM to ignore specific GPUs

[Stephan Roth]

On 02.02.22 18:32, Michael Di Domenico wrote:

On Mon, Jan 31, 2022 at 3:57 PM Stephan Roth <stephan.r...@ee.ethz.ch> wrote:

The problem is to identify the cards physically from the information we
have, like what's reported with nvidia-smi or available in
/proc/driver/nvidia/gpus/*/information
The serial number isn't shown for every type of GPU and I'm not sure the
ones shown match the stickers on the GPUs.
If anybody were to know of a practical solution for this, I'd be happy
to read it.

i hadn't seen this proc driver reference before.  checking a few of my
A100's and V100's and some off hand Quadro cards, i don't see the
serial number for any of them in the /proc.  sadly this would be
pretty handy, does anyone know which cards do support this?  i wonder
if there's some obscure something or other that needs to be turned on
to dump out the serial number in /proc instead of running nvidia-smi

Sorry, I didn't state cleary what I was referring to.


nvidia-smi -q |grep -E '^\s+Serial Number\s+:'
    Serial Number                   : N/A

Stephan

I never saw the serial number in /proc/driver/nvidia/gpus/*/information, but by using nvidia-smi. The information was also sometimes empty:

smime.p7s
Description: S/MIME Cryptographic Signature