Just to add to my earlier comment below. I think numad is something
that's really meant for non-HPC environments where latency-hiding is
more important than all-out performance. Kinda like hyperthreading - on
HPC workloads, it provides marginal improvement at best, but is very
helpful on non-HPC workloads (or so I've been told - I have no firsthand
professional experience with hyperthreading)
Prentice
On 1/18/22 2:56 PM, Prentice Bisbal wrote:
Mike,
I turn it off. When I had it on, it would cause performance to tank.
Doing some basic analysis, it appeared numad would move all the work
to a single core, leaving all the others idle. Without knowing the
inner workings of numad, my guess is that it saw the processes
accessing the same region of memory, so moved all the processes to the
core "closest" to that memory.
I didn't do any in-depth analysis, but turning off numad definitely
fixed that problem. The problem first appeared with a user code, and I
was able to reproduce it with HPL. It took 10 - 20 minutes for numad
to start migrating processes to the same core, so smaller "test" jobs
didn't trigger the behavior, causing first attempts at reproducing it
were unsuccessful. It wasn't until I ran "full" HPL tests on a node
that I was to reproduce the problem.
I think I used turbostat or something like that to watch the load
and/or processor freqs on the individual cores.
Prentice
On 1/18/22 1:18 PM, Michael Di Domenico wrote:
does anyone turn-on/off numad on their clusters? I'm running RHEL7.9
on Intel CPU's and seeing a heavy performance impact on MPI jobs when
running numad.
diagnosis is pretty prelim right now, so i'm light on details. when
running numad i'm seeing MPI jobs stall while numad pokes at the job.
the stall is notable, like 10-12 seconds
it's particularly interesting because if one rank stalls while numad
runs, the others wait. once it frees they all continue, but then
another rank gets hit, so i end up seeing this cyclic stall
like i said i'm still looking into things, but i curious what
everyone's take on numa is. my consensus is we probably don't even
really need it since slurm/openmpi should be handling process
placement anyhow
thoughts?
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
