Hey John,
thanks for the workaround.
After some more testing, I've noticed that this does not occur when using Intel
MPI 2018.4, only with the 2019 versions. The slurmd logs show that slurm does
not set any binding in that case.
Best,
Marcus
________________________________
From: slurm-users <slurm-users-boun...@lists.schedmd.com> on behalf of Donners,
John <john.donn...@atos.net>
Sent: Tuesday, February 18, 2020 10:41:29 PM
To: slurm-users@lists.schedmd.com
Subject: Re: [slurm-users] Inconsistent cpu bindings with cpu-bind=none
Hi all,
I have a few more remarks about this question (I have been in contact with
Marcus about this):
- the idea of the jobscript is that SLURM does not do any binding and leaves
binding up to
mpirun.
- this works fine on the first node, where SLURM does not bind the
processes (so mpirun can do this)
- on the second node SLURM uses (faulty) core binding (all processes are
bound round-robin to the hyperthreads
of the first core). Intel’s mpirun respects the cpuset and as a result
the processes are bound incorrectly.
This looks like a SLURM issue to me. SLURM version 19.05.5 is used.
A workaround is to use I_MPI_PIN_RESPECT_CPUSET=no.
Cheers,
John
Hi everyone,
I am facing a bit of a weird issue with CPU bindings and mpirun:
My jobscript:
#SBATCH -N 20
#SBATCH --tasks-per-node=40
#SBATCH -p medium40
#SBATCH -t 30
#SBATCH -o out/%J.out
#SBATCH -e out/%J.err
#SBATCH --reservation=root_98
module load impi/2019.4 2>&1
export I_MPI_DEBUG=6
export SLURM_CPU_BIND=none
.
/sw/comm/impi/compilers_and_libraries_2019.4.243/linux/mpi/intel64/bin/mpivars.sh
realease
BENCH=/sw/comm/impi/compilers_and_libraries_2019.4.243/linux/mpi/intel64/bin/IMB-MPI1
mpirun -np 800 $BENCH -npmin 800 -iter 50 -time 120 -msglog 16:18 -include
Allreduce Bcast Barrier Exchange Gather PingPing PingPong Reduce Scatter
Allgather Alltoall Reduce_scatter
My output is as follows:
[...]
[0] MPI startup(): 37 154426 gcn1311 {37,77}
[0] MPI startup(): 38 154427 gcn1311 {38,78}
[0] MPI startup(): 39 154428 gcn1311 {39,79}
[0] MPI startup(): 40 161061 gcn1312 {0}
[0] MPI startup(): 41 161062 gcn1312 {40}
[0] MPI startup(): 42 161063 gcn1312 {0}
[0] MPI startup(): 43 161064 gcn1312 {40}
[0] MPI startup(): 44 161065 gcn1312 {0}
[...]
On 8 out of 20 nodes I got the wrong pinning. In the slurmd logs I found
that on nodes, where the pinning was correct, manual binding was
communicated correctly:
lllp_distribution jobid [2065227] manual binding: none
On those, where it did not work, not so much:
lllp_distribution jobid [2065227] default auto binding: cores, dist 1
So, for some reason, slurm told some task to use CPU bindings and for
some, the cpu binding was (correctly) disabled.
Any ideas what could cause this?
Best,
Marcus
