Hi all,
We are using PETSc 3.20 in our code and running succesfully several solvers on
Nvidia GPU with OpenMPI library which are not GPU aware (so I need to add the
flag -use_gpu_aware_mpi 0).
But now, when using OpenMPI GPU Aware library (OpenMPI 4.0.5 ou 4.1.5 from
NVHPC), some parallel calculations failed with KSP_DIVERGED_ITS or
KSP_DIVERGED_DTOL
with several configurations. It may run wells on a small test case with (matrix
is symmetric):
-ksp_type cg -pc_type gamg -pc_gamg_type classical
But suddenly with a number of devices for instance bigger than 4 or 8, it may
fail.
If I switch to another solver (BiCGstab), it may converge:
-ksp_type bcgs -pc_type gamg -pc_gamg_type classical
The more sensitive cases where it diverges are the following:
-ksp_type cg -pc_type hypre -pc_hypre_type boomeramg
-ksp_type cg -pc_type gamg -pc_gamg_type classical
And the bcgs turnaroud doesn't work each time...
It seems to work without problem with aggregation (at least 128 GPUs on my
simulation):
-ksp_type cg -pc_type gamg -pc_gamg_type agg
So I guess there is a weird thing happening in my code during the solve in
PETSc with MPI GPU Aware, as all the previous configurations works with non GPU
aware MPI.
Here is the -ksp_view log during one fail with the first configuration:
KSP Object: () 8 MPI processes
type: cg
maximum iterations=10000, nonzero initial guess
tolerances: relative=0., absolute=0.0001, divergence=10000.
left preconditioning
using UNPRECONDITIONED norm type for convergence test
PC Object: () 8 MPI processes
type: hypre
HYPRE BoomerAMG preconditioning
Cycle type V
Maximum number of levels 25
Maximum number of iterations PER hypre call 1
Convergence tolerance PER hypre call 0.
Threshold for strong coupling 0.7
Interpolation truncation factor 0.
Interpolation: max elements per row 0
Number of levels of aggressive coarsening 0
Number of paths for aggressive coarsening 1
Maximum row sums 0.9
Sweeps down 1
Sweeps up 1
Sweeps on coarse 1
Relax down l1scaled-Jacobi
Relax up l1scaled-Jacobi
Relax on coarse Gaussian-elimination
Relax weight (all) 1.
Outer relax weight (all) 1.
Maximum size of coarsest grid 9
Minimum size of coarsest grid 1
Not using CF-relaxation
Not using more complex smoothers.
Measure type local
Coarsen type PMIS
Interpolation type ext+i
SpGEMM type cusparse
linear system matrix = precond matrix:
Mat Object: () 8 MPI processes
type: mpiaijcusparse
rows=64000, cols=64000
total: nonzeros=311040, allocated nonzeros=311040
total number of mallocs used during MatSetValues calls=0
not using I-node (on process 0) routines
I didn't succeed for the moment creating a reproducer with ex.c examples...
Did you see this kind of behaviour before?
Should I update my PETSc version ?
Thanks for any advice,
Pierre LEDAC
Commissariat à l’énergie atomique et aux énergies alternatives
Centre de SACLAY
DES/ISAS/DM2S/SGLS/LCAN
Bâtiment 451 – point courrier n°43
F-91191 Gif-sur-Yvette
+33 1 69 08 04 03
+33 6 83 42 05 79
