Mark,
The problem is a 2D plane elasticity problem, definitely SPD.
If I add -pc_gamg_esteig_ksp_type cg, it works perfectly.
I also tried adding -info but that caused some other type of weird
crash; the options I used were
-ksp_type cg -ksp_monitor -log_view -pc_type gamg -pc_gamg_type agg
-pc_gamg_agg_nsmooths 1 -pc_gamg_esteig_ksp_type cg -info -options_left
The resulting error was:
[0] petscinitialize_internal(): (Fortran):PETSc successfully
started: procs 1
[0] petscinitialize_internal(): Running on machine:
vulture.ce.berkeley.edu
--> ERRORS OCCURRED: For details see file: Ofo
[0] PetscFinalize(): PetscFinalize() called
[0] PetscCommDuplicate(): Duplicating a communicator 140367846615616
28408096 max tags = 2147483647
************************************************************************************************************************
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript
-r -fCourier9' to print this document ***
************************************************************************************************************************
---------------------------------------------- PETSc Performance
Summary: ----------------------------------------------
##########################################################
# #
# WARNING!!! #
# #
# This code was compiled with a debugging option. #
# To get timing results run ./configure #
# using --with-debugging=no, the performance will #
# be generally two or three times faster. #
# #
##########################################################
/home/sg/Feap/ver86/parfeap/feap on a gnu named
vulture.ce.berkeley.edu with 1 processor, by sg Sat Jun 13 12:47:06 2020
Using Petsc Release Version 3.13.2, unknown
Max Max/Min Avg Total
Time (sec): 1.516e-03 1.000 1.516e-03
Objects: 1.000e+00 1.000 1.000e+00
Flop: 0.000e+00 0.000 0.000e+00 0.000e+00
Flop/sec: 0.000e+00 0.000 0.000e+00 0.000e+00
Memory: 5.122e+04 1.000 5.122e+04 5.122e+04
MPI Messages: 0.000e+00 0.000 0.000e+00 0.000e+00
MPI Message Lengths: 0.000e+00 0.000 0.000e+00 0.000e+00
MPI Reductions: 0.000e+00 0.000
Flop counting convention: 1 flop = 1 real number operation of type
(multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of
length N --> 2N flop
and VecAXPY() for complex vectors of
length N --> 8N flop
Summary of Stages: ----- Time ------ ----- Flop ------ ---
Messages --- -- Message Lengths -- -- Reductions --
Avg %Total Avg %Total Count
%Total Avg %Total Count %Total
0: Main Stage: 1.5061e-03 99.3% 0.0000e+00 0.0%
0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0%
------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on
interpreting output.
Phase summary info:
Count: number of times phase was executed
Time and Flop: Max - maximum over all processors
Ratio - ratio of maximum to minimum over all
processors
Mess: number of messages sent
AvgLen: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with
PetscLogStagePush() and PetscLogStagePop().
%T - percent time in this phase %F - percent flop in
this phase
%M - percent messages in this phase %L - percent message
lengths in this phase
%R - percent reductions in this phase
Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max
time over all processors)
------------------------------------------------------------------------------------------------------------------------
##########################################################
# #
# WARNING!!! #
# #
# This code was compiled with a debugging option. #
# To get timing results run ./configure #
# using --with-debugging=no, the performance will #
# be generally two or three times faster. #
# #
##########################################################
Event Count Time (sec)
Flop --- Global --- --- Stage ---- Total
Max Ratio Max Ratio Max Ratio Mess
AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
------------------------------------------------------------------------------------------------------------------------
Memory usage is given in bytes:
Object Type Creations Destructions Memory
Descendants' Mem.
Reports information only for process 0.
--- Event Stage 0: Main Stage
Viewer 1 0 0 0.
========================================================================================================================
Average time to get PetscTime(): 4.38071e-08
#PETSc Option Table entries:
-info
-ksp_monitor
-ksp_type cg
-log_view
-options_left
-pc_gamg_agg_nsmooths 1
-pc_gamg_esteig_ksp_type cg
-pc_gamg_type agg
-pc_type gamg
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8
sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: --download-parmetis --download-superlu_dist
--download-openmpi --download-ml --download-hypre --download-metis
--download-mumps --download-scalapack --download-blacs
-----------------------------------------
Libraries compiled on 2020-06-13 05:13:15 on vulture.ce.berkeley.edu
Machine characteristics:
Linux-5.6.13-100.fc30.x86_64-x86_64-with-fedora-30-Thirty
Using PETSc directory: /home/sg/petsc-3.13.2
Using PETSc arch: gnu
-----------------------------------------
Using C compiler: /home/sg/petsc-3.13.2/gnu/bin/mpicc -fPIC -Wall
-Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
-fstack-protector -fvisibility=hidden -g3
Using Fortran compiler: /home/sg/petsc-3.13.2/gnu/bin/mpif90 -fPIC
-Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g
-----------------------------------------
Using include paths: -I/home/sg/petsc-3.13.2/include
-I/home/sg/petsc-3.13.2/gnu/include
-----------------------------------------
Using C linker: /home/sg/petsc-3.13.2/gnu/bin/mpicc
Using Fortran linker: /home/sg/petsc-3.13.2/gnu/bin/mpif90
Using libraries: -Wl,-rpath,/home/sg/petsc-3.13.2/gnu/lib
-L/home/sg/petsc-3.13.2/gnu/lib -lpetsc
-Wl,-rpath,/home/sg/petsc-3.13.2/gnu/lib
-L/home/sg/petsc-3.13.2/gnu/lib
-Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/9
-L/usr/lib/gcc/x86_64-redhat-linux/9 -lHYPRE -lcmumps -ldmumps
-lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_dist
-lml -llapack -lblas -lX11 -lparmetis -lmetis -lm -lstdc++ -ldl
-lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran
-lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lstdc++ -ldl
-----------------------------------------
##########################################################
# #
# WARNING!!! #
# #
# This code was compiled with a debugging option. #
# To get timing results run ./configure #
# using --with-debugging=no, the performance will #
# be generally two or three times faster. #
# #
##########################################################
[0] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm
28408096
[0] Petsc_OuterComm_Attr_Delete_Fn(): Removing reference to PETSc
communicator embedded in a user MPI_Comm 28408096
[0] Petsc_InnerComm_Attr_Delete_Fn(): User MPI_Comm 140367846615616
is being unlinked from inner PETSc comm 28408096
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 28408096
[0] Petsc_Counter_Attr_Delete_Fn(): Deleting counter data in an
MPI_Comm 28408096
[0] PetscCommDuplicate(): Duplicating a communicator 140367846615616
28408096 max tags = 2147483647
#PETSc Option Table entries:
-info
-ksp_monitor
-ksp_type cg
-log_view
-options_left
-pc_gamg_agg_nsmooths 1
-pc_gamg_esteig_ksp_type cg
-pc_gamg_type agg
-pc_type gamg
#End of PETSc Option Table entries
[0] Petsc_OuterComm_Attr_Delete_Fn(): Removing reference to PETSc
communicator embedded in a user MPI_Comm 28408096
[0] Petsc_InnerComm_Attr_Delete_Fn(): User MPI_Comm 140367846615616
is being unlinked from inner PETSc comm 28408096
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm 28408096
[0] Petsc_Counter_Attr_Delete_Fn(): Deleting counter data in an
MPI_Comm 28408096
WARNING! There are options you set that were not used!
WARNING! could be spelling mistake, etc!
There are 6 unused database options. They are:
Option left: name:-ksp_monitor (no value)
Option left: name:-ksp_type value: cg
Option left: name:-pc_gamg_agg_nsmooths value: 1
Option left: name:-pc_gamg_esteig_ksp_type value: cg
Option left: name:-pc_gamg_type value: agg
Option left: name:-pc_type value: gamg
[sg@vulture main]$
On 6/13/20 5:10 AM, Mark Adams wrote:
That is odd. Are these problems symmetric positive definite?
Eigen estimates are a pain in practice but I've never seen this. Hypre
has (better) smoothers that don't need this and the AMG algorithm does
not need them either. I think ML does pretty much the same thing as me.
If SPD then you definitely want '-pc_gamg_esteig_ksp_type cg'. CG
converges faster and is more robust. Even if not strictly SPD like
with large deformation or plasticity CG is probably better. You can
run with -info and grep on GAMG to see what eigen (high) estimates it
came up with for each level. They should be >1 and < 4, or so.
I wonder if your LAPACK is funky. You might try a fresh build (delete
the "arch" directory) and download LAPACK, but this problem will go
away with CG.
Note, if you use Chebyshev smoothing then it needs eigen estimates
also. THis is usually where eigen estimate problems come up. If you
use jacobi PC in the smoother, GAMG will use the eigen estimate from
the smoothed aggregation smoothing (what is failing for you here).
-pc_gamg_use_sa_esteig <T,F> will force it to use, or not use, it. For
instance, if you sor/ilu/asm then the eigen estimates are probably
fine to reuse but GAMG does not by default. If the cheby eigen
estimates are low then the solver fails. THere is a safety factor to
increase the eigen estimate in chychev to avoid this failure.
And, -pc_gamg_esteig_ksp_max_it X sets the number of iterations in the
estimator. You can increase this and you should see your eigen
estimates go up and converge. You should see that CG converges much
faster than GMRES (the default).
On Sat, Jun 13, 2020 at 2:32 AM Sanjay Govindjee <s...@berkeley.edu
<mailto:s...@berkeley.edu>> wrote:
I have a FEA problem that I am trying to solve with GAMG. The
problem solves
just fine with direct solvers (mumps, superlu) and iterative
solvers (gmres, ml, hypre-boomer) etc.
However with GAMG I am getting a divide by zero that I am having
trouble tracking down. Below
is the gdb stack trace and the source lines going up the stack.
When I run in valgrind the problem runs fine (and gets the correct
answer).
Valgrind reports nothing of note (just lots of indirectly lost
blocks related to PMP_INIT).
I'm only running on one processor.
Any suggestions on where to start to trace the problem?
-sanjay
#0 0x00007fb262dc5be1 in ieeeck_ () from /lib64/liblapack.so.3
#1 0x00007fb262dc5332 in ilaenv_ () from /lib64/liblapack.so.3
#2 0x00007fb262dbbcef in dlasq2_ () from /lib64/liblapack.so.3
#3 0x00007fb262dbb78c in dlasq1_ () from /lib64/liblapack.so.3
#4 0x00007fb262da1e2e in dbdsqr_ () from /lib64/liblapack.so.3
#5 0x00007fb262960110 in dgesvd_ () from /lib64/liblapack.so.3
#6 0x00007fb264e74b66 in
KSPComputeExtremeSingularValues_GMRES (ksp=0x1816560,
emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at
/home/sg/petsc-3.13.2/src/ksp/ksp/impls/gmres/gmreig.c:32
#7 0x00007fb264dfe69a in KSPComputeExtremeSingularValues
(ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at
/home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:64
#8 0x00007fb264b44a1f in PCGAMGOptProlongator_AGG
(pc=0x12f3d30, Amat=0x11a2630, a_P=0x7ffc5010ebe0) at
/home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/agg.c:1145
#9 0x00007fb264b248a1 in PCSetUp_GAMG (pc=0x12f3d30) at
/home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/gamg.c:557
#10 0x00007fb264d8535b in PCSetUp (pc=0x12f3d30) at
/home/sg/petsc-3.13.2/src/ksp/pc/interface/precon.c:898
#11 0x00007fb264e01a93 in KSPSetUp (ksp=0x128dd80) at
/home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:376
#12 0x00007fb264e057af in KSPSolve_Private (ksp=0x128dd80,
b=0x1259f30, x=0x125d910) at
/home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:633
#13 0x00007fb264e086b9 in KSPSolve (ksp=0x128dd80,
b=0x1259f30, x=0x125d910) at
/home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:853
#14 0x00007fb264e46216 in kspsolve_ (ksp=0x832670
<__pfeapc_MOD_kspsol>, b=0x832698 <__pfeapc_MOD_rhs>,
x=0x8326a0 <__pfeapc_MOD_sol>, __ierr=0x7ffc5010f358)
at
/home/sg/petsc-3.13.2/src/ksp/ksp/interface/ftn-auto/itfuncf.c:266
#15 0x000000000043298d in usolve (flags=..., b=...) at
usolve.F:313
#16 0x000000000044afba in psolve (stype=-3, b=..., fp=...,
factor=.TRUE., solve=.TRUE., cfr=.FALSE., prnt=.TRUE.) at
psolve.f:212
#17 0x00000000006b7393 in pmacr1 (lct=..., ct=..., j=3,
_lct=_lct@entry=15) at pmacr1.f:578
#18 0x00000000005c247b in pmacr (initf=.FALSE.) at pmacr.f:578
#19 0x000000000044ff20 in pcontr () at pcontr.f:1307
#20 0x0000000000404d9b in feap () at feap86.f:162
#21 main (argc=<optimized out>, argv=<optimized out>) at
feap86.f:168
#22 0x00007fb261aaef43 in __libc_start_main () from
/lib64/libc.so.6
#23 0x0000000000404dde in _start ()
(gdb) list
1 <built-in>: No such file or directory.
(gdb) up
#1 0x00007fb262dc5332 in ilaenv_ () from /lib64/liblapack.so.3
(gdb) up
#2 0x00007fb262dbbcef in dlasq2_ () from /lib64/liblapack.so.3
(gdb) up
#3 0x00007fb262dbb78c in dlasq1_ () from /lib64/liblapack.so.3
(gdb) up
#4 0x00007fb262da1e2e in dbdsqr_ () from /lib64/liblapack.so.3
(gdb) up
#5 0x00007fb262960110 in dgesvd_ () from /lib64/liblapack.so.3
(gdb) up
#6 0x00007fb264e74b66 in
KSPComputeExtremeSingularValues_GMRES (ksp=0x1816560,
emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at
/home/sg/petsc-3.13.2/src/ksp/ksp/impls/gmres/gmreig.c:32
32
PetscStackCallBLAS("LAPACKgesvd",LAPACKgesvd_("N","N",&bn,&bn,R,&bN,realpart,&sdummy,&idummy,&sdummy,&idummy,work,&lwork,&lierr));
(gdb) up
#7 0x00007fb264dfe69a in KSPComputeExtremeSingularValues
(ksp=0x1816560, emax=0x7ffc5010e7c8, emin=0x7ffc5010e7d0) at
/home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:64
64 ierr =
(*ksp->ops->computeextremesingularvalues)(ksp,emax,emin);CHKERRQ(ierr);
(gdb) up
#8 0x00007fb264b44a1f in PCGAMGOptProlongator_AGG
(pc=0x12f3d30, Amat=0x11a2630, a_P=0x7ffc5010ebe0) at
/home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/agg.c:1145
1145 ierr = KSPComputeExtremeSingularValues(eksp,
&emax, &emin);CHKERRQ(ierr);
(gdb) up
#9 0x00007fb264b248a1 in PCSetUp_GAMG (pc=0x12f3d30) at
/home/sg/petsc-3.13.2/src/ksp/pc/impls/gamg/gamg.c:557
557 ierr = pc_gamg->ops->optprolongator(pc,
Aarr[level], &Prol11);CHKERRQ(ierr);
(gdb) up
#10 0x00007fb264d8535b in PCSetUp (pc=0x12f3d30) at
/home/sg/petsc-3.13.2/src/ksp/pc/interface/precon.c:898
898 ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr);
(gdb) up
#11 0x00007fb264e01a93 in KSPSetUp (ksp=0x128dd80) at
/home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:376
376 ierr = PCSetUp(ksp->pc);CHKERRQ(ierr);
(gdb) up
#12 0x00007fb264e057af in KSPSolve_Private (ksp=0x128dd80,
b=0x1259f30, x=0x125d910) at
/home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:633
633 ierr = KSPSetUp(ksp);CHKERRQ(ierr);
(gdb) up
#13 0x00007fb264e086b9 in KSPSolve (ksp=0x128dd80,
b=0x1259f30, x=0x125d910) at
/home/sg/petsc-3.13.2/src/ksp/ksp/interface/itfunc.c:853
853 ierr = KSPSolve_Private(ksp,b,x);CHKERRQ(ierr);
(gdb) up
#14 0x00007fb264e46216 in kspsolve_ (ksp=0x832670
<__pfeapc_MOD_kspsol>, b=0x832698 <__pfeapc_MOD_rhs>,
x=0x8326a0 <__pfeapc_MOD_sol>, __ierr=0x7ffc5010f358)
at
/home/sg/petsc-3.13.2/src/ksp/ksp/interface/ftn-auto/itfuncf.c:266
266 *__ierr = KSPSolve(
(gdb) up
#15 0x000000000043298d in usolve (flags=..., b=...) at
usolve.F:313
313 call KSPSolve (kspsol, rhs, sol, ierr)
