On Tue, Jul 21, 2020 at 12:06 PM Pierpaolo Minelli <pierpaolo.mine...@cnr.it> wrote:
> > > Il giorno 21 lug 2020, alle ore 16:56, Mark Adams <mfad...@lbl.gov> ha > scritto: > > > > On Tue, Jul 21, 2020 at 9:46 AM Matthew Knepley <knep...@gmail.com> wrote: > >> On Tue, Jul 21, 2020 at 9:35 AM Pierpaolo Minelli < >> pierpaolo.mine...@cnr.it> wrote: >> >>> Thanks for your reply. >>> As I wrote before, I use these settings: >>> >>> -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg >>> -pc_hypre_boomeramg_relax_type_all SOR/Jacobi >>> -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 >>> -ksp_type richardson >>> >>> Is there a way to emulate this features also with GAMG? >>> >> >> Smoothers: You have complete control here >> >> -mg_levels_pc_type sor (the default is Chebyshev which you could also >> try) >> > > And you set the KSP type. You have -ksp_type richardson above but that is > not used for Hypre. It is for GAMG. Chebyshev is a ksp type (-ksp_type > chebyshev). > > > > Hypre is very good on Poisson. THe grid complexity (cost per iteration) > can be high but the convergence rate will be better than GAMG. > > But, you should be able to get hypre to work. > > > Yes it is very good for Poisson, and on a smaller case, at the beginning > of my code development, I have tried Hypre, ML, and GAMG (without adding > more options I have to admit) and hypre was faster without losing in > precision and accurateness of results (I have checked them with > -ksp_monitor_true_residual). > I left -kps_type Richardson instead of default gmres only because from > residuals it seems more accurate. > Whoops, I made a mistake. I was talking about the smoother. So -mg_levels_pc_type sor -mg_levels_ksp_type chebyshev > > So first, i will try again to see if hypre (with integer 64bit) is able > to work on a smaller case as suggested by Stefano. > Then I will investigate GAMG options and I will give you a feedback. > The problem is that I need 64bit integers because of my problem size so I > have to follow both path, but I hope that I will be able to continue to use > hypre. > > Thanks > > Pierpaolo > > > > >> >> Coarsening: This is much different in agglomeration AMG. There is a >> discussion here: >> >> >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html >> >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetSquareGraph.html >> >> Interpolation: This is built-in for agglomeration AMG. >> >> >>> It would be better to use only native Petsc implementations, but these >>> settings, up to single precision indexing for integers, gave me optimal >>> performances. >>> For this reason I asked also, if it was possible to configure hypre >>> (inside Petsc) with 64bit integers. >>> >> >> Yes. That happened when you reconfigured for 64 bits. You may have >> encountered a Hypre bug. >> >> Thanks, >> >> Matt >> >> >>> Pierpaolo >>> >>> >>> Il giorno 21 lug 2020, alle ore 13:36, Dave May <dave.mayhe...@gmail.com> >>> ha scritto: >>> >>> >>> >>> On Tue, 21 Jul 2020 at 12:32, Pierpaolo Minelli < >>> pierpaolo.mine...@cnr.it> wrote: >>> >>>> Hi, >>>> >>>> I have asked to compile a Petsc Version updated and with 64bit indices. >>>> Now I have Version 3.13.3 and these are the configure options used: >>>> >>>> #!/bin/python >>>> if __name__ == '__main__': >>>> import sys >>>> import os >>>> sys.path.insert(0, os.path.abspath('config')) >>>> import configure >>>> configure_options = [ >>>> '--CC=mpiicc', >>>> '--CXX=mpiicpc', >>>> '--download-hypre', >>>> '--download-metis', >>>> '--download-mumps=yes', >>>> '--download-parmetis', >>>> '--download-scalapack', >>>> '--download-superlu_dist', >>>> '--known-64-bit-blas-indices', >>>> >>>> >>>> '--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', >>>> '--with-64-bit-indices=1', >>>> >>>> >>>> '--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', >>>> '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', >>>> '--with-debugging=0', >>>> '--with-fortran-interfaces=1', >>>> '--with-fortran=1', >>>> 'FC=mpiifort', >>>> 'PETSC_ARCH=arch-linux2-c-opt', >>>> ] >>>> configure.petsc_configure(configure_options) >>>> >>>> Now, I receive an error on hypre: >>>> >>>> forrtl: error (78): process killed (SIGTERM) >>>> Image PC Routine Line >>>> Source >>>> libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl >>>> Unknown Unknown >>>> libpthread-2.17.s 00002B33D5BFD370 Unknown >>>> Unknown Unknown >>>> libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait >>>> Unknown Unknown >>>> libiomp5.so 00002B33DBA14E07 Unknown >>>> Unknown Unknown >>>> libiomp5.so 00002B33DB98810C Unknown >>>> Unknown Unknown >>>> libiomp5.so 00002B33DB990578 Unknown >>>> Unknown Unknown >>>> libiomp5.so 00002B33DB9D9659 Unknown >>>> Unknown Unknown >>>> libiomp5.so 00002B33DB9D8C39 Unknown >>>> Unknown Unknown >>>> libiomp5.so 00002B33DB993BCE __kmpc_fork_call >>>> Unknown Unknown >>>> PIC_3D 00000000004071C0 Unknown >>>> Unknown Unknown >>>> PIC_3D 0000000000490299 Unknown >>>> Unknown Unknown >>>> PIC_3D 0000000000492C17 Unknown >>>> Unknown Unknown >>>> PIC_3D 000000000040562E Unknown >>>> Unknown Unknown >>>> libc-2.17.so 00002B33DC5BEB35 __libc_start_main >>>> Unknown Unknown >>>> PIC_3D 0000000000405539 Unknown >>>> Unknown Unknown >>>> >>>> Is it possible that I need to ask also to compile hypre with an option >>>> for 64bit indices? >>>> Is it possible to instruct this inside Petsc configure? >>>> Alternatively, is it possible to use a different multigrid PC inside >>>> PETSc that accept 64bit indices? >>>> >>> >>> You can use >>> -pc_type gamg >>> All native PETSc implementations support 64bit indices. >>> >>> >>>> >>>> Thanks in advance >>>> >>>> Pierpaolo >>>> >>>> >>>> Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini < >>>> stefano.zamp...@gmail.com> ha scritto: >>>> >>>> You need a version of PETSc compiled with 64bit indices, since the >>>> message indicates the number of dofs in this case is larger the INT_MAX >>>> 2501×3401×1601 = 13617947501 >>>> >>>> I also suggest you upgrade to a newer version, 3.8.3 is quite old as >>>> the error message reports >>>> >>>> Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli < >>>> pierpaolo.mine...@cnr.it> ha scritto: >>>> >>>>> Hi, >>>>> >>>>> I am trying to solve a Poisson equation on this grid: >>>>> >>>>> Nx = 2501 >>>>> Ny = 3401 >>>>> Nz = 1601 >>>>> >>>>> I received this error: >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Overflow in integer operation: >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices >>>>> [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 >>>>> bit indices >>>>> [0]PETSC ERROR: See >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>> shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 >>>>> [0]PETSC ERROR: >>>>> /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a >>>>> arch-linux2-c-opt named r129c09s02 by pminelli Tu >>>>> e May 26 20:16:34 2020 >>>>> [0]PETSC ERROR: Configure options >>>>> --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary >>>>> CC=mpiicc FC=mpiifort CXX=mpiicpc >>>>> F77=mpiifort F90=mpiifort --with-debugging=0 >>>>> --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl >>>>> --with-fortran=1 >>>>> --with-fortran-interfaces=1 >>>>> --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none >>>>> --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- >>>>> 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes >>>>> --download-hypre --download-superlu_dist --download-parmetis --downlo >>>>> ad-metis >>>>> [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in >>>>> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ >>>>> impls/da/da3.c >>>>> [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in >>>>> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl >>>>> s/da/dareg.c >>>>> [0]PETSC ERROR: #3 DMSetUp() line 720 in >>>>> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf >>>>> ace/dm.c >>>>> forrtl: error (76): Abort trap signal >>>>> >>>>> >>>>> I am on an HPC facility and after I loaded PETSC module, I have seen >>>>> that it is configured with INTEGER size = 32 >>>>> >>>>> I solve my problem with these options and it works perfectly with >>>>> smaller grids: >>>>> >>>>> -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg >>>>> -pc_hypre_boomeramg_relax_type_all SOR/Jacobi >>>>> -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 >>>>> -ksp_type richardson >>>>> >>>>> Is it possible to overcome this if I ask them to install a version >>>>> with INTEGER SIZE = 64? >>>>> Alternatively, is it possible to overcome this using intel compiler >>>>> options? >>>>> >>>>> Thanks in advance >>>>> >>>>> Pierpaolo Minelli >>>> >>>> >>>> >>>> -- >>>> Stefano >>>> >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> > >
