I assume PIC_3D is your code and you are using OpenMP?
Are you calling hypre from inside your OpenMP parallelism? From inside PIC_3D?
The SIGTERM is confusing to me. Are you using signals in any way? Usually a
sigterm comes outside a process not process or thread crash.
I assume for__signal_handl... is a Fortran signal handler
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown
libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown
libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown
libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown
libiomp5.so 00002B33DB98810C Unknown Unknown Unknown
libiomp5.so 00002B33DB990578 Unknown Unknown Unknown
libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown
libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown
libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown
PIC_3D 00000000004071C0 Unknown Unknown Unknown
PIC_3D 0000000000490299 Unknown Unknown Unknown
PIC_3D 0000000000492C17 Unknown Unknown Unknown
PIC_3D 000000000040562E Unknown Unknown Unknown
libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown
PIC_3D 0000000000405539 Unknown Unknown Unknown
> On Jul 21, 2020, at 6:32 AM, Pierpaolo Minelli <pierpaolo.mine...@cnr.it>
> wrote:
>
> Hi,
>
> I have asked to compile a Petsc Version updated and with 64bit indices.
> Now I have Version 3.13.3 and these are the configure options used:
>
> #!/bin/python
> if __name__ == '__main__':
> import sys
> import os
> sys.path.insert(0, os.path.abspath('config'))
> import configure
> configure_options = [
> '--CC=mpiicc',
> '--CXX=mpiicpc',
> '--download-hypre',
> '--download-metis',
> '--download-mumps=yes',
> '--download-parmetis',
> '--download-scalapack',
> '--download-superlu_dist',
> '--known-64-bit-blas-indices',
>
> '--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary',
> '--with-64-bit-indices=1',
>
> '--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl',
> '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none',
> '--with-debugging=0',
> '--with-fortran-interfaces=1',
> '--with-fortran=1',
> 'FC=mpiifort',
> 'PETSC_ARCH=arch-linux2-c-opt',
> ]
> configure.petsc_configure(configure_options)
>
> Now, I receive an error on hypre:
>
> forrtl: error (78): process killed (SIGTERM)
> Image PC Routine Line Source
>
> libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown
> libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown
> libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown
> libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown
> libiomp5.so 00002B33DB98810C Unknown Unknown Unknown
> libiomp5.so 00002B33DB990578 Unknown Unknown Unknown
> libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown
> libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown
> libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown
> PIC_3D 00000000004071C0 Unknown Unknown Unknown
> PIC_3D 0000000000490299 Unknown Unknown Unknown
> PIC_3D 0000000000492C17 Unknown Unknown Unknown
> PIC_3D 000000000040562E Unknown Unknown Unknown
> libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown
> PIC_3D 0000000000405539 Unknown Unknown Unknown
>
> Is it possible that I need to ask also to compile hypre with an option for
> 64bit indices?
> Is it possible to instruct this inside Petsc configure?
> Alternatively, is it possible to use a different multigrid PC inside PETSc
> that accept 64bit indices?
>
> Thanks in advance
>
> Pierpaolo
>
>
>> Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini
>> <stefano.zamp...@gmail.com <mailto:stefano.zamp...@gmail.com>> ha scritto:
>>
>> You need a version of PETSc compiled with 64bit indices, since the message
>> indicates the number of dofs in this case is larger the INT_MAX
>> 2501×3401×1601 = 13617947501
>>
>> I also suggest you upgrade to a newer version, 3.8.3 is quite old as the
>> error message reports
>>
>> Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli
>> <pierpaolo.mine...@cnr.it <mailto:pierpaolo.mine...@cnr.it>> ha scritto:
>> Hi,
>>
>> I am trying to solve a Poisson equation on this grid:
>>
>> Nx = 2501
>> Ny = 3401
>> Nz = 1601
>>
>> I received this error:
>>
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Overflow in integer operation:
>> http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices
>> <http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices>
>> [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit
>> indices
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017
>> [0]PETSC ERROR:
>> /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a
>> arch-linux2-c-opt named r129c09s02 by pminelli Tu
>> e May 26 20:16:34 2020
>> [0]PETSC ERROR: Configure options
>> --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary
>> CC=mpiicc FC=mpiifort CXX=mpiicpc
>> F77=mpiifort F90=mpiifort --with-debugging=0
>> --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl
>> --with-fortran=1
>> --with-fortran-interfaces=1
>> --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none
>> --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe-
>> 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes
>> --download-hypre --download-superlu_dist --download-parmetis --downlo
>> ad-metis
>> [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in
>> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/
>> impls/da/da3.c
>> [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in
>> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl
>> s/da/dareg.c
>> [0]PETSC ERROR: #3 DMSetUp() line 720 in
>> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf
>> ace/dm.c
>> forrtl: error (76): Abort trap signal
>>
>>
>> I am on an HPC facility and after I loaded PETSC module, I have seen that it
>> is configured with INTEGER size = 32
>>
>> I solve my problem with these options and it works perfectly with smaller
>> grids:
>>
>> -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg
>> -pc_hypre_boomeramg_relax_type_all SOR/Jacobi
>> -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1
>> -ksp_type richardson
>>
>> Is it possible to overcome this if I ask them to install a version with
>> INTEGER SIZE = 64?
>> Alternatively, is it possible to overcome this using intel compiler options?
>>
>> Thanks in advance
>>
>> Pierpaolo Minelli
>>
>>
>> --
>> Stefano
>
