Thank you for your advice! My sparse matrix seems to be very stiff so I have decided to concentrate on the direct solvers. I have very good results with MUMPS. Due to a lack of time I haven’t got a good result with SuperLU_DIST and haven’t compiled PETSc with Pastix yet but I have a feeling that MUMPS is the best. I have run a sequential test case with built-in PETSc LU (-pc_type lu -ksp_type preonly) and MUMPs (-pc_type lu -ksp_type preonly -pc_factor_mat_solver_type mumps) with default settings and found that MUMPs was about 50 times faster than the built-in LU and used about 3 times less RAM. Do you have any idea why it could be?
My test case has about 100,000 complex equations with about 3,000,000 non-zeros. PETSc was compiled with the following options: ./configure --with-blaslapack-dir=/opt/crc/i/intel/19.0/mkl --enable-g --with-valgrind-dir=/opt/crc/v/valgrind/3.14/ompi --with-scalar-type=complex --with-clanguage=c --with-openmp --with-debugging=0 COPTFLAGS='-mkl=parallel -O2 -mavx -axCORE-AVX2 -no-prec-div -fp-model fast=2' FOPTFLAGS='-mkl=parallel -O2 -mavx -axCORE-AVX2 -no-prec-div -fp-model fast=2' CXXOPTFLAGS='-mkl=parallel -O2 -mavx -axCORE-AVX2 -no-prec-div -fp-model fast=2' --download-superlu_dist --download-mumps --download-scalapack --download-metis --download-cmake --download-parmetis --download-ptscotch. Running MUPMS in parallel using MPI also gave me a significant gain in performance (about 10 times on a single cluster node). Could you, please, advise me whether I can adjust some options for the direct solvers to improve performance? Should I try MUMPS in OpenMP mode? On Sat, Sep 19, 2020 at 7:40 AM Mark Adams <mfad...@lbl.gov> wrote: > As Jed said high frequency is hard. AMG, as-is, can be adapted ( > https://link.springer.com/article/10.1007/s00466-006-0047-8) with > parameters. > AMG for convection: use richardson/sor and not chebyshev smoothers and in > smoothed aggregation (gamg) don't smooth (-pc_gamg_agg_nsmooths 0). > Mark > > On Sat, Sep 19, 2020 at 2:11 AM Alexey Kozlov <alexey.v.kozlo...@nd.edu> > wrote: > >> Thanks a lot! I'll check them out. >> >> On Sat, Sep 19, 2020 at 1:41 AM Barry Smith <bsm...@petsc.dev> wrote: >> >>> >>> These are small enough that likely sparse direct solvers are the best >>> use of your time and for general efficiency. >>> >>> PETSc supports 3 parallel direct solvers, SuperLU_DIST, MUMPs and >>> Pastix. I recommend configuring PETSc for all three of them and then >>> comparing them for problems of interest to you. >>> >>> --download-superlu_dist --download-mumps --download-pastix >>> --download-scalapack (used by MUMPS) --download-metis --download-parmetis >>> --download-ptscotch >>> >>> Barry >>> >>> >>> On Sep 18, 2020, at 11:28 PM, Alexey Kozlov <alexey.v.kozlo...@nd.edu> >>> wrote: >>> >>> Thanks for the tips! My matrix is complex and unsymmetric. My typical >>> test case has of the order of one million equations. I use a 2nd-order >>> finite-difference scheme with 19-point stencil, so my typical test case >>> uses several GB of RAM. >>> >>> On Fri, Sep 18, 2020 at 11:52 PM Jed Brown <j...@jedbrown.org> wrote: >>> >>>> Unfortunately, those are hard problems in which the "good" methods are >>>> technical and hard to make black-box. There are "sweeping" methods that >>>> solve on 2D "slabs" with PML boundary conditions, H-matrix based methods, >>>> and fancy multigrid methods. Attempting to solve with STRUMPACK is >>>> probably the easiest thing to try (--download-strumpack). >>>> >>>> >>>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATSOLVERSSTRUMPACK.html >>>> >>>> Is the matrix complex symmetric? >>>> >>>> Note that you can use a direct solver (MUMPS, STRUMPACK, etc.) for a 3D >>>> problem like this if you have enough memory. I'm assuming the memory or >>>> time is unacceptable and you want an iterative method with much lower setup >>>> costs. >>>> >>>> Alexey Kozlov <alexey.v.kozlo...@nd.edu> writes: >>>> >>>> > Dear all, >>>> > >>>> > I am solving a convected wave equation in a frequency domain. This >>>> equation >>>> > is a 3D Helmholtz equation with added first-order derivatives and >>>> mixed >>>> > derivatives, and with complex coefficients. The discretized PDE >>>> results in >>>> > a sparse linear system (about 10^6 equations) which is solved in >>>> PETSc. I >>>> > am having difficulty with the code convergence at high frequency, >>>> skewed >>>> > grid, and high Mach number. I suspect it may be due to the >>>> preconditioner I >>>> > use. I am currently using the ILU preconditioner with the number of >>>> fill >>>> > levels 2 or 3, and BCGS or GMRES solvers. I suspect the state of the >>>> art >>>> > has evolved and there are better preconditioners for Helmholtz-like >>>> > problems. Could you, please, advise me on a better preconditioner? >>>> > >>>> > Thanks, >>>> > Alexey >>>> > >>>> > -- >>>> > Alexey V. Kozlov >>>> > >>>> > Research Scientist >>>> > Department of Aerospace and Mechanical Engineering >>>> > University of Notre Dame >>>> > >>>> > 117 Hessert Center >>>> > Notre Dame, IN 46556-5684 >>>> > Phone: (574) 631-4335 >>>> > Fax: (574) 631-8355 >>>> > Email: akoz...@nd.edu >>>> >>> >>> >>> -- >>> Alexey V. Kozlov >>> >>> Research Scientist >>> Department of Aerospace and Mechanical Engineering >>> University of Notre Dame >>> >>> 117 Hessert Center >>> Notre Dame, IN 46556-5684 >>> Phone: (574) 631-4335 >>> Fax: (574) 631-8355 >>> Email: akoz...@nd.edu >>> >>> >>> >> >> -- >> Alexey V. Kozlov >> >> Research Scientist >> Department of Aerospace and Mechanical Engineering >> University of Notre Dame >> >> 117 Hessert Center >> Notre Dame, IN 46556-5684 >> Phone: (574) 631-4335 >> Fax: (574) 631-8355 >> Email: akoz...@nd.edu >> > -- Alexey V. Kozlov Research Scientist Department of Aerospace and Mechanical Engineering University of Notre Dame 117 Hessert Center Notre Dame, IN 46556-5684 Phone: (574) 631-4335 Fax: (574) 631-8355 Email: akoz...@nd.edu
