> On 17 Oct 2020, at 5:47 AM, Alexey Kozlov <alexey.v.kozlo...@nd.edu> wrote: > > Thank you for your advice! My sparse matrix seems to be very stiff so I have > decided to concentrate on the direct solvers. I have very good results with > MUMPS. Due to a lack of time I haven’t got a good result with SuperLU_DIST > and haven’t compiled PETSc with Pastix yet but I have a feeling that MUMPS is > the best. I have run a sequential test case with built-in PETSc LU (-pc_type > lu -ksp_type preonly) and MUMPs (-pc_type lu -ksp_type preonly > -pc_factor_mat_solver_type mumps) with default settings and found that MUMPs > was about 50 times faster than the built-in LU and used about 3 times less > RAM. Do you have any idea why it could be? > > My test case has about 100,000 complex equations with about 3,000,000 > non-zeros. PETSc was compiled with the following options: ./configure > --with-blaslapack-dir=/opt/crc/i/intel/19.0/mkl --enable-g > --with-valgrind-dir=/opt/crc/v/valgrind/3.14/ompi --with-scalar-type=complex > --with-clanguage=c --with-openmp --with-debugging=0 COPTFLAGS='-mkl=parallel > -O2 -mavx -axCORE-AVX2 -no-prec-div -fp-model fast=2' > FOPTFLAGS='-mkl=parallel -O2 -mavx -axCORE-AVX2 -no-prec-div -fp-model > fast=2' CXXOPTFLAGS='-mkl=parallel -O2 -mavx -axCORE-AVX2 -no-prec-div > -fp-model fast=2' --download-superlu_dist --download-mumps > --download-scalapack --download-metis --download-cmake --download-parmetis > --download-ptscotch. > > Running MUPMS in parallel using MPI also gave me a significant gain in > performance (about 10 times on a single cluster node). > > Could you, please, advise me whether I can adjust some options for the direct > solvers to improve performance?
Your problem may be too small, but if you stick to full MUMPS, it may be worth playing around with the block low-rank (BLR) options. Here are some references: http://mumps.enseeiht.fr/doc/Thesis_TheoMary.pdf#page=191 <http://mumps.enseeiht.fr/doc/Thesis_TheoMary.pdf#page=191> http://mumps.enseeiht.fr/doc/ud_2017/Shantsev_Talk.pdf <http://mumps.enseeiht.fr/doc/ud_2017/Shantsev_Talk.pdf> The relevant options in PETSc are -mat_mumps_icntl_35, -mat_mumps_icntl_36, and -mat_mumps_cntl_7 Thanks, Pierre > Should I try MUMPS in OpenMP mode? > > On Sat, Sep 19, 2020 at 7:40 AM Mark Adams <mfad...@lbl.gov > <mailto:mfad...@lbl.gov>> wrote: > As Jed said high frequency is hard. AMG, as-is, can be adapted > (https://link.springer.com/article/10.1007/s00466-006-0047-8 > <https://link.springer.com/article/10.1007/s00466-006-0047-8>) with > parameters. > AMG for convection: use richardson/sor and not chebyshev smoothers and in > smoothed aggregation (gamg) don't smooth (-pc_gamg_agg_nsmooths 0). > Mark > > On Sat, Sep 19, 2020 at 2:11 AM Alexey Kozlov <alexey.v.kozlo...@nd.edu > <mailto:alexey.v.kozlo...@nd.edu>> wrote: > Thanks a lot! I'll check them out. > > On Sat, Sep 19, 2020 at 1:41 AM Barry Smith <bsm...@petsc.dev > <mailto:bsm...@petsc.dev>> wrote: > > These are small enough that likely sparse direct solvers are the best use > of your time and for general efficiency. > > PETSc supports 3 parallel direct solvers, SuperLU_DIST, MUMPs and Pastix. I > recommend configuring PETSc for all three of them and then comparing them for > problems of interest to you. > > --download-superlu_dist --download-mumps --download-pastix > --download-scalapack (used by MUMPS) --download-metis --download-parmetis > --download-ptscotch > > Barry > > >> On Sep 18, 2020, at 11:28 PM, Alexey Kozlov <alexey.v.kozlo...@nd.edu >> <mailto:alexey.v.kozlo...@nd.edu>> wrote: >> >> Thanks for the tips! My matrix is complex and unsymmetric. My typical test >> case has of the order of one million equations. I use a 2nd-order >> finite-difference scheme with 19-point stencil, so my typical test case uses >> several GB of RAM. >> >> On Fri, Sep 18, 2020 at 11:52 PM Jed Brown <j...@jedbrown.org >> <mailto:j...@jedbrown.org>> wrote: >> Unfortunately, those are hard problems in which the "good" methods are >> technical and hard to make black-box. There are "sweeping" methods that >> solve on 2D "slabs" with PML boundary conditions, H-matrix based methods, >> and fancy multigrid methods. Attempting to solve with STRUMPACK is probably >> the easiest thing to try (--download-strumpack). >> >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATSOLVERSSTRUMPACK.html >> >> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATSOLVERSSTRUMPACK.html> >> >> Is the matrix complex symmetric? >> >> Note that you can use a direct solver (MUMPS, STRUMPACK, etc.) for a 3D >> problem like this if you have enough memory. I'm assuming the memory or >> time is unacceptable and you want an iterative method with much lower setup >> costs. >> >> Alexey Kozlov <alexey.v.kozlo...@nd.edu <mailto:alexey.v.kozlo...@nd.edu>> >> writes: >> >> > Dear all, >> > >> > I am solving a convected wave equation in a frequency domain. This equation >> > is a 3D Helmholtz equation with added first-order derivatives and mixed >> > derivatives, and with complex coefficients. The discretized PDE results in >> > a sparse linear system (about 10^6 equations) which is solved in PETSc. I >> > am having difficulty with the code convergence at high frequency, skewed >> > grid, and high Mach number. I suspect it may be due to the preconditioner I >> > use. I am currently using the ILU preconditioner with the number of fill >> > levels 2 or 3, and BCGS or GMRES solvers. I suspect the state of the art >> > has evolved and there are better preconditioners for Helmholtz-like >> > problems. Could you, please, advise me on a better preconditioner? >> > >> > Thanks, >> > Alexey >> > >> > -- >> > Alexey V. Kozlov >> > >> > Research Scientist >> > Department of Aerospace and Mechanical Engineering >> > University of Notre Dame >> > >> > 117 Hessert Center >> > Notre Dame, IN 46556-5684 >> > Phone: (574) 631-4335 >> > Fax: (574) 631-8355 >> > Email: akoz...@nd.edu <mailto:akoz...@nd.edu> >> >> >> -- >> Alexey V. Kozlov >> >> Research Scientist >> Department of Aerospace and Mechanical Engineering >> University of Notre Dame >> >> 117 Hessert Center >> Notre Dame, IN 46556-5684 >> Phone: (574) 631-4335 >> Fax: (574) 631-8355 >> Email: akoz...@nd.edu <mailto:akoz...@nd.edu> > > > > -- > Alexey V. Kozlov > > Research Scientist > Department of Aerospace and Mechanical Engineering > University of Notre Dame > > 117 Hessert Center > Notre Dame, IN 46556-5684 > Phone: (574) 631-4335 > Fax: (574) 631-8355 > Email: akoz...@nd.edu <mailto:akoz...@nd.edu> > > > -- > Alexey V. Kozlov > > Research Scientist > Department of Aerospace and Mechanical Engineering > University of Notre Dame > > 117 Hessert Center > Notre Dame, IN 46556-5684 > Phone: (574) 631-4335 > Fax: (574) 631-8355 > Email: akoz...@nd.edu <mailto:akoz...@nd.edu>
