On Fri, Oct 16, 2020 at 11:48 PM Alexey Kozlov <alexey.v.kozlo...@nd.edu>
wrote:
> Thank you for your advice! My sparse matrix seems to be very stiff so I
> have decided to concentrate on the direct solvers. I have very good results
> with MUMPS. Due to a lack of time I haven’t got a good result with
> SuperLU_DIST and haven’t compiled PETSc with Pastix yet but I have a
> feeling that MUMPS is the best. I have run a sequential test case with
> built-in PETSc LU (-pc_type lu -ksp_type preonly) and MUMPs (-pc_type lu
> -ksp_type preonly -pc_factor_mat_solver_type mumps) with default settings
> and found that MUMPs was about 50 times faster than the built-in LU and
> used about 3 times less RAM. Do you have any idea why it could be?
>
The numbers do not sound realistic, but of course we do not have your
particular problem. In particular, the memory figure seems impossible.
> My test case has about 100,000 complex equations with about 3,000,000
> non-zeros. PETSc was compiled with the following options: ./configure
> --with-blaslapack-dir=/opt/crc/i/intel/19.0/mkl --enable-g
> --with-valgrind-dir=/opt/crc/v/valgrind/3.14/ompi
> --with-scalar-type=complex --with-clanguage=c --with-openmp
> --with-debugging=0 COPTFLAGS='-mkl=parallel -O2 -mavx -axCORE-AVX2
> -no-prec-div -fp-model fast=2' FOPTFLAGS='-mkl=parallel -O2 -mavx
> -axCORE-AVX2 -no-prec-div -fp-model fast=2' CXXOPTFLAGS='-mkl=parallel -O2
> -mavx -axCORE-AVX2 -no-prec-div -fp-model fast=2' --download-superlu_dist
> --download-mumps --download-scalapack --download-metis --download-cmake
> --download-parmetis --download-ptscotch.
>
> Running MUPMS in parallel using MPI also gave me a significant gain in
> performance (about 10 times on a single cluster node).
>
Again, this does not appear to make sense. The performance should be
limited by memory bandwidth, and a single cluster node will not usually have
10x the bandwidth of a CPU, although it might be possible with a very old
CPU.
It would help to understand the performance if you would send the output of
-log_view.
Thanks,
Matt
> Could you, please, advise me whether I can adjust some options for the
> direct solvers to improve performance? Should I try MUMPS in OpenMP mode?
>
> On Sat, Sep 19, 2020 at 7:40 AM Mark Adams <mfad...@lbl.gov> wrote:
>
>> As Jed said high frequency is hard. AMG, as-is, can be adapted (
>> https://link.springer.com/article/10.1007/s00466-006-0047-8) with
>> parameters.
>> AMG for convection: use richardson/sor and not chebyshev smoothers and in
>> smoothed aggregation (gamg) don't smooth (-pc_gamg_agg_nsmooths 0).
>> Mark
>>
>> On Sat, Sep 19, 2020 at 2:11 AM Alexey Kozlov <alexey.v.kozlo...@nd.edu>
>> wrote:
>>
>>> Thanks a lot! I'll check them out.
>>>
>>> On Sat, Sep 19, 2020 at 1:41 AM Barry Smith <bsm...@petsc.dev> wrote:
>>>
>>>>
>>>> These are small enough that likely sparse direct solvers are the best
>>>> use of your time and for general efficiency.
>>>>
>>>> PETSc supports 3 parallel direct solvers, SuperLU_DIST, MUMPs and
>>>> Pastix. I recommend configuring PETSc for all three of them and then
>>>> comparing them for problems of interest to you.
>>>>
>>>> --download-superlu_dist --download-mumps --download-pastix
>>>> --download-scalapack (used by MUMPS) --download-metis --download-parmetis
>>>> --download-ptscotch
>>>>
>>>> Barry
>>>>
>>>>
>>>> On Sep 18, 2020, at 11:28 PM, Alexey Kozlov <alexey.v.kozlo...@nd.edu>
>>>> wrote:
>>>>
>>>> Thanks for the tips! My matrix is complex and unsymmetric. My typical
>>>> test case has of the order of one million equations. I use a 2nd-order
>>>> finite-difference scheme with 19-point stencil, so my typical test case
>>>> uses several GB of RAM.
>>>>
>>>> On Fri, Sep 18, 2020 at 11:52 PM Jed Brown <j...@jedbrown.org> wrote:
>>>>
>>>>> Unfortunately, those are hard problems in which the "good" methods are
>>>>> technical and hard to make black-box. There are "sweeping" methods that
>>>>> solve on 2D "slabs" with PML boundary conditions, H-matrix based methods,
>>>>> and fancy multigrid methods. Attempting to solve with STRUMPACK is
>>>>> probably the easiest thing to try (--download-strumpack).
>>>>>
>>>>>
>>>>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATSOLVERSSTRUMPACK.html
>>>>>
>>>>> Is the matrix complex symmetric?
>>>>>
>>>>> Note that you can use a direct solver (MUMPS, STRUMPACK, etc.) for a
>>>>> 3D problem like this if you have enough memory. I'm assuming the memory
>>>>> or
>>>>> time is unacceptable and you want an iterative method with much lower
>>>>> setup
>>>>> costs.
>>>>>
>>>>> Alexey Kozlov <alexey.v.kozlo...@nd.edu> writes:
>>>>>
>>>>> > Dear all,
>>>>> >
>>>>> > I am solving a convected wave equation in a frequency domain. This
>>>>> equation
>>>>> > is a 3D Helmholtz equation with added first-order derivatives and
>>>>> mixed
>>>>> > derivatives, and with complex coefficients. The discretized PDE
>>>>> results in
>>>>> > a sparse linear system (about 10^6 equations) which is solved in
>>>>> PETSc. I
>>>>> > am having difficulty with the code convergence at high frequency,
>>>>> skewed
>>>>> > grid, and high Mach number. I suspect it may be due to the
>>>>> preconditioner I
>>>>> > use. I am currently using the ILU preconditioner with the number of
>>>>> fill
>>>>> > levels 2 or 3, and BCGS or GMRES solvers. I suspect the state of the
>>>>> art
>>>>> > has evolved and there are better preconditioners for Helmholtz-like
>>>>> > problems. Could you, please, advise me on a better preconditioner?
>>>>> >
>>>>> > Thanks,
>>>>> > Alexey
>>>>> >
>>>>> > --
>>>>> > Alexey V. Kozlov
>>>>> >
>>>>> > Research Scientist
>>>>> > Department of Aerospace and Mechanical Engineering
>>>>> > University of Notre Dame
>>>>> >
>>>>> > 117 Hessert Center
>>>>> > Notre Dame, IN 46556-5684
>>>>> > Phone: (574) 631-4335
>>>>> > Fax: (574) 631-8355
>>>>> > Email: akoz...@nd.edu
>>>>>
>>>>
>>>>
>>>> --
>>>> Alexey V. Kozlov
>>>>
>>>> Research Scientist
>>>> Department of Aerospace and Mechanical Engineering
>>>> University of Notre Dame
>>>>
>>>> 117 Hessert Center
>>>> Notre Dame, IN 46556-5684
>>>> Phone: (574) 631-4335
>>>> Fax: (574) 631-8355
>>>> Email: akoz...@nd.edu
>>>>
>>>>
>>>>
>>>
>>> --
>>> Alexey V. Kozlov
>>>
>>> Research Scientist
>>> Department of Aerospace and Mechanical Engineering
>>> University of Notre Dame
>>>
>>> 117 Hessert Center
>>> Notre Dame, IN 46556-5684
>>> Phone: (574) 631-4335
>>> Fax: (574) 631-8355
>>> Email: akoz...@nd.edu
>>>
>>
>
> --
> Alexey V. Kozlov
>
> Research Scientist
> Department of Aerospace and Mechanical Engineering
> University of Notre Dame
>
> 117 Hessert Center
> Notre Dame, IN 46556-5684
> Phone: (574) 631-4335
> Fax: (574) 631-8355
> Email: akoz...@nd.edu
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
