On Wed, Jul 29, 2020 at 6:30 AM Bin Liu <lbllm2...@hotmail.com> wrote:
> Thanks for your suggestions. Sure. I will check my codes of MatZeroRows > from a small problem. The program does not terminate during solving in > distributed memory, but repeating this error in run time. In fact, this is > the entire error message repeating itself during runtime. > Are you not checking the return codes for PETSc calls in your code? Matt > > > > > *From:* Matthew Knepley [mailto:knep...@gmail.com] > *Sent:* Wednesday, 29 July 2020 5:21 PM > *To:* Bin Liu <lbllm2...@hotmail.com> > *Cc:* Barry Smith <bsm...@petsc.dev>; petsc-users@mcs.anl.gov > *Subject:* Re: [petsc-users] solving saddle point problem in distributed > memory > > > > On Tue, Jul 28, 2020 at 11:33 PM Bin Liu <lbllm2...@hotmail.com> wrote: > > Dear Barry, > > > > Thanks for your suggestions. I just tested it. I still get the same error. > I attached it below: > > > > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [1]PETSC ERROR: Object is in wrong state > > [1]PETSC ERROR: Matrix is missing diagonal entry in the zeroed row 24214 > > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [1]PETSC ERROR: Petsc Release Version 3.9.1, Apr, 29, 2018 > > [1]PETSC ERROR: ./re100 on a named kdp-MS-7850 by kdp Wed Jul 29 11:26:57 > 2020 > > [1]PETSC ERROR: Configure options --prefix=/home/kdp/petsc --debugging=0 > --with-mpi-dir=/home/kdp/mpich --download-fblaslapack --download-metis > --download-parmetis --download-hypre=/home/kdp/hypre-2.14.0.tar.gz > --download-superlu_dist > > [1]PETSC ERROR: #1 MatZeroRows_SeqAIJ() line 1819 in > /home/kdp/petsc-3.9.1/src/mat/impls/aij/seq/aij.c > > [1]PETSC ERROR: #2 MatZeroRows() line 5710 in > /home/kdp/petsc-3.9.1/src/mat/interface/matrix.c > > [1]PETSC ERROR: #3 MatZeroRows_MPIAIJ() line 797 in > /home/kdp/petsc-3.9.1/src/mat/impls/aij/mpi/mpiaij.c > > [1]PETSC ERROR: #4 MatZeroRows() line 5710 in > /home/kdp/petsc-3.9.1/src/mat/interface/matrix.c > > > It still report missing diagonal entry. However I have epxlicitly assign > 0.0 to the diagonal element of the system matrix and solve it in sequential > mode without error. Where could possibly my wrong set up be? > > > > 1) Please send the complete error message. The stack trace here is cut off. > > > > 2) It does not look like it is failing in the solver, but in the call to > MatZeroRows(), which as it says, requires diagonal elements > > > > 3) It looks like you are not assembling the matrix correctly in parallel > > > > I recommend making as small a problem as possible, running on 1 and 2 > processes, and printing out the matrix using -mat_view to compare them > entry-by-entry. > > > > Thanks, > > > > Matt > > > > Best > > > ------------------------------ > > *From:* Barry Smith <bsm...@petsc.dev> > *Sent:* Wednesday, July 29, 2020 10:49 AM > *To:* Bin Liu <lbllm2...@hotmail.com> > *Cc:* petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov> > *Subject:* Re: [petsc-users] solving saddle point problem in distributed > memory > > > > > > No, PETSc direct solvers do not work in parallel. Only SuperLU_DIST, > MUMPS, and PasTix. > > > > -pc_type lu (or cholesky depending on the package) > -pc_factor_mat_solver_type superlu_dist or mumps or pastix > > > > to run the direct solver in parallel. > > > > > > > > On Jul 28, 2020, at 9:01 PM, Bin Liu <lbllm2...@hotmail.com> wrote: > > > > Dear Barry, > > > > Thanks for your explanation. Does it mean the default director solver LU > in PETSc can also run in the distributed memory mode? I only used iterative > solvers with preconditioners in distributed memory before. My experience > in using director solver in parallel is indeed limited, especially the > saddle point problem in parallel. I have done some search online, but I did > not find a working set up at this moment. Could you give a sample setup of > the direct solver for parallel run? It is really appreciated. > > > > *From:* Barry Smith [mailto:bsm...@petsc.dev <bsm...@petsc.dev>] > *Sent:* Wednesday, 29 July 2020 9:48 AM > *To:* Bin Liu <lbllm2...@hotmail.com> > *Cc:* petsc-users@mcs.anl.gov > *Subject:* Re: [petsc-users] solving saddle point problem in distributed > memory > > > > > > SuperLU_DIST won't "magically" run on saddle point problems. It only > does limited pivoting, realistically a parallel LU cannot always do > complete pivoting or it becomes a sequential algorithm. For parallel you > need to use PCFIELDSPLIT, for sequential you can use SuperLU (not > SuperLU_DIST) since it can do more pivoting, being a sequential algorithm. > > > > Barry > > > > > > On Jul 28, 2020, at 7:59 PM, Bin Liu <lbllm2...@hotmail.com> wrote: > > > > Thanks for your tutorials. Yes. I tried PCFIELDSPLIT. However, it only > works for sequential runs. When I run the code in distributed memory, it > reports errors. In fact, the essence of my wonder is (a) how to set up > superlu_dist in petsc for solving saddle point problem in distributed > memory? (b) does the direct solvers in petsc can run in distributed memory > for solving saddle point problem? > > > > *From:* Stefano Zampini [mailto:stefano.zamp...@gmail.com > <stefano.zamp...@gmail.com>] > *Sent:* Tuesday, 28 July 2020 6:55 PM > *To:* Bin Liu <lbllm2...@hotmail.com> > *Cc:* petsc-users@mcs.anl.gov > *Subject:* Re: [petsc-users] solving saddle point problem in distributed > memory > > > > If you want advice you should post the error trace PETSc reports. > > > > Anyway, solving Stokes is not so trivial (without direct solvers, you may > need mesh dependent information), but we have examples for it > > > > https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex42.c > > https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex43.c > > https://gitlab.com/petsc/petsc/-/blob/master/src/snes/tutorials/ex69.c > > > > If you scroll to the end of those files, you see a bunch of possible > options either using PCFIELDSPLIT, PCBDDC or KSPFETIDP. > > > > > > On Jul 28, 2020, at 12:37 PM, Bin Liu <lbllm2...@hotmail.com> wrote: > > > > I would like to solve a saddle point problem arising from the stokes > equation. I got successful to use the direct solvers in sequential runs. > However, I would like to extend it for distributed memory computation. I > tried to use superlu_dist, but the program returns errors. Is it possible > to solve a saddle point problem in distributed memory using superlu_dist? > Could anyone give a simple sample code to set up the parameters of the > solver? > > > > Thanks > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
