Use -mg_coarse_pc_type svd
Verify that when you apply the restriction to a constant vector (all ones)
it returns a constant vector, maybe scaled
Verify that when you apply the interpolation to a constant vector it returns
a constant vector
> On Oct 2, 2024, at 12:29 PM, Gautam Luhana <gkluh...@cs.ubc.ca> wrote:
>
> GMRES is similar i.e. converges but iteration counts go +1-2 with every
> doubling of the mesh.
>
> I am attaching the output from a cg and gmres run for a single stokes
> iteration with a velocity and pressure subdomain solves, subsequent
> iterations are similar. The pressure subdomain ksp is titled pre_mgcg.
>
> Adding more information in case it is relevant to isolating the issue:
>
> -> The coarse grid operators are formed by setting PCMGSetGalerkin to
> PC_MG_GALERKIN_BOTH.
>
> -> The interpolation / restriction operators are formed using the
> DMCreateInterpolation and DMCreateRestriction functions on coarsened DMs
> (DMStag implementation) and sent to PCMGSetInterpolation/PCMGSetRestriction.
>
> Thanks for the help!
>
> -Gautam
>
>
> On 2024-10-02 7:42 a.m., Barry Smith wrote:
>> [CAUTION: Non-UBC Email]
>>
>> Using
>>
>> PetscCall(MatNullSpaceCreate(PetscObjectComm((PetscObject)dmPressure),
>> PETSC_TRUE, 0, nullptr, &matNullSpace));
>>
>> should be fine (the other approach is equivalent).
>>
>> Any change if you use -ksp_type gmres?
>>
>> Please send all the output from a sample run with -ksp_monitor_true_residual
>> -ksp_view
>>
>>
>>
>>
>>
>>
>>
>>> On Oct 1, 2024, at 6:42 PM, Gautam Luhana <gkluh...@cs.ubc.ca> wrote:
>>>
>>> I did think that I accounted for that but now I see that I am getting the
>>> same number of iterations in the pressure space with and without the call
>>> to MatSetNullspace so maybe I am not doing that correctly and that's where
>>> the problem is.
>>>
>>> I tried it in the following way first:
>>>
>>> PetscCall(MatNullSpaceCreate(PetscObjectComm((PetscObject)dmPressure),
>>> PETSC_TRUE, 0, nullptr, &matNullSpace));
>>>
>>> PetscCall(MatSetNullSpace(*A, matNullSpace));
>>>
>>> and then also as:
>>>
>>> PetscCall(DMGetGlobalVector(dmPressure, &constantPressure));
>>> PetscCall(VecSet(constantPressure, 1.0));
>>> PetscCall(VecNorm(constantPressure, NORM_2, &nrm));
>>> PetscCall(VecScale(constantPressure, 1.0 / nrm));
>>> PetscCall(DMCreateGlobalVector(dmPressure, &basis));
>>> PetscCall(VecCopy(constantPressure, basis));
>>> PetscCall(MatNullSpaceCreate(PetscObjectComm((PetscObject)dmPressure),
>>> PETSC_FALSE, 1, &basis, &matNullSpace));
>>> PetscCall(MatSetNullSpace(*A, matNullSpace));
>>>
>>> Is this the right way? are there other options that need to be set?
>>>
>>> -Gautam
>>>
>>> On 2024-10-01 12:47 p.m., Barry Smith wrote:
>>>> [CAUTION: Non-UBC Email]
>>>>
>>>> Are you handling the null space? Like with MatSetNullSpace()?
>>>>
>>>> Can you try src/ksp/ksp/tutorials/ex32.c it solves the same problem but
>>>> with a simpler DMDA. I don't see any reason why using DMStag with the
>>>> cell-centered values should be different.
>>>>
>>>> Barry
>>>>
>>>>
>>>>> On Oct 1, 2024, at 3:37 PM, Gautam Luhana <gkluh...@cs.ubc.ca> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I am solving the stokes equations on the MAC grid using DMStag with an
>>>>> approximate Schur complement based preconditioner. The preconditioner
>>>>> needs a subdomain solve of the Poisson equation on both the velocity as
>>>>> well as the pressure space. I am using multigrid preconditioned CG for
>>>>> both the subdomain solves. While the subdomain solve on the velocity
>>>>> space converges in 4-5 iterations and is independent of mesh-size, the
>>>>> pressure solve is not mesh-size independent. Relevant information:
>>>>>
>>>>> Pressure space operator has Neumann boundary conditions.
>>>>>
>>>>> Coarse-grid operators are being generated using the Galerkin approach (A
>>>>> = P*Af*Pt) for both of the subdomains.
>>>>>
>>>>> For the pressure space, the jacobi and sor smoothers were tried with
>>>>> several omega and damping parameter choices.
>>>>>
>>>>> Symmetric smoothing (same number of pre- and post-smoothing iterations)
>>>>>
>>>>> Has anyone tried solving the poisson operator on the cell-centered
>>>>> pressure space with DMStag? Or know what might be going wrong with the
>>>>> mg-cg convergence?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> -Gautam
>>>>>
isNull : 1
MatNullSpace Object: 1 MPI process
type not yet set
Contains 0 vectors and the constant
debug: 0
mu,rho,spring_stiffness: 0.01, 1, 39300
Full System lx,Lx: 0, 1
Full System ly,Ly: 0, 1
Full System Nx,Ny: 128, 128
Full System N_struct: 93
Full System mg_levels: 5
Full System hx,hy: 0.0078125, 0.0078125
Full System dt: 0.005
isNull : 1
MatNullSpace Object: 1 MPI process
type not yet set
Contains 0 vectors and the constant
Truncated type: 1
Truncated mg_levels: 2
Truncated lx,Lx: 0.375,0.625
Truncated ly,Ly: 0.375,0.625
Truncated Nx,Ny: 32,32
Truncated hx,hy for A_s: 0.0078125,0.0078125
Truncated hx,hy for J: 0.0078125,0.0078125
Residual norms for ib_stokes_ solve.
0 KSP unpreconditioned resid norm 2.270243941051e+03 true resid norm
2.270243941051e+03 ||r(i)||/||b|| 1.000000000000e+00
Residual norms for stokes_ solve.
0 KSP unpreconditioned resid norm 4.775149126311e+03 true resid norm
4.775149126311e+03 ||r(i)||/||b|| 1.000000000000e+00
Residual norms for vel_mgcg_ solve.
0 KSP natural resid norm 6.267432921678e-02 true resid norm
1.000000000000e+00 ||r(i)||/||b|| 1.000000000000e+00
1 KSP natural resid norm 1.131540017491e-04 true resid norm
2.248640407923e-03 ||r(i)||/||b|| 2.248640407923e-03
2 KSP natural resid norm 2.108903131764e-07 true resid norm
4.316116468476e-06 ||r(i)||/||b|| 4.316116468476e-06
Residual norms for pre_mgcg_ solve.
0 KSP unpreconditioned resid norm 5.190249753491e+01 true resid norm
5.190249753491e+01 ||r(i)||/||b|| 1.000000000000e+00
1 KSP unpreconditioned resid norm 2.613646142274e+01 true resid norm
2.613646142274e+01 ||r(i)||/||b|| 5.035684728882e-01
2 KSP unpreconditioned resid norm 5.727486843402e+00 true resid norm
5.727486843402e+00 ||r(i)||/||b|| 1.103508909094e-01
3 KSP unpreconditioned resid norm 1.235965380015e+00 true resid norm
1.235965380015e+00 ||r(i)||/||b|| 2.381321590900e-02
4 KSP unpreconditioned resid norm 3.159901559858e-01 true resid norm
3.159901559858e-01 ||r(i)||/||b|| 6.088149337578e-03
5 KSP unpreconditioned resid norm 1.156369368898e-01 true resid norm
1.156369368898e-01 ||r(i)||/||b|| 2.227964787476e-03
6 KSP unpreconditioned resid norm 4.084491453854e-02 true resid norm
4.084491453854e-02 ||r(i)||/||b|| 7.869547031156e-04
7 KSP unpreconditioned resid norm 1.687558469769e-02 true resid norm
1.687558469768e-02 ||r(i)||/||b|| 3.251401281091e-04
8 KSP unpreconditioned resid norm 8.895600308256e-03 true resid norm
8.895600308257e-03 ||r(i)||/||b|| 1.713906022013e-04
9 KSP unpreconditioned resid norm 3.977199533817e-03 true resid norm
5.190249753491e+01 ||r(i)||/||b|| 1.000000000000e+00
KSP Object: (pre_mgcg_) 1 MPI process
type: gmres
restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization
with no iterative refinement
happy breakdown tolerance 1e-30
maximum iterations=20, initial guess is zero
tolerances: relative=0.0001, absolute=1e-50, divergence=10000.
right preconditioning
using UNPRECONDITIONED norm type for convergence test
PC Object: (pre_mgcg_) 1 MPI process
type: mg
type is MULTIPLICATIVE, levels=5 cycles=v
Cycles per PCApply=1
Using Galerkin computed coarse grid matrices
Coarse grid solver -- level 0 -------------------------------
KSP Object: (pre_mgcg_mg_coarse_) 1 MPI process
type: preonly
maximum iterations=10000, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (pre_mgcg_mg_coarse_) 1 MPI process
type: ilu
out-of-place factorization
0 levels of fill
tolerance for zero pivot 2.22045e-14
matrix ordering: natural
factor fill ratio given 1., needed 1.
Factored matrix follows:
Mat Object: (pre_mgcg_mg_coarse_) 1 MPI process
type: seqaij
rows=64, cols=64
package used to perform factorization: petsc
total: nonzeros=484, allocated nonzeros=484
not using I-node routines
linear system matrix = precond matrix:
Mat Object: 1 MPI process
type: seqaij
rows=64, cols=64
total: nonzeros=484, allocated nonzeros=484
total number of mallocs used during MatSetValues calls=0
not using I-node routines
Down solver (pre-smoother) on level 1 -------------------------------
KSP Object: (pre_mgcg_mg_levels_1_) 1 MPI process
type: richardson
damping factor=1.
maximum iterations=2, nonzero initial guess
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (pre_mgcg_mg_levels_1_) 1 MPI process
type: sor
type = local_symmetric, iterations = 1, local iterations = 1, omega =
1.85
linear system matrix = precond matrix:
Mat Object: 1 MPI process
type: seqaij
rows=256, cols=256
total: nonzeros=2116, allocated nonzeros=2116
total number of mallocs used during MatSetValues calls=0
not using I-node routines
Up solver (post-smoother) same as down solver (pre-smoother)
Down solver (pre-smoother) on level 2 -------------------------------
KSP Object: (pre_mgcg_mg_levels_2_) 1 MPI process
type: richardson
damping factor=1.
maximum iterations=2, nonzero initial guess
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (pre_mgcg_mg_levels_2_) 1 MPI process
type: sor
type = local_symmetric, iterations = 1, local iterations = 1, omega =
1.85
linear system matrix = precond matrix:
Mat Object: 1 MPI process
type: seqaij
rows=1024, cols=1024
total: nonzeros=8836, allocated nonzeros=8836
total number of mallocs used during MatSetValues calls=0
not using I-node routines
Up solver (post-smoother) same as down solver (pre-smoother)
Down solver (pre-smoother) on level 3 -------------------------------
KSP Object: (pre_mgcg_mg_levels_3_) 1 MPI process
type: richardson
damping factor=1.
maximum iterations=2, nonzero initial guess
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (pre_mgcg_mg_levels_3_) 1 MPI process
type: sor
type = local_symmetric, iterations = 1, local iterations = 1, omega =
1.85
linear system matrix = precond matrix:
Mat Object: 1 MPI process
type: seqaij
rows=4096, cols=4096
total: nonzeros=36100, allocated nonzeros=36100
total number of mallocs used during MatSetValues calls=0
not using I-node routines
Up solver (post-smoother) same as down solver (pre-smoother)
Down solver (pre-smoother) on level 4 -------------------------------
KSP Object: (pre_mgcg_mg_levels_4_) 1 MPI process
type: richardson
damping factor=1.
maximum iterations=2, nonzero initial guess
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (pre_mgcg_mg_levels_4_) 1 MPI process
type: sor
type = local_symmetric, iterations = 1, local iterations = 1, omega =
1.85
linear system matrix = precond matrix:
Mat Object: 1 MPI process
type: seqaij
rows=16384, cols=16384
total: nonzeros=145924, allocated nonzeros=145924
total number of mallocs used during MatSetValues calls=0
has attached null space
not using I-node routines
Up solver (post-smoother) same as down solver (pre-smoother)
linear system matrix = precond matrix:
Mat Object: 1 MPI process
type: seqaij
rows=16384, cols=16384
total: nonzeros=145924, allocated nonzeros=145924
total number of mallocs used during MatSetValues calls=0
has attached null space
not using I-node routines
1 KSP unpreconditioned resid norm 3.035714337034e-01 true resid norm
3.035714337034e-01 ||r(i)||/||b|| 6.357318393070e-05
1 KSP unpreconditioned resid norm 6.888912829419e+02 true resid norm
6.888912829406e+02 ||r(i)||/||b|| 3.034437271184e-01
IB Iterations: 1
Total S2 Iterations: 7
Avg S2 Iterations: 7
Total Full Stokes Iterations: 1
Avg Full Stokes Iterations: 1
Total Truncated Stokes Iterations: 25
Avg Truncated Stokes Iterations per S2 iteration: 3.57143
WARNING! There are options you set that were not used!
WARNING! could be spelling mistake, etc!
There are 3 unused database options. They are:
Option left: name:-pre_mgcg_mg_coarse_ksp_rtol value: 1e-12 source: file
Option left: name:-stokes_inner_ksp_max_it value: 2 source: file
Option left: name:-stokes_inner_ksp_rtol value: 1e-6 source: file
isNull : 1
MatNullSpace Object: 1 MPI process
type not yet set
Contains 0 vectors and the constant
debug: 0
mu,rho,spring_stiffness: 0.01, 1, 39300
Full System lx,Lx: 0, 1
Full System ly,Ly: 0, 1
Full System Nx,Ny: 128, 128
Full System N_struct: 93
Full System mg_levels: 5
Full System hx,hy: 0.0078125, 0.0078125
Full System dt: 0.005
isNull : 1
MatNullSpace Object: 1 MPI process
type not yet set
Contains 0 vectors and the constant
Truncated type: 1
Truncated mg_levels: 2
Truncated lx,Lx: 0.375,0.625
Truncated ly,Ly: 0.375,0.625
Truncated Nx,Ny: 32,32
Truncated hx,hy for A_s: 0.0078125,0.0078125
Truncated hx,hy for J: 0.0078125,0.0078125
Residual norms for ib_stokes_ solve.
0 KSP unpreconditioned resid norm 2.270243941051e+03 true resid norm
2.270243941051e+03 ||r(i)||/||b|| 1.000000000000e+00
Residual norms for stokes_ solve.
0 KSP unpreconditioned resid norm 4.775149126311e+03 true resid norm
4.775149126311e+03 ||r(i)||/||b|| 1.000000000000e+00
Residual norms for vel_mgcg_ solve.
0 KSP natural resid norm 6.267432921678e-02 true resid norm
1.000000000000e+00 ||r(i)||/||b|| 1.000000000000e+00
1 KSP natural resid norm 1.131540017491e-04 true resid norm
2.248640407923e-03 ||r(i)||/||b|| 2.248640407923e-03
2 KSP natural resid norm 2.108903131764e-07 true resid norm
4.316116468476e-06 ||r(i)||/||b|| 4.316116468476e-06
Residual norms for pre_mgcg_ solve.
0 KSP unpreconditioned resid norm 5.190249753491e+01 true resid norm
5.190249753491e+01 ||r(i)||/||b|| 1.000000000000e+00
1 KSP unpreconditioned resid norm 2.977882401331e+01 true resid norm
2.977882401331e+01 ||r(i)||/||b|| 5.737454925609e-01
2 KSP unpreconditioned resid norm 5.833014669614e+00 true resid norm
5.833014669614e+00 ||r(i)||/||b|| 1.123840845172e-01
3 KSP unpreconditioned resid norm 1.444102892017e+00 true resid norm
1.444102892017e+00 ||r(i)||/||b|| 2.782337961763e-02
4 KSP unpreconditioned resid norm 4.509577825604e-01 true resid norm
4.509577825604e-01 ||r(i)||/||b|| 8.688556504571e-03
5 KSP unpreconditioned resid norm 1.758864538938e-01 true resid norm
1.758864538938e-01 ||r(i)||/||b|| 3.388785939934e-03
6 KSP unpreconditioned resid norm 5.102840236970e-02 true resid norm
5.102840236970e-02 ||r(i)||/||b|| 9.831588997308e-04
7 KSP unpreconditioned resid norm 2.800704127003e-02 true resid norm
2.800704127003e-02 ||r(i)||/||b|| 5.396087394675e-04
8 KSP unpreconditioned resid norm 1.457526347604e-02 true resid norm
1.457526347604e-02 ||r(i)||/||b|| 2.808200793466e-04
9 KSP unpreconditioned resid norm 6.285550574266e-03 true resid norm
6.285550574266e-03 ||r(i)||/||b|| 1.211030465352e-04
10 KSP unpreconditioned resid norm 3.456018793076e-03 true resid norm
3.456018793076e-03 ||r(i)||/||b|| 6.658675318565e-05
KSP Object: (pre_mgcg_) 1 MPI process
type: cg
maximum iterations=20, initial guess is zero
tolerances: relative=0.0001, absolute=1e-50, divergence=10000.
left preconditioning
using UNPRECONDITIONED norm type for convergence test
PC Object: (pre_mgcg_) 1 MPI process
type: mg
type is MULTIPLICATIVE, levels=5 cycles=v
Cycles per PCApply=1
Using Galerkin computed coarse grid matrices
Coarse grid solver -- level 0 -------------------------------
KSP Object: (pre_mgcg_mg_coarse_) 1 MPI process
type: preonly
maximum iterations=10000, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (pre_mgcg_mg_coarse_) 1 MPI process
type: ilu
out-of-place factorization
0 levels of fill
tolerance for zero pivot 2.22045e-14
matrix ordering: natural
factor fill ratio given 1., needed 1.
Factored matrix follows:
Mat Object: (pre_mgcg_mg_coarse_) 1 MPI process
type: seqaij
rows=64, cols=64
package used to perform factorization: petsc
total: nonzeros=484, allocated nonzeros=484
not using I-node routines
linear system matrix = precond matrix:
Mat Object: 1 MPI process
type: seqaij
rows=64, cols=64
total: nonzeros=484, allocated nonzeros=484
total number of mallocs used during MatSetValues calls=0
not using I-node routines
Down solver (pre-smoother) on level 1 -------------------------------
KSP Object: (pre_mgcg_mg_levels_1_) 1 MPI process
type: richardson
damping factor=1.
maximum iterations=2, nonzero initial guess
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (pre_mgcg_mg_levels_1_) 1 MPI process
type: sor
type = local_symmetric, iterations = 1, local iterations = 1, omega =
1.85
linear system matrix = precond matrix:
Mat Object: 1 MPI process
type: seqaij
rows=256, cols=256
total: nonzeros=2116, allocated nonzeros=2116
total number of mallocs used during MatSetValues calls=0
not using I-node routines
Up solver (post-smoother) same as down solver (pre-smoother)
Down solver (pre-smoother) on level 2 -------------------------------
KSP Object: (pre_mgcg_mg_levels_2_) 1 MPI process
type: richardson
damping factor=1.
maximum iterations=2, nonzero initial guess
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (pre_mgcg_mg_levels_2_) 1 MPI process
type: sor
type = local_symmetric, iterations = 1, local iterations = 1, omega =
1.85
linear system matrix = precond matrix:
Mat Object: 1 MPI process
type: seqaij
rows=1024, cols=1024
total: nonzeros=8836, allocated nonzeros=8836
total number of mallocs used during MatSetValues calls=0
not using I-node routines
Up solver (post-smoother) same as down solver (pre-smoother)
Down solver (pre-smoother) on level 3 -------------------------------
KSP Object: (pre_mgcg_mg_levels_3_) 1 MPI process
type: richardson
damping factor=1.
maximum iterations=2, nonzero initial guess
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (pre_mgcg_mg_levels_3_) 1 MPI process
type: sor
type = local_symmetric, iterations = 1, local iterations = 1, omega =
1.85
linear system matrix = precond matrix:
Mat Object: 1 MPI process
type: seqaij
rows=4096, cols=4096
total: nonzeros=36100, allocated nonzeros=36100
total number of mallocs used during MatSetValues calls=0
not using I-node routines
Up solver (post-smoother) same as down solver (pre-smoother)
Down solver (pre-smoother) on level 4 -------------------------------
KSP Object: (pre_mgcg_mg_levels_4_) 1 MPI process
type: richardson
damping factor=1.
maximum iterations=2, nonzero initial guess
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (pre_mgcg_mg_levels_4_) 1 MPI process
type: sor
type = local_symmetric, iterations = 1, local iterations = 1, omega =
1.85
linear system matrix = precond matrix:
Mat Object: 1 MPI process
type: seqaij
rows=16384, cols=16384
total: nonzeros=145924, allocated nonzeros=145924
total number of mallocs used during MatSetValues calls=0
has attached null space
not using I-node routines
Up solver (post-smoother) same as down solver (pre-smoother)
linear system matrix = precond matrix:
Mat Object: 1 MPI process
type: seqaij
rows=16384, cols=16384
total: nonzeros=145924, allocated nonzeros=145924
total number of mallocs used during MatSetValues calls=0
has attached null space
not using I-node routines
1 KSP unpreconditioned resid norm 2.358306765752e-01 true resid norm
2.358306765752e-01 ||r(i)||/||b|| 4.938708097634e-05
1 KSP unpreconditioned resid norm 5.332451126777e+02 true resid norm
5.332451126795e+02 ||r(i)||/||b|| 2.348844998712e-01
IB Iterations: 1
Total S2 Iterations: 7
Avg S2 Iterations: 7
Total Full Stokes Iterations: 1
Avg Full Stokes Iterations: 1
Total Truncated Stokes Iterations: 25
Avg Truncated Stokes Iterations per S2 iteration: 3.57143
WARNING! There are options you set that were not used!
WARNING! could be spelling mistake, etc!
There are 3 unused database options. They are:
Option left: name:-pre_mgcg_mg_coarse_ksp_rtol value: 1e-12 source: file
Option left: name:-stokes_inner_ksp_max_it value: 2 source: file
Option left: name:-stokes_inner_ksp_rtol value: 1e-6 source: file
