I should have read your initial email more carefully:
----
Loop over block PCs {
PCSetType(subPc, PCLU); set subPC type
PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU
SetPardisoParams("pardiso"); set mkl related options including
-mat_mkl_pardiso_65 equal to the desired #threads; return code after
PetscOptionsSetValue is ok, so I presume the option is set correctly.
PCSetFromOptions(subPc);
}
Then what I see under the debugger
PCSetUp after a series of calls gets to MatLUFactorSymbolic_AIJMKL_PARDISO
which in turn calls MatSetFromOptions_MKL_PARDISO(F, A) and there
in these lines it seems to ignore what I’ve set for -mat_mkl_pardiso_65 and
sets #threads to the default 1
PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads
to use within PARDISO", "None", threads, &threads, &flg));
if (flg) PetscSetMKL_PARDISOThreads((int)threads);
-----
PCSetFromOptions() actually does not call MatSetFromOptions() on the factored
matrix. There is no call to MatSetFromOptions() from PCSetFromOptions() or
PetscSetFromOptions_Factor() because the factored matrix F does not yet exist
(it is created in PCSetUp()). This is why your call to
PCSetFromOptions(subPc); does not process any of your mat mkl options.
Since MatSetFromOptions() cannot be called from PCSetFromOptions(),
MatSetFromOptions_MKL_PARDISO(F, A) is called in
MatFactorSymbolic_AIJMKL_PARDISO_Private() as you noted when the PC is being
set up.
You say "here in these lines it seems to ignore what I’ve set for
-mat_mkl_pardiso_65 and sets #threads to the default 1"
This is the part a do not understand, if your values have been inserted in the
database earlier in the code and remain in the database they should get
processed inside the
MatSetFromOptions_MKL_PARDISO(F, A) called by
MatFactorSymbolic_AIJMKL_PARDISO_Private().
Can you please put a break point at MatSetFromOptions_MKL_PARDISO(F, A) and
then when it breaks put
call PetscOptionsView(0,0)
it should print out the contents of the options database so you can verify if
the options you added are there. You can then step through the code in
MatSetFromOptions_MKL_PARDISO to see what is happening as it tries to acess the
database to see why it skips
PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads
to use within MKL PARDISO", "None", threads, &threads, &flg));
if (flg) PetscSetMKL_PARDISOThreads((int)threads);
> On Sep 27, 2024, at 11:58 AM, Voinov, Boris <boris.voi...@intel.com> wrote:
>
> Thanks a lot Barry,
>
> It looks like something wrong with my own settings though. It doesn’t make an
> effect. I mean this call
> SetPardisoParams("pardiso");
> followed by
> PCSetFromOptions(subPc);
> doesn’t end up with a call of PetscSetMKL_PARDISOThreads()
> This function is called later when PCSetUp is invoked.
>
> Eventually what I see in the end is
> Option left: name:-mat_mkl_pardiso_65 value: 8 source: code listed among
> unused database options
> That is what I’ve set with my own settings, meaning it’s in the database, I
> guess.
>
> Could it be that it hits a “not proper” section in the options database and
> is not being used by PCSetUp?
>
> Best regards,
> Boris
>
> From: Barry Smith <bsm...@petsc.dev <mailto:bsm...@petsc.dev>>
> Sent: Friday, September 27, 2024 4:21 PM
> To: Voinov, Boris <boris.voi...@intel.com <mailto:boris.voi...@intel.com>>
> Cc: petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>
> Subject: Re: [petsc-users] Ask for help in tracking down bugs - setting
> #threads for pardiso block LU solver
>
>
> Boris,
>
> There is nothing obviously wrong with what you outline, so let's dive
> down into the code with the debugger and see what is happening.
>
> PetscSetMKL_PARDISOThreads() set from options is called when the command line
> option is provided, and this calls
>
> PETSC_EXTERN void PetscSetMKL_PARDISOThreads(int threads)
> {
> mkl_domain_set_num_threads(threads, MKL_DOMAIN_PARDISO);
> }
>
> so in the debugger can you check that mkl_domain_set_num_threads() gets
> called with your requested number of threads?
>
> BTW: Since you are hardwiring the use of Pardiso in the code with function
> calls you couldsimply call PetscSetMKL_PARDISOThreads() directly
> after your line of code
>
> PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for LU
>
> instead of feeding it in through the options database.
>
> Let us know how this turns out
>
> Barry
>
>
>
>
>
>
> On Sep 27, 2024, at 7:24 AM, Voinov, Boris <boris.voi...@intel.com
> <mailto:boris.voi...@intel.com>> wrote:
>
> Hello,
>
> Could you please help me to figure out what’s wrong with the way I’m trying
> to make mkl pardiso use the number of threads more than one while setting it
> as a sub PC solver for ASM preconditioner.
> Here is how I do this
> Loop over block PCs {
> PCSetType(subPc, PCLU); set subPC type
> PCFactorSetMatSolverType(subPc, MATSOLVERMKL_PARDISO); set solver for
> LU
> SetPardisoParams("pardiso"); set mkl related options including
> -mat_mkl_pardiso_65 equal to the desired #threads; return code after
> PetscOptionsSetValue is ok, so I presume the option is set correctly.
> PCSetFromOptions(subPc);
> }
>
> Then what I see under the debugger
> PCSetUp after a series of calls gets to MatLUFactorSymbolic_AIJMKL_PARDISO
> which in turn calls MatSetFromOptions_MKL_PARDISO(F, A) and there
> in these lines it seems to ignore what I’ve set for -mat_mkl_pardiso_65 and
> sets #threads to the default 1
> PetscCall(PetscOptionsInt("-mat_mkl_pardiso_65", "Suggested number of threads
> to use within PARDISO", "None", threads, &threads, &flg));
> if (flg) PetscSetMKL_PARDISOThreads((int)threads);
>
> This is about petsc-3.20.
> PS when I do this in a standalone code which reads a matrix and solves the
> linear system and set -mat_mkl_pardiso_65 in the command line it’s all right
> but I need the number of threads to be set in other way in my code.
>
> Thank you and best regards,
> Boris
>
> -------------------------------------------------------------
> Intel Ireland Limited (Branch)
> Collinstown Industrial Park, Leixlip, County Kildare, Ireland
> Registered Number: E902934
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>
>
> -------------------------------------------------------------
> Intel Ireland Limited (Branch)
> Collinstown Industrial Park, Leixlip, County Kildare, Ireland
> Registered Number: E902934
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>
