Sanjay,
You have one more reason to use VecScatter, which is heavily used and
well-tested.
--Junchao Zhang
On Wed, Jun 5, 2019 at 5:47 PM Sanjay Govindjee
mailto:s...@berkeley.edu>> wrote:
I found the bug (naturally in my own code). When I made the MPI_Wait( )
changes, I missed one location
Thanks for letting us know, now we can rest easy.
> On Jun 5, 2019, at 1:00 PM, Zhang, Junchao wrote:
>
> OK, I see. I mistakenly read PetscMemoryGetCurrentUsage as
> PetscMallocGetCurrentUsage. You should also do PetscMallocGetCurrentUsage(),
> so that we know whether the increased me
I found the bug (naturally in my own code). When I made the MPI_Wait( )
changes, I missed one location where this
was needed. See the attached graphs for openmpi and mpich using
CG+Jacobi and GMRES+BJacobi.
Interesting that openmpi did not care about this but mpich did. Also
interesting tha
Are you reusing the same KSP the whole time, just making calls to KSPSolve,
or are you creating a new KSP object?
Do you make any calls to KSPReset()?
Are you doing any MPI_Comm_dup()?
Are you attaching any attributes to MPI communicators?
Thanks
> On Jun 5, 2019, at 1:18 AM,
OK, I see. I mistakenly read PetscMemoryGetCurrentUsage as
PetscMallocGetCurrentUsage. You should also do PetscMallocGetCurrentUsage(),
so that we know whether the increased memory is allocated by PETSc.
On Wed, Jun 5, 2019, 9:58 AM Sanjay GOVINDJEE
mailto:s...@berkeley.edu>> wrote:
PetscMemo
PetscMemoryGetCurrentUsage( ) is just a cover for rgetusage( ), so the use of
the function is unrelated to Petsc. The only difference here is mpich versus
openmpi.
Notwithstanding, I can make a plot of the sum of the deltas around kspsolve.
Sent from my iPad
> On Jun 5, 2019, at 7:22 AM, Zhang
Junchao,
I won't be feasible to share the code but I will run a similar test
as you have done (large problem); I will
try with both MPICH and OpenMPI. I also agree that deltas are not ideal
as there they do not account for latency in the freeing of memory
etc. But I will note when we have th
On Mon, Jun 3, 2019 at 6:56 PM Zhang, Junchao via petsc-users <
petsc-users@mcs.anl.gov> wrote:
> On Mon, Jun 3, 2019 at 5:23 PM Stefano Zampini
> wrote:
>
>>
>>
>> On Jun 4, 2019, at 1:17 AM, Zhang, Junchao via petsc-users <
>> petsc-users@mcs.anl.gov> wrote:
>>
>> Sanjay & Barry,
>> Sorry, I
On Mon, Jun 3, 2019 at 5:23 PM Stefano Zampini
mailto:stefano.zamp...@gmail.com>> wrote:
On Jun 4, 2019, at 1:17 AM, Zhang, Junchao via petsc-users
mailto:petsc-users@mcs.anl.gov>> wrote:
Sanjay & Barry,
Sorry, I made a mistake that I said I could reproduced Sanjay's experiments.
I found
Sanjay & Barry,
Sorry, I made a mistake that I said I could reproduced Sanjay's experiments.
I found 1) to correctly use PetscMallocGetCurrentUsage() when petsc is
configured without debugging, I have to add -malloc to run the program. 2) I
have to instrument the code outside of KSPSolve(). In
On Sat, Jun 1, 2019 at 3:21 AM Sanjay Govindjee
mailto:s...@berkeley.edu>> wrote:
Barry,
If you look at the graphs I generated (on my Mac), you will see that
OpenMPI and MPICH have very different values (along with the fact that
MPICH does not seem to adhere
to the standard (for releasing MPI_
Barry,
If you look at the graphs I generated (on my Mac), you will see that
OpenMPI and MPICH have very different values (along with the fact that
MPICH does not seem to adhere
to the standard (for releasing MPI_ISend resources following and MPI_Wait).
-sanjay
PS: I agree with Barry's asses
Junchao,
This is insane. Either the OpenMPI library or something in the OS
underneath related to sockets and interprocess communication is grabbing
additional space for each round of MPI communication! Does MPICH have the same
values or different values than OpenMP? When you run on Li
On Fri, May 31, 2019 at 3:48 PM Sanjay Govindjee via petsc-users
mailto:petsc-users@mcs.anl.gov>> wrote:
Thanks Stefano.
Reading the manual pages a bit more carefully,
I think I can see what I should be doing. Which should be roughly to
1. Set up target Seq vectors on PETSC_COMM_SELF
2. Use I
Thanks Stefano.
Reading the manual pages a bit more carefully,
I think I can see what I should be doing. Which should be roughly to
1. Set up target Seq vectors on PETSC_COMM_SELF
2. Use ISCreateGeneral to create ISs for the target Vecs and the source
Vec which will be MPI on PETSC_COMM_WORLD
Yes, the issue is running out of memory on long runs.
Perhaps some clean-up happens latter when the memory pressure builds but
that
is a bit non-ideal.
-sanjay
On 5/31/19 12:53 PM, Zhang, Junchao wrote:
Sanjay,
I tried petsc with MPICH and OpenMPI on my Macbook. I
inserted PetscMemoryGetCurr
Sanjay,
I tried petsc with MPICH and OpenMPI on my Macbook. I inserted
PetscMemoryGetCurrentUsage/PetscMallocGetCurrentUsage at the beginning and end
of KSPSolve and then computed the delta and summed over processes. Then I
tested with src/ts/examples/tutorials/advection-diffusion-reaction/ex5.c
Matt,
Here is the process as it currently stands:
1) I have a PETSc Vec (sol), which come from a KSPSolve
2) Each processor grabs its section of sol via VecGetOwnershipRange and
VecGetArrayReadF90
and inserts parts of its section of sol in a local array (locarr) using
a complex but easily co
Hi Juanchao,
Thanks for the hints below, they will take some time to absorb as the
vectors that are being moved around
are actually partly petsc vectors and partly local process vectors.
Attached is the modified routine that now works (on leaking memory) with
openmpi.
-sanjay
On 5/30/19 8
Hi, Sanjay,
Could you send your modified data exchange code (psetb.F) with MPI_Waitall?
See other inlined comments below. Thanks.
On Thu, May 30, 2019 at 1:49 PM Sanjay Govindjee via petsc-users
mailto:petsc-users@mcs.anl.gov>> wrote:
Lawrence,
Thanks for taking a look! This is what I had be
1) Correct: Placing a WaitAll before the MPI_Barrier solve the problem
in our send-get routine for OPENMPI
2) Correct: The problem persists with KSPSolve
3) Correct: WaitAll did not fix the problem in our send-get nor in
KSPSolve when using MPICH
Also correct. Commenting out the call to KSPSo
Thanks for the update. So the current conclusions are that using the Waitall
in your code
1) solves the memory issue with OpenMPI in your code
2) does not solve the memory issue with PETSc KSPSolve
3) MPICH has memory issues both for your code and PETSc KSPSolve (despite) the
wait all fi
Great observation Lawrence.
https://www.slideshare.net/jsquyres/friends-dont-let-friends-leak-mpirequests
You can add the following option to --download-mpich
--download-mpich-configure-arguments="--enable-error-messages=all --enable-g"
then MPICH will report all MPI resources that hav
Hi Sanjay,
> On 30 May 2019, at 08:58, Sanjay Govindjee via petsc-users
> wrote:
>
> The problem seems to persist but with a different signature. Graphs attached
> as before.
>
> Totals with MPICH (NB: single run)
>
> For the CG/Jacobi data_exchange_total = 41,385,984; kspsolve_tot
The problem seems to persist but with a different signature. Graphs
attached as before.
Totals with MPICH (NB: single run)
For the CG/Jacobi data_exchange_total = 41,385,984; kspsolve_total =
38,289,408
For the GMRES/BJACOBI data_exchange_total = 41,324,544; kspsolve_total =
41
Let us know how it goes with MPICH
> On May 30, 2019, at 2:01 AM, Sanjay Govindjee wrote:
>
> I put in calls to PetscMemoryGetCurrentUsage( ) around KSPSolve and my data
> exchange routine. The problem is clearly mostly in my data exchange routine.
> Attached are graphs of the change in m
This is indeed worrisome.
Would it be possible to put PetscMemoryGetCurrentUsage() around each call
to KSPSolve() and each call to your data exchange? See if at each step they
increase?
One thing to be aware of with "max resident set size" is that it measures
the number of pages
I am trying to track down a memory issue with my code; apologies in
advance for the longish message.
I am solving a FEA problem with a number of load steps involving about 3000
right hand side and tangent assemblies and solves. The program is
mainly Fortran, with a C memory allocator.
When I
28 matches
Mail list logo
