Re: [petsc-users] Code

Sorry for the long delay in getting back to you. I think there is not directly a problem with the code; rather, with these Q0 elements, Galerkin is not a good approach. I went back to src/ksp/ksp/tutorials/ex32.c and ran a little deeper and noticed that when running without Galerkin it gives

Re: [petsc-users] code with TS throws error at the end

Can you try running with valgrind? https://petsc.org/release/faq/?highlight=valgrind#what-does-corrupt-argument-or-caught-signal-or-segv-or-segmentation-violation-or-bus-error-mean-can-i-use-valgrind-or-cuda-memcheck-to-debug-memory-corruption-issues

Re: [petsc-users] Code hangs when calling PetscIntView (MPI, fortran)

. From: Barry Smith Sent: Thursday, May 20, 2021 5:00 PM To: dazza simplythebest Cc: PETSc users list Subject: Re: [petsc-users] Code hangs when calling PetscIntView (MPI, fortran) You can also have the processes with no values print an array of length zero

Re: [petsc-users] Code hangs when calling PetscIntView (MPI, fortran)

Dan From: Matthew Knepley Sent: Thursday, May 20, 2021 10:31 AM To: dazza simplythebest Cc: Jose E. Roman ; PETSc users list Subject: Re: [petsc-users] Code hangs when calling PetscIntView (MPI, fortran) On Thu, May 20, 2021 at 5:32 AM dazza simplythebest mailto:sayos...@hotmail.c

Re: [petsc-users] Code hangs when calling PetscIntView (MPI, fortran)

9:06 AM > To: dazza simplythebest mailto:sayos...@hotmail.com>> > Cc: PETSc users list <mailto:petsc-users@mcs.anl.gov>> > Subject: Re: [petsc-users] Code hangs when calling PetscIntView (MPI, fortran) > > If you look at the manpage > https://www.mcs.anl.

Re: [petsc-users] Code hangs when calling PetscIntView (MPI, fortran)

ay 20, 2021 9:06 AM > *To:* dazza simplythebest > *Cc:* PETSc users list > *Subject:* Re: [petsc-users] Code hangs when calling PetscIntView (MPI, > fortran) > > If you look at the manpage > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscIntView.html > you will s

Re: [petsc-users] Code hangs when calling PetscIntView (MPI, fortran)

: dazza simplythebest Cc: PETSc users list Subject: Re: [petsc-users] Code hangs when calling PetscIntView (MPI, fortran) If you look at the manpage https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscIntView.html you will see that PetscIntView() is collective. This means that all

Re: [petsc-users] Code hangs when calling PetscIntView (MPI, fortran)

If you look at the manpage https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscIntView.html you will see that PetscIntView() is collective. This means that all MPI processes must call this function, so it is forbidden to call it within an IF rank==... Jose > El 20 may 2021,

[petsc-users] Code hangs when calling PetscIntView (MPI, fortran)

Dear All, As part of preparing a code to call the SLEPC eigenvalue solving library, I am constructing a matrix in sparse CSR format row-by-row. Just for debugging purposes I write out the column values for a given row, which are stored in a PetscInt allocatable vector, using PetscIntV

Re: [petsc-users] Code speedup after upgrading

It's an option that he would have set explicitly via MatSetOption, following Lawrence's suggestion. He can either not call that function or use PETSC_FALSE to unset it. Junchao Zhang writes: > Is there an option to turn off MAT_SUBSET_OFF_PROC_ENTRIES for Mohammad to > try? > > --Junchao Zhang

Re: [petsc-users] Code speedup after upgrading

Is there an option to turn off MAT_SUBSET_OFF_PROC_ENTRIES for Mohammad to try? --Junchao Zhang On Sun, Mar 28, 2021 at 3:34 PM Jed Brown wrote: > I take it this was using MAT_SUBSET_OFF_PROC_ENTRIES. I implemented that > to help performance of PHASTA and other applications that assemble matri

Re: [petsc-users] Code speedup after upgrading

I take it this was using MAT_SUBSET_OFF_PROC_ENTRIES. I implemented that to help performance of PHASTA and other applications that assemble matrices that are relatively cheap to solve (so assembly cost is significant compared to preconditioner setup and KSPSolve) and I'm glad it helps so much he

Re: [petsc-users] Code speedup after upgrading

Here is the plot of run time in old and new petsc using 1,2,4,8, and 16 CPUs (in logarithmic scale): [image: Screenshot from 2021-03-28 10-48-56.png] On Thu, Mar 25, 2021 at 12:51 PM Mohammad Gohardoust wrote: > That's right, these loops also take roughly half time as well. If I am not > mis

Re: [petsc-users] Code speedup after upgrading

Thanks Lawrence for your suggestion. It did work and the BuildTwoSided time is almost zero now: BuildTwoSided 4 1.0 1.6279e-0310.1 0.00e+00 0.0 2.0e+02 4.0e+00 4.0e+00 BuildTwoSidedF 3 1.0 1.5969e-0310.5 0.00e+00 0.0 2.0e+02 3.6e+03 3.0e+00 Mohammad On Thu, Mar 25, 2021 at 4:54 AM Lawrence Mitch

Re: [petsc-users] Code speedup after upgrading

That's right, these loops also take roughly half time as well. If I am not mistaken, petsc (MatSetValue) is called after doing some calculations over each tetrahedral element. Thanks for your suggestion. I will try that and will post the results. Mohammad On Wed, Mar 24, 2021 at 3:23 PM Junchao Z

Re: [petsc-users] Code speedup after upgrading

> On 24 Mar 2021, at 01:30, Matthew Knepley wrote: > > This is true, but all the PETSc operations are speeding up by a factor 2x. It > is hard to believe these were run on the same machine. > For example, VecScale speeds up!?! So it is not network, or optimizations. I > cannot explain this.

Re: [petsc-users] Code speedup after upgrading

On Wed, Mar 24, 2021 at 2:17 AM Mohammad Gohardoust wrote: > So the code itself is a finite-element scheme and in stage 1 and 3 there > are expensive loops over entire mesh elements which consume a lot of time. > So these expensive loops must also take half time with newer petsc? And these loops

Re: [petsc-users] Code speedup after upgrading

I built petsc 3.13.4 and got results similar to the old ones. I am attaching the log-view output file here. Mohammad On Tue, Mar 23, 2021 at 6:49 PM Satish Balay via petsc-users < petsc-users@mcs.anl.gov> wrote: > On Tue, 23 Mar 2021, Matthew Knepley wrote: > > > On Tue, Mar 23, 2021 at 9:08 PM

Re: [petsc-users] Code speedup after upgrading

So the code itself is a finite-element scheme and in stage 1 and 3 there are expensive loops over entire mesh elements which consume a lot of time. Mohammad On Tue, Mar 23, 2021 at 6:08 PM Junchao Zhang wrote: > In the new log, I saw > > Summary of Stages: - Time -- - Flop --

Re: [petsc-users] Code speedup after upgrading

On Tue, 23 Mar 2021, Matthew Knepley wrote: > On Tue, Mar 23, 2021 at 9:08 PM Junchao Zhang > wrote: > > > In the new log, I saw > > > > Summary of Stages: - Time -- - Flop -- --- Messages --- > > -- Message Lengths -- -- Reductions -- > > Avg %

Re: [petsc-users] Code speedup after upgrading

On Tue, Mar 23, 2021 at 9:08 PM Junchao Zhang wrote: > In the new log, I saw > > Summary of Stages: - Time -- - Flop -- --- Messages --- > -- Message Lengths -- -- Reductions -- > Avg %Total Avg %TotalCount %Total > Avg

Re: [petsc-users] Code speedup after upgrading

In the new log, I saw Summary of Stages: - Time -- - Flop -- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %TotalCount %Total Avg %TotalCount %Total 0: Main Stage: 5.4095e+00 2.3% 4.37

Re: [petsc-users] Code speedup after upgrading

Thanks Dave for your reply. For sure PETSc is awesome :D Yes, in both cases petsc was configured with --with-debugging=0 and fortunately I do have the old and new -log-veiw outputs which I attached. Best, Mohammad On Tue, Mar 23, 2021 at 1:37 AM Dave May wrote: > Nice to hear! > The answer is

Re: [petsc-users] Code speedup after upgrading

Nice to hear! The answer is simple, PETSc is awesome :) Jokes aside, assuming both petsc builds were configured with —with-debugging=0, I don’t think there is a definitive answer to your question with the information you provided. It could be as simple as one specific implementation you use was i

[petsc-users] Code speedup after upgrading

Hi, I am using a code which is based on petsc (and also parmetis). Recently I made the following changes and now the code is running about two times faster than before: - Upgraded Ubuntu 18.04 to 20.04 - Upgraded petsc 3.13.4 to 3.14.5 - This time I installed parmetis and metis directly

Re: [petsc-users] Code (possibly) not running on GPU with CUDA

I do not have DMSetFromOptions in the code, so this may well be the issue --- Dr Gordon P S Gibb EPCC, The University of Edinburgh Tel: +44 131 651 3459 On 5 Aug 2020, at 18:48, Barry Smith mailto:bsm...@petsc.dev>> wrote: Gordon, Do you have

Re: [petsc-users] Code (possibly) not running on GPU with CUDA

Gordon, Do you have a call to DMSetFromOptions()? Barry > On Aug 5, 2020, at 10:24 AM, GIBB Gordon wrote: > > Hi, > > I’ve built PETSc with NVIDIA support for our GPU machine > (https://cirrus.readthedocs.io/en/master/user-guide/gpu.html >

Re: [petsc-users] Code (possibly) not running on GPU with CUDA

On Wed, Aug 5, 2020 at 1:09 PM GIBB Gordon wrote: > Hi, > > I used nvidia-smi before, essentially a kind of ’top’ for nvidia-gpus. > > The log output I get is: > You can see that all flops are done on the GPU by looking at the last column: EventCount Time (sec) Flop

Re: [petsc-users] Code (possibly) not running on GPU with CUDA

> Configure options: CC=nvcc FC=mpif90 CXX=mpicxx > --prefix=/lustre/home/z04/gpsgibb/TPLS/petsc --with-cudac=nvcc --with-cuda=1 > --with-mpi-dir= --with-batch This is weird. suggest using: CC=mpicc FC=mpif90 CXX=mpicxx --prefix=/lustre/home/z04/gpsgibb/TPLS/petsc --with-cudac=nvcc --with-cuda

Re: [petsc-users] Code (possibly) not running on GPU with CUDA

On Wed, Aug 5, 2020 at 12:47 PM GIBB Gordon wrote: > Hi Matt, > > It runs, however it doesn’t produce any output, and I have no way of > checking to see if it actually ran on the GPU. It was run with: > > srun -n 1 ./ex28 -vec_type cuda -use_gpu_aware_mpi 0 > 1) How did you check last time? 2)

Re: [petsc-users] Code (possibly) not running on GPU with CUDA

Hi Matt, It runs, however it doesn’t produce any output, and I have no way of checking to see if it actually ran on the GPU. It was run with: srun -n 1 ./ex28 -vec_type cuda -use_gpu_aware_mpi 0 Cheers, Gordon --- Dr Gordon P S Gibb EPCC, The Univer

Re: [petsc-users] Code (possibly) not running on GPU with CUDA

On Wed, Aug 5, 2020 at 11:24 AM GIBB Gordon wrote: > Hi, > > I’ve built PETSc with NVIDIA support for our GPU machine ( > https://cirrus.readthedocs.io/en/master/user-guide/gpu.html), and then > compiled our executable against this PETSc (using version 3.13.3). I should > add that the MPI on our

[petsc-users] Code (possibly) not running on GPU with CUDA

Hi, I’ve built PETSc with NVIDIA support for our GPU machine (https://cirrus.readthedocs.io/en/master/user-guide/gpu.html), and then compiled our executable against this PETSc (using version 3.13.3). I should add that the MPI on our system is not GPU-aware so I have to use -use_gpu_aware_mpi 0

Re: [petsc-users] Code break when PETSc not debugging

Thanks, i was able to find the bug and correct it, it is working now, I was calling a wrong DM for some DA, Regards, On Tue, Aug 14, 2018 at 2:13 PM, Jed Brown wrote: > Manuel Valera writes: > > > Thanks Jed, > > > > I got the attached, it looks is coming from one of my routines > > CorrectU4

Re: [petsc-users] Code break when PETSc not debugging

Manuel Valera writes: > Thanks Jed, > > I got the attached, it looks is coming from one of my routines > CorrectU4Pressure.F90, what other information can i get from this log? If you compile with debug info (COPTFLAGS=-g FOPTFLAGS=-g) you'll be able to see the source line that's causing this err

Re: [petsc-users] Code break when PETSc not debugging

Thanks Jed, I got the attached, it looks is coming from one of my routines CorrectU4Pressure.F90, what other information can i get from this log? Thanks, On Tue, Aug 14, 2018 at 12:57 PM, Jed Brown wrote: > Manuel Valera writes: > > > Hello everyone, > > > > I am working on running part of my

Re: [petsc-users] Code break when PETSc not debugging

Manuel Valera writes: > Hello everyone, > > I am working on running part of my code in a GPU, recently i was able to > run the whole model using one P100 GPU and one processor with good timing > but using --with-debugging=1 as configure argument, > > With this in mind i compiled PETSc in a separ

[petsc-users] Code break when PETSc not debugging

Hello everyone, I am working on running part of my code in a GPU, recently i was able to run the whole model using one P100 GPU and one processor with good timing but using --with-debugging=1 as configure argument, With this in mind i compiled PETSc in a separate folder with the same exact flags

Re: [petsc-users] Code performance for solving multiple RHS

> On Aug 24, 2016, at 8:07 PM, Harshad Ranadive > wrote: > > Thanks a lot Barry for your help. I have resolved the issue of slow linear > solver for excessively large RHS vectors to some degree. > > To summarize: > > 1) In my case, I wanted to solve the same linear system a large number of

Re: [petsc-users] Code performance for solving multiple RHS

Thanks a lot Barry for your help. I have resolved the issue of slow linear solver for excessively large RHS vectors to some degree. *To summarize:* 1) In my case, I wanted to solve the *same linear system a large number of times for different RHS vectors*. Previously I was using the iterative sol

Re: [petsc-users] Code performance for solving multiple RHS

> On Aug 11, 2016, at 10:14 PM, Harshad Ranadive > wrote: > > Hi Barry, > > Thanks for this recommendation. > > As you mention, the matrix factorization should be on a single processor. > If the factored matrix A is available on all processors can I then use > MatMatSolve(A,B,X) in paralle

Re: [petsc-users] Code performance for solving multiple RHS

Hi Barry, Thanks for this recommendation. As you mention, the matrix factorization should be on a single processor. If the factored matrix A is available on all processors can I then use MatMatSolve(A,B,X) in parallel? That is could the RHS block matrix 'B' and solution matrix 'X' be distributed

Re: [petsc-users] Code performance for solving multiple RHS

If it is sequential, which it probably should be, then you can you MatLUFactorSymbolic(), MatLUFactorNumeric() and MatMatSolve() where you put a bunch of your right hand side vectors into a dense array; not all million of them but maybe 10 to 100 at a time. Barry > On Aug 10, 2016, at

Re: [petsc-users] Code performance for solving multiple RHS

Effectively utilizing multiple right hand sides with the same system can result in roughly 2 or at absolute most 3 times improvement in solve time. A great improvement but when you have a million right hand sides not a giant improvement. The first step is to get the best (most efficient

Re: [petsc-users] Code performance for solving multiple RHS

On Wed, Aug 10, 2016 at 9:54 PM, Harshad Ranadive wrote: > Hi All, > > I have currently added the PETSc library with our CFD solver. > > In this I need to use KSPSolve(...) multiple time for the same matrix A. I > have read that PETSc does not support passing multiple RHS vectors in the > form of

[petsc-users] Code performance for solving multiple RHS

Hi All, I have currently added the PETSc library with our CFD solver. In this I need to use KSPSolve(...) multiple time for the same matrix A. I have read that PETSc does not support passing multiple RHS vectors in the form of a matrix and the only solution to this is calling KSPSolve multiple ti

Re: [petsc-users] Code sometimes work, sometimes hang when increase cpu usage

Hi, I've been debugging and removing some errors. Now the code works on most cluster nodes but fails on 2 of them. The strange thing is that I'm using the same gnu compiler but only deploying to the newer setup nodes. The newer nodes work when using my old code, which is similar except that

Re: [petsc-users] Code sometimes work, sometimes hang when increase cpu usage

It appears that you have an uninitialized variable (or more than one). When compiled with debugging, variables are normally initialized to zero. Thanks, Matt On Fri, Dec 25, 2015 at 5:41 AM, TAY wee-beng wrote: > Hi, > > Sorry, there seems to be some problems with my valgrind. I have re

Re: [petsc-users] Code sometimes work, sometimes hang when increase cpu usage

Hi, I just realised that the nodes which I tested on may have some problems, as it has just been setup. So there's problem in the MPI communication. I now do my test on the old nodes. I have reduced my problem to a serial one. The code works fine with the debug version. But for the optimized

Re: [petsc-users] Code sometimes work, sometimes hang when increase cpu usage

> On Dec 24, 2015, at 10:37 PM, TAY wee-beng wrote: > > Hi, > > I tried valgrind in MPI but it aborts very early, with the error msg > regarding PETSc initialize. It shouldn't "abort" it should print some error message and continue. Please send all the output when running with valgrind.

Re: [petsc-users] Code sometimes work, sometimes hang when increase cpu usage

Hi, I tried valgrind in MPI but it aborts very early, with the error msg regarding PETSc initialize. I retry again, using a lower resolution. GAMG works, but BoomerAMG and hypre doesn't. Increasing cpu too high (80) also cause it to hang. 60 works fine. My grid size is 98x169x169 But whe

Re: [petsc-users] Code sometimes work, sometimes hang when increase cpu usage

It sounds like you have memory corruption in a different part of the code. Run in valgrind. Matt On Thu, Dec 24, 2015 at 10:14 AM, TAY wee-beng wrote: > Hi, > > I have this strange error. I converted my CFD code from a z directon only > partition to the yz direction partition. The code works

[petsc-users] Code sometimes work, sometimes hang when increase cpu usage

Hi, I have this strange error. I converted my CFD code from a z directon only partition to the yz direction partition. The code works fine but when I increase the cpu no, strange things happen when solving the Poisson eqn. I increase cpu no from 24 to 40. Sometimes it works, sometimes it do

[petsc-users] code hangs inside SUPERLU DIST

Sorry for the last incomplete email. Please forget about it. Let me finish it up here. Hello Everyone, I am using KSPPRECONLY type ksp with SUPERLU DIST. However, I found my code hangs inside superlu_dist solver and never finishes. The problem I am solving is a time-dependent nonlinear pde and I

[petsc-users] code hangs inside SUPERLU DIST

Hello Everyone, I am using KSPPRECONLY type ksp with SUPERLU DIST. However, I found my code hangs inside superlu_dist solver and never finishes. The problem I am solving is a time-dependent nonlinear pde and I use fully Newton-Raphson for each time step. So my code will look like, For Time STEP

[petsc-users] code hangs inside SUPERLU DIST

On Sun, Apr 1, 2012 at 2:18 PM, Wen Jiang wrote: > Sorry for the last incomplete email. Please forget about it. Let me finish > it up here. > > > Hello Everyone, > > I am using KSPPRECONLY type ksp with SUPERLU DIST. However, I found my > code hangs inside superlu_dist solver and never finishes.

[petsc-users] code changes from CPU to GPU

On Wed, Feb 8, 2012 at 8:56 PM, recrusader wrote: > Dear Barry, > > I just found your discussion about this problem. > Will you add relevant functions for it recently? > No Matt > Thanks a lot. > > Best, > Yujie > > On Wed, Feb 8, 2012 at 2:58 PM, Barry Smith wrote: > >> >> The "ghosted

[petsc-users] code changes from CPU to GPU

Dear Barry, I just found your discussion about this problem. Will you add relevant functions for it recently? Thanks a lot. Best, Yujie On Wed, Feb 8, 2012 at 2:58 PM, Barry Smith wrote: > > The "ghosted" forms of vectors is not supported currently with GPUs. > > Barry > > On Feb 8, 2012,

[petsc-users] code changes from CPU to GPU

Thanks, Barry. On Wed, Feb 8, 2012 at 2:58 PM, Barry Smith wrote: > > The "ghosted" forms of vectors is not supported currently with GPUs. > > Barry > > On Feb 8, 2012, at 2:53 PM, recrusader wrote: > > > Dear PETSc developers, > > > > I have FEM codes. It works very well in CPU computation.

[petsc-users] code changes from CPU to GPU

The "ghosted" forms of vectors is not supported currently with GPUs. Barry On Feb 8, 2012, at 2:53 PM, recrusader wrote: > Dear PETSc developers, > > I have FEM codes. It works very well in CPU computation. > Now, I add '-vec_type seqcusp -mat_type aijcusp' when running the > codes, I me

[petsc-users] code changes from CPU to GPU

Dear PETSc developers, I have FEM codes. It works very well in CPU computation. Now, I add '-vec_type seqcusp -mat_type aijcusp' when running the codes, I met the following errors. My question that whether there is an example to demonstrate how to revise the codes for the conversion from CPU to GP