We need the output when run with -log_view to see where the time is being spent and what communication between the CPU and GPU is occurring.
Barry > On Jun 14, 2024, at 2:59 PM, Sircar, Arpan <sirc...@ornl.gov> wrote: > > Hi Barry, > > Thanks for your prompt response. These are run with with the same PETSc > solvers but the one on GPUs (log_pKok) uses mataijkokkos while the other one > does not. > > Please let me know if you need any other information. > > Thanks, > Arpan > From: Barry Smith <bsm...@petsc.dev <mailto:bsm...@petsc.dev>> > Sent: Friday, June 14, 2024 1:47 PM > To: Sircar, Arpan <sirc...@ornl.gov <mailto:sirc...@ornl.gov>> > Cc: petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov> > <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>>; Gottiparthi, > Kalyan <gottipart...@ornl.gov <mailto:gottipart...@ornl.gov>> > Subject: [EXTERNAL] Re: [petsc-users] Running PETSc with a Kokkos backend on > OLCF Frontier > > > Please run both the CPU solvers and GPU solvers cases with -log_view and > send the two outputs. > > Barry > > >> On Jun 14, 2024, at 1:35 PM, Sircar, Arpan via petsc-users >> <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> wrote: >> >> This Message Is From an External Sender >> This message came from outside your organization. >> Hi, >> >> We have been working with OpenFOAM (an open-source CFD software) which can >> transfer its matrices to PETSc to use its linear solvers. This has been >> tested and is working well on OCLF's Frontier machine. Next we are trying to >> use the Kokkos backend to run it on Frontier GPUs. While the >> OpenFOAM+PETSc+Kokkos environment built correctly on Frontier using the >> modules sourced (attached file bash_petsc4foam_gpu) and configuring PETSc >> correctly (attached file config_gpu), the GPU solve seems to take more time >> than the CPU solve. >> >> The PETSc run-time options we are using are attached to this email (file >> fvSolution_petsc_pKok_Uof). Could you please take a look and let us know if >> this combination of options is fine? In this approach we are trying to solve >> the pressure equation only on the GPUs. >> >> Thanks, >> Arpan >> >> Arpan Sircar >> R&D Associate Staff >> Thermal Hydraulics Group >> Nuclear Energy and Fuel Cycle Division >> Oak Ridge National Laboratory >> >> <fvSolution_petsc_pKok_Uof><bash_petsc4foam_gpu><config_gpu> > > <log_pKok><log_pPet>
