On Sun, Jun 2, 2024 at 10:27 AM Matthew Knepley <knep...@gmail.com> wrote:
> On Sat, Jun 1, 2024 at 11:39 PM Carpenter, Mark H. (LARC-D302) via > petsc-users <petsc-users@mcs.anl.gov> wrote: > >> Mark Carpenter, NASA Langley. I am a novice PETSC user of about 10 years. >> I’ve build a DG-FEM code with petsc as one of the solver paths (I have my >> own as well). Furthermore, I use petsc for MPI communication. I’m running >> the DG-FEM >> ZjQcmQRYFpfptBannerStart >> This Message Is From an External Sender >> This message came from outside your organization. >> >> ZjQcmQRYFpfptBannerEnd >> >> Mark Carpenter, NASA Langley. >> >> >> >> I am a novice PETSC user of about 10 years. I’ve build a DG-FEM code >> with petsc as one of the solver paths (I have my own as well). >> Furthermore, I use petsc for MPI communication. >> >> >> >> I’m running the DG-FEM code on our NAS supercomputer. Everything works >> when my integer sizes are small. When I exceed the 2^32 limit of integer >> arithmetic the code fails in very strange ways. >> >> The users that originally set up the petsc infrastructure in the code are >> no longer at NASA and I’m “dead in the water”. >> > One additional point. I have looked at the error message. When you make PETSc calls, each call should be wrapped in PetscCall(). Here is a Fortran example: https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/ksp/tutorials/ex22f.F90?ref_type=heads__;!!G_uCfscf7eWS!eOkbaTOpui-YHhrX_HYLmYerXOaaGtlJn04-tdLvQzfRqa6gaCs2x-YtPn7xNTWzRRgD-wze7GkX5hkXqc8i$ This checks the return value after each call and ends early if there is an error. It would make your error output much more readable. Thanks, Matt > >> >> I think I’ve promoted all the integers that are problematic in my code >> (F95). On PETSC side: I’ve tried >> >> 1. Reinstall petsc with –with-64-bit-integers (no luck) >> >> > That option does not exist, so this will not work. > > >> >> 1. >> 2. Reinstall petsc with –with-64-bit-integers and >> –with-64-bit-indices (code will not compile with these options. >> Additional variables on F90 side require promotion and then the errors >> cascade through code when making PETSC calls. >> >> > We should fix this. I feel confident we can get the code to compile. > > >> >> 1. >> 2. It’s possible that I’ve missed offending integers, but the petsc >> error messages are so cryptic that I can’t even tell where it is failing. >> >> >> >> Further complicating matters: >> >> The problem by definition needs to be HUGE. Problem sizes requiring 1000 >> cores (10^6 elements at P5) are needed to experience the errors, which >> involves waiting in queues for ½ day at least. >> >> >> >> Attached are the >> >> 1. Install script used to install PETSC on our machine >> 2. The Makefile used on the fortran side >> 3. A data dump from an offending simulation (which is huge and I >> can’t see any useful information.) >> >> >> >> How do I attack this problem. >> >> (I’ve never gotten debugging working properly). >> > > Let's get the install for 64-bit indices to work. So we > > 1) Configure PETSc adding --with-64bit-indices to the configure line. Does > this work? If not, send configure.log > > 2) Compile PETSc. Does this work? If not, send make.log > > 3) Compile your code. Does this work? If not, send all output. > > 4) Do one of the 1/2 day runs and let us know what happens. An alternative > is to run a small number > of processes on a large memory workstation. We do this to test at the > lab. > > Thanks, > > Matt > > >> Mark >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eOkbaTOpui-YHhrX_HYLmYerXOaaGtlJn04-tdLvQzfRqa6gaCs2x-YtPn7xNTWzRRgD-wze7GkX5gB4gnrA$ > > <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eOkbaTOpui-YHhrX_HYLmYerXOaaGtlJn04-tdLvQzfRqa6gaCs2x-YtPn7xNTWzRRgD-wze7GkX5r8KJKDw$ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eOkbaTOpui-YHhrX_HYLmYerXOaaGtlJn04-tdLvQzfRqa6gaCs2x-YtPn7xNTWzRRgD-wze7GkX5gB4gnrA$ <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eOkbaTOpui-YHhrX_HYLmYerXOaaGtlJn04-tdLvQzfRqa6gaCs2x-YtPn7xNTWzRRgD-wze7GkX5r8KJKDw$ >
