On Thu, Sep 26, 2024 at 11:20 AM MIGUEL MOLINOS PEREZ <mmoli...@us.es> wrote:
> Thank you Matt. > > Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic > implementation > to see if there is some workaround. > > The idea of adding new particles is interesting. However, in that case, we > need to initialize the new (ghost) particles using the fields of the > “real” particle, right? This can be done using something like: > > VecGhostUpdateBegin > <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhostUpdateBegin/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgp0DqZlu$ > >(Vec > <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgod4XH0M$ > > globalout,InsertMode > <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertMode/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgo6Ycd6D$ > > ADD_VALUES > <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VALUES/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgqRE6BHz$ > >, ScatterMode > <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/ScatterMode/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNglNJk_18$ > > SCATTER_REVERSE > <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER_REVERSE/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgvURZcIx$ > >);VecGhostUpdateEnd > <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhostUpdateEnd/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgnpOD3Mg$ > >(Vec > <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgod4XH0M$ > > globalout,InsertMode > <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertMode/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgo6Ycd6D$ > > ADD_VALUES > <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VALUES/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgqRE6BHz$ > >, ScatterMode > <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/ScatterMode/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNglNJk_18$ > > SCATTER_REVERSE > <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER_REVERSE/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgvURZcIx$ > >); > > for the particle fields (?). > I think we can just copy from the local particle. For example, suppose I decide that particle 1 should go to rank 5, 12, and 27. Then I first set p1.rank = 5, then I add two new particles with the same values as particle 1, but with rank = 12 and 27. Then when I call migrate, it will move these three particles to the correct processes, and delete the original particles and the copies from the local set. Thanks, Matt > Thanks, > Miguel > > > On Sep 26, 2024, at 3:53 PM, Matthew Knepley <knep...@gmail.com> wrote: > > On Thu, Sep 26, 2024 at 6:31 AM MIGUEL MOLINOS PEREZ <mmoli...@us.es> > wrote: > >> Hi Matt et al, >> >> I’ve been working on the scheme that you proposed to create ghost >> particles (atoms in my case), and it works! With a couple of caveats: >> -1º In general the overlap particles will be migrate from their own rank >> to more than one neighbor rank, this is specially relevant for those >> located close to the corners. Therefore, you'll need to call DMSwarmMigrate >> several times (27 times for 3D cells), during the migration process. >> > > That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends > the particle to all overlapping neighbors at once. It can't be that hard. > > >> -2º You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed >> algorithm will not work at all! >> > > Oh, I should have thought of that. Sorry. > > I can help code up that extension. Can you take a quick look at the BASIC > code? Right now, we just use the rank attached to the particle > to send it. We could have an arrays of ranks, but that seems crazy, and > would blow up particle storage. How about just adding new particles > with the other ranks right before migration? > > Thanks, > > Matt > > >> Hope this helps to other folks! >> >> I have a follow-up question about periodic bcc on this context, should I >> open a new thread of keep posting here? >> >> Thanks, >> Miguel >> >> On Aug 7, 2024, at 4:22 AM, MIGUEL MOLINOS PEREZ <mmoli...@us.es> wrote: >> >> Thanks Matt, I think I'll start by making a small program as a proof of >> concept. Then, if it works I'll implement it in my code and I'll be happy >> to share it too :-) >> >> Miguel >> >> On Aug 4, 2024, at 3:30 AM, Matthew Knepley <knep...@gmail.com> wrote: >> >> On Fri, Aug 2, 2024 at 7:15 PM MIGUEL MOLINOS PEREZ <mmoli...@us.es> >> wrote: >> >>> Thanks again Matt, that makes a lot more sense !! >>> >>> Just to check that we are on the same page. You are saying: >>> >>> 1. create a field define a field called "owner rank" for each particle. >>> >>> 2. Identify the phantom particles and modify the internal variable >>> defined by the DMSwarmField_rank variable. >>> >>> 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new >>> local vector including the ghost particles. >>> >>> 4. Then, once the calculations are done, rename the DMSwarmField_rank >>> variable using the "owner rank" variable and call >>> DMSwarmMigrate(*,PETSC_FALSE) once again. >>> >> >> I don't think we need this last step. We can just remove those ghost >> particles for the next step I think. >> >> Thanks, >> >> Matt >> >> >>> Thank you, >>> Miguel >>> >>> >>> On Aug 2, 2024, at 5:33 PM, Matthew Knepley <knep...@gmail.com> wrote: >>> >>> On Fri, Aug 2, 2024 at 11:15 AM MIGUEL MOLINOS PEREZ <mmoli...@us.es> >>> wrote: >>> >>>> Thank you Matt for your time, >>>> >>>> What you describe seems to me the ideal approach. >>>> >>>> 1) Add a particle field 'ghost' that identifies ghost vs owned >>>> particles. I think it needs options OWNED, OVERLAP, and GHOST >>>> >>>> This means, locally, I need to allocate Nlocal + ghost particles >>>> (duplicated) for my model? >>>> >>> >>> I would do it another way. I would allocate the particles with no >>> overlap and set them up. Then I would identify the halo particles, mark >>> them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as >>> GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work >>> since we cannot tell the difference between particles we received and >>> particles we sent. Okay, instead of the `ghost` field we need an `owner >>> rank` field. So then we >>> >>> 1) Setup the non-overlapping particles >>> >>> 2) Identify the halo particles >>> >>> 3) Change the `rank`, but not the `owner rank` >>> >>> 4) Call DMSwarmMigrate() >>> >>> Now we can identify ghost particles by the `owner rank` >>> >>> >>>> If that so, how to do the communication between the ghost particles >>>> living in the rank i and their “real” counterpart in the rank j. >>>> >>>> Algo, as an alternative, what about: >>>> 1) Use an IS tag which contains, for each rank, a list of the global >>>> index of the neighbors particles outside of the rank. >>>> 2) Use VecCreateGhost to create a new vector which contains extra >>>> local space for the ghost components of the vector. >>>> 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the >>>> transference of data between a vector obtained with >>>> DMSwarmCreateGlobalVectorFromField >>>> 4) Do necessary computations using the vectors created with >>>> VecCreateGhost. >>>> >>> >>> This is essentially what Migrate() does. I was trying to reuse the code. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> Miguel >>>> >>>> On Aug 2, 2024, at 8:58 AM, Matthew Knepley <knep...@gmail.com> wrote: >>>> >>>> On Thu, Aug 1, 2024 at 4:40 PM MIGUEL MOLINOS PEREZ <mmoli...@us.es> >>>> wrote: >>>> >>>>> This Message Is From an External Sender >>>>> This message came from outside your organization. >>>>> >>>>> >>>>> Dear all, >>>>> >>>>> I am implementing a Molecular Dynamics (MD) code using the DMSWARM >>>>> interface. In the MD simulations we evaluate on each particle (atoms) >>>>> some kind of scalar functional using data from the neighbouring atoms. My >>>>> problem lies in the parallel implementation of the model, because >>>>> sometimes, some of these neighbours lie on a different processor. >>>>> >>>>> This is usually solved by using ghost particles. A similar approach >>>>> (with nodes instead) is already implemented for other PETSc mesh >>>>> structures like DMPlexConstructGhostCells. Unfortunately, I don't see >>>>> this kind of constructs for DMSWARM. Am I missing something? >>>>> >>>>> I this could be done by applying a buffer region by exploiting the >>>>> background DMDA mesh that I already use to do domain decomposition. Then >>>>> using the buffer region of each cell to locate the ghost particles and >>>>> finally using VecCreateGhost. Is this feasible? Or is there an easier >>>>> approach using other PETSc functions. >>>>> >>>>> >>>> This is feasible, but it would be good to develop a set of best >>>> practices, since we have been mainly focused on the case of non-redundant >>>> particles. Here is how I think I would do what you want. >>>> >>>> 1) Add a particle field 'ghost' that identifies ghost vs owned >>>> particles. I think it needs options OWNED, OVERLAP, and GHOST >>>> >>>> 2) At some interval identify particles that should be sent to other >>>> processes as ghosts. I would call these "overlap particles". The >>>> determination >>>> seems application specific, so I would leave this determination to >>>> the user right now. We do two things to these particles >>>> >>>> a) Mark chosen particles as OVERLAP >>>> >>>> b) Change rank to process we are sending to >>>> >>>> 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag >>>> >>>> 4) Mark OVERLAP particles as GHOST when they arrive >>>> >>>> There is one problem in the above algorithm. It does not allow sending >>>> particles to multiple ranks. We would have to do this >>>> in phases right now, or make a small adjustment to the interface >>>> allowing replication of particles when a set of ranks is specified. >>>> >>>> THanks, >>>> >>>> Matt >>>> >>>> >>>>> Thank you, >>>>> Miguel >>>>> >>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgtTo5Ip8$ >>>> >>>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgjAy34J0$ >>>> > >>>> >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgtTo5Ip8$ >>> >>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgjAy34J0$ >>> > >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgtTo5Ip8$ >> >> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgjAy34J0$ >> > >> >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgtTo5Ip8$ > > <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgjAy34J0$ > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgtTo5Ip8$ <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgjAy34J0$ >
