Hi Miguel,
On Tue 1. Oct 2024 at 07:56, MIGUEL MOLINOS PEREZ <mmoli...@us.es> wrote:
> Thank you Matt, it works!
>
> The implementation is straightforward:
> - 1º Define the paddle regions using DMGetLocalBoundingBox with the
> background DMDA mesh as an auxiliary mesh for the domain-partitioning.
> - 2º Create an integer to count the local number of particles to be used
> as ghost particle for other processors (N_ghost). One particle can be
> counted more than one time. At the same time, fill two arrays:
> - one with the index of the "main particle” (local particle),
> - and the other with the target rank of the "main particle”.
> - 3º Create the new particles using DMSwarmAddNPoints
> - 4º Fill the new particles with the information of the “main particle”
> but set the internal variable DMSwarmField_rank with the target rank.
> - 5º Call DMSwarmMigrate(*,PETSC_TRUE). Therefore, we send the ghost
> particles to the corresponding processors and we delete them from the
> original processor.
> - 6º Do stuff…
> - 7º Delete ghost particles. This is very easy, we just have to call
> DMSwarmRemovePoint N_ghost times.
>
> I think this can be easily implemented as closed routine for the DMSwarm
> class.
>
> The remaining question is: how to do the communication between the
> “original" particle and the ghost particles? For instance, if we update
> some particle variable (locally) inside of a SNES context, this same
> variable should be updated in the ghost particles at the other processors.
>
I think what you are asking about is an operation similar to
VecGhostUpdate{Begin,End}(). In the case of a DMSwarm I’m not sure how to
define the InsertMode = ADD_VALUES? Some swarm fields do not make sense to
be added. INSERT_VALUES is fine.
One solution might be to have something like this
DMSwarmCollectViewUpdateGhostOwners(DM dm, InsertMode mode, PetscInt
nfields, const char *fieldNames[]);
where one can specify the insert mode and the fields on which the insert
mode will apply.
Would something like that work?
Cheers,
Dave
> PS: Hope this helps someone in the future :-)
>
>
> On Sep 27, 2024, at 10:50 AM, MIGUEL MOLINOS PEREZ <mmoli...@us.es> wrote:
>
> Thank you Matt, let me give it try.
>
> Miguel
>
> On Sep 27, 2024, at 3:44 AM, Matthew Knepley <knep...@gmail.com> wrote:
>
> On Thu, Sep 26, 2024 at 7:18 PM MIGUEL MOLINOS PEREZ <mmoli...@us.es>
> wrote:
>
>> I see, you mean:
>>
>> Create the ghost particles at the local cell with the same properties as
>> particle 1 (duplicate the original particle) but different value
>> DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do the
>> migration and delete the local copies of the particle 1. Right?
>>
>
> Yep. I think it will work, from what I know about BASIC.
>
> Thanks,
>
> Matt
>
>
>> Thanks,
>> Miguel
>>
>> On Sep 26, 2024, at 11:09 PM, Matthew Knepley <knep...@gmail.com> wrote:
>>
>> On Thu, Sep 26, 2024 at 11:20 AM MIGUEL MOLINOS PEREZ <mmoli...@us.es>
>> wrote:
>>
>>> Thank you Matt.
>>>
>>> Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic
>>> implementation
>>> to see if there is some workaround.
>>>
>>> The idea of adding new particles is interesting. However, in that case,
>>> we need to initialize the new (ghost) particles using the fields of the
>>> “real” particle, right? This can be done using something like:
>>>
>>> VecGhostUpdateBegin
>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhostUpdateBegin/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEHvOdaqj$
>>> >(Vec
>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEJ7VDssN$
>>> > globalout,InsertMode
>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertMode/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEHXxW4d9$
>>> > ADD_VALUES
>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VALUES/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAOykCv4$
>>> >, ScatterMode
>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/ScatterMode/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAENQNA5xS$
>>> > SCATTER_REVERSE
>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER_REVERSE/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAN6FYSr$
>>> >);VecGhostUpdateEnd
>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhostUpdateEnd/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEOdwyTUf$
>>> >(Vec
>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEJ7VDssN$
>>> > globalout,InsertMode
>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertMode/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEHXxW4d9$
>>> > ADD_VALUES
>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VALUES/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAOykCv4$
>>> >, ScatterMode
>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/ScatterMode/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAENQNA5xS$
>>> > SCATTER_REVERSE
>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER_REVERSE/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAN6FYSr$
>>> >);
>>>
>>> for the particle fields (?).
>>>
>>
>> I think we can just copy from the local particle. For example, suppose I
>> decide that particle 1 should go to rank 5, 12, and 27. Then
>> I first set p1.rank = 5, then I add two new particles with the same
>> values as particle 1, but with rank = 12 and 27. Then when I call migrate,
>> it will move these three particles to the correct processes, and delete the
>> original particles and the copies from the local set.
>>
>> Thanks,
>>
>> Matt
>>
>>
>>> Thanks,
>>> Miguel
>>>
>>>
>>> On Sep 26, 2024, at 3:53 PM, Matthew Knepley <knep...@gmail.com> wrote:
>>>
>>> On Thu, Sep 26, 2024 at 6:31 AM MIGUEL MOLINOS PEREZ <mmoli...@us.es>
>>> wrote:
>>>
>>>> Hi Matt et al,
>>>>
>>>> I’ve been working on the scheme that you proposed to create ghost
>>>> particles (atoms in my case), and it works! With a couple of caveats:
>>>> -1º In general the overlap particles will be migrate from their own
>>>> rank to more than one neighbor rank, this is specially relevant for those
>>>> located close to the corners. Therefore, you'll need to call DMSwarmMigrate
>>>> several times (27 times for 3D cells), during the migration process.
>>>>
>>>
>>> That is terrible. Let's just fix DMSwarmMigrate to have a mode that
>>> sends the particle to all overlapping neighbors at once. It can't be that
>>> hard.
>>>
>>>
>>>> -2º You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed
>>>> algorithm will not work at all!
>>>>
>>>
>>> Oh, I should have thought of that. Sorry.
>>>
>>> I can help code up that extension. Can you take a quick look at the
>>> BASIC code? Right now, we just use the rank attached to the particle
>>> to send it. We could have an arrays of ranks, but that seems crazy, and
>>> would blow up particle storage. How about just adding new particles
>>> with the other ranks right before migration?
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>>
>>>> Hope this helps to other folks!
>>>>
>>>> I have a follow-up question about periodic bcc on this context, should
>>>> I open a new thread of keep posting here?
>>>>
>>>> Thanks,
>>>> Miguel
>>>>
>>>> On Aug 7, 2024, at 4:22 AM, MIGUEL MOLINOS PEREZ <mmoli...@us.es>
>>>> wrote:
>>>>
>>>> Thanks Matt, I think I'll start by making a small program as a proof of
>>>> concept. Then, if it works I'll implement it in my code and I'll be happy
>>>> to share it too :-)
>>>>
>>>> Miguel
>>>>
>>>> On Aug 4, 2024, at 3:30 AM, Matthew Knepley <knep...@gmail.com> wrote:
>>>>
>>>> On Fri, Aug 2, 2024 at 7:15 PM MIGUEL MOLINOS PEREZ <mmoli...@us.es>
>>>> wrote:
>>>>
>>>>> Thanks again Matt, that makes a lot more sense !!
>>>>>
>>>>> Just to check that we are on the same page. You are saying:
>>>>>
>>>>> 1. create a field define a field called "owner rank" for each
>>>>> particle.
>>>>>
>>>>> 2. Identify the phantom particles and modify the internal variable
>>>>> defined by the DMSwarmField_rank variable.
>>>>>
>>>>> 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the
>>>>> new local vector including the ghost particles.
>>>>>
>>>>> 4. Then, once the calculations are done, rename the DMSwarmField_rank
>>>>> variable using the "owner rank" variable and call
>>>>> DMSwarmMigrate(*,PETSC_FALSE) once again.
>>>>>
>>>>
>>>> I don't think we need this last step. We can just remove those ghost
>>>> particles for the next step I think.
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>>
>>>>> Thank you,
>>>>> Miguel
>>>>>
>>>>>
>>>>> On Aug 2, 2024, at 5:33 PM, Matthew Knepley <knep...@gmail.com> wrote:
>>>>>
>>>>> On Fri, Aug 2, 2024 at 11:15 AM MIGUEL MOLINOS PEREZ <mmoli...@us.es>
>>>>> wrote:
>>>>>
>>>>>> Thank you Matt for your time,
>>>>>>
>>>>>> What you describe seems to me the ideal approach.
>>>>>>
>>>>>> 1) Add a particle field 'ghost' that identifies ghost vs owned
>>>>>> particles. I think it needs options OWNED, OVERLAP, and GHOST
>>>>>>
>>>>>> This means, locally, I need to allocate Nlocal + ghost particles
>>>>>> (duplicated) for my model?
>>>>>>
>>>>>
>>>>> I would do it another way. I would allocate the particles with no
>>>>> overlap and set them up. Then I would identify the halo particles, mark
>>>>> them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as
>>>>> GHOST, then unmark the OVERLAP particles. Shoot! That marking will not
>>>>> work
>>>>> since we cannot tell the difference between particles we received and
>>>>> particles we sent. Okay, instead of the `ghost` field we need an `owner
>>>>> rank` field. So then we
>>>>>
>>>>> 1) Setup the non-overlapping particles
>>>>>
>>>>> 2) Identify the halo particles
>>>>>
>>>>> 3) Change the `rank`, but not the `owner rank`
>>>>>
>>>>> 4) Call DMSwarmMigrate()
>>>>>
>>>>> Now we can identify ghost particles by the `owner rank`
>>>>>
>>>>>
>>>>>> If that so, how to do the communication between the ghost particles
>>>>>> living in the rank i and their “real” counterpart in the rank j.
>>>>>>
>>>>>> Algo, as an alternative, what about:
>>>>>> 1) Use an IS tag which contains, for each rank, a list of the global
>>>>>> index of the neighbors particles outside of the rank.
>>>>>> 2) Use VecCreateGhost to create a new vector which contains extra
>>>>>> local space for the ghost components of the vector.
>>>>>> 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do
>>>>>> the transference of data between a vector obtained with
>>>>>> DMSwarmCreateGlobalVectorFromField
>>>>>> 4) Do necessary computations using the vectors created with
>>>>>> VecCreateGhost.
>>>>>>
>>>>>
>>>>> This is essentially what Migrate() does. I was trying to reuse the
>>>>> code.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Matt
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>> Miguel
>>>>>>
>>>>>> On Aug 2, 2024, at 8:58 AM, Matthew Knepley <knep...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> On Thu, Aug 1, 2024 at 4:40 PM MIGUEL MOLINOS PEREZ <mmoli...@us.es>
>>>>>> wrote:
>>>>>>
>>>>>>> This Message Is From an External Sender
>>>>>>> This message came from outside your organization.
>>>>>>>
>>>>>>>
>>>>>>> Dear all,
>>>>>>>
>>>>>>> I am implementing a Molecular Dynamics (MD) code using the DMSWARM
>>>>>>> interface. In the MD simulations we evaluate on each particle (atoms)
>>>>>>> some kind of scalar functional using data from the neighbouring atoms.
>>>>>>> My problem lies in the parallel implementation of the model, because
>>>>>>> sometimes, some of these neighbours lie on a different processor.
>>>>>>>
>>>>>>> This is usually solved by using ghost particles. A similar approach
>>>>>>> (with nodes instead) is already implemented for other PETSc mesh
>>>>>>> structures like DMPlexConstructGhostCells. Unfortunately, I don't see
>>>>>>> this kind of constructs for DMSWARM. Am I missing something?
>>>>>>>
>>>>>>> I this could be done by applying a buffer region by exploiting the
>>>>>>> background DMDA mesh that I already use to do domain decomposition.
>>>>>>> Then using the buffer region of each cell to locate the ghost particles
>>>>>>> and finally using VecCreateGhost. Is this feasible? Or is there an
>>>>>>> easier approach using other PETSc functions.
>>>>>>>
>>>>>>>
>>>>>> This is feasible, but it would be good to develop a set of best
>>>>>> practices, since we have been mainly focused on the case of non-redundant
>>>>>> particles. Here is how I think I would do what you want.
>>>>>>
>>>>>> 1) Add a particle field 'ghost' that identifies ghost vs owned
>>>>>> particles. I think it needs options OWNED, OVERLAP, and GHOST
>>>>>>
>>>>>> 2) At some interval identify particles that should be sent to other
>>>>>> processes as ghosts. I would call these "overlap particles". The
>>>>>> determination
>>>>>> seems application specific, so I would leave this determination
>>>>>> to the user right now. We do two things to these particles
>>>>>>
>>>>>> a) Mark chosen particles as OVERLAP
>>>>>>
>>>>>> b) Change rank to process we are sending to
>>>>>>
>>>>>> 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag
>>>>>>
>>>>>> 4) Mark OVERLAP particles as GHOST when they arrive
>>>>>>
>>>>>> There is one problem in the above algorithm. It does not allow
>>>>>> sending particles to multiple ranks. We would have to do this
>>>>>> in phases right now, or make a small adjustment to the interface
>>>>>> allowing replication of particles when a set of ranks is specified.
>>>>>>
>>>>>> THanks,
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>>
>>>>>>> Thank you,
>>>>>>> Miguel
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which
>>>>>> their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAECyeEhTc$
>>>>>>
>>>>>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAifLp3_$
>>>>>> >
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAECyeEhTc$
>>>>>
>>>>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAifLp3_$
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAECyeEhTc$
>>>>
>>>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAifLp3_$
>>>> >
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAECyeEhTc$
>>>
>>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAifLp3_$
>>> >
>>>
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAECyeEhTc$
>>
>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAifLp3_$
>> >
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAECyeEhTc$
>
> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAifLp3_$
> >
>
>
>
>
