Hi Mike, Thanks for looking into it. However, thing is, not all entries all cleared in the case of the Java client, just the keys which interest itโs registered. I think that might not be correct :S
BR, Mario Obtener Outlook para iOS<https://aka.ms/o0ukef> ________________________________ De: Mike Martell <marte...@vmware.com> Enviado: Thursday, March 4, 2021 6:13:19 PM Para: dev@geode.apache.org <dev@geode.apache.org> Asunto: Re: Cached regions are not synchronized after restore It seems like this is a NC bug. I'll write a test for your scenario, but it seems like the NC is not dropping its local cache (i.e., clearing all local entries) on a server restore. As Dan has indicated, the Java client will clear a local cache when it loses connectivity to all servers, which is implied by a restore operation. I'll report back on my findings. Mike ________________________________ From: Mario Salazar de Torres <mario.salazar.de.tor...@est.tech> Sent: Thursday, March 4, 2021 1:34 AM To: dev@geode.apache.org <dev@geode.apache.org> Subject: Re: Cached regions are not synchronized after restore Hi, Well, thing is Mike you raised an interesting topic here, because I wasn't seeing the behavior of GF_CACHE_CONCURRENT_MODIFICATION_EXCEPTION as incorrect, rather the fact that entries where not cleaned up after subscription redundancy is lost. But interestingly, I've been asked that why after restarting the cluster, some entries events are allowed, and some others are not. I thought it was worth looking into it, and now that you mentioned about GF_CACHE_CONCURRENT_MODIFICATION_EXCEPTION, I had the following idea: To ease debugging of this issue I've decoded and serialized the notification messages into a JSON-like style. Event for key "key" in region "/Region" before the cluster restart: { "region": "/Region", "key": "key", "delta": false, "value": "value", "version": { "flags": "HAS_MEMBER_ID | VERSION_TWO_BYTES", "bits": "", "dsid": -1, "entry_version": 1, "region_version": 1, "timestamp": 1614444651497, "member_id": "745dac94-74cf-4d18-9f7b-aab3877f99b9", "prev_member_id": null }, "interest_list": true, "cqs": false, "event": { "member": { "id": "Native_bgddbbgfab1", "name": "default_GeodeDS", "type": "MemberType.LONER_DM_TYPE", "flags": "PARTIAL_ID_BIT", "address": "::", "port": 2, "uuid": "00000000-0000-0000-0000-000000000000", "weight": 0 }, "thread_id": 1, "sequence_id": 0, "bucket_id": -1, "breadcrumb": 0 } } Event for the same key after the cluster restart: { "region": "/Region", "key": "key", "delta": false, "value": "value", "version": { "flags": "HAS_MEMBER_ID | VERSION_TWO_BYTES", "bits": "BITS_RECORDED | BITS_IS_REMOTE_TAG", "dsid": -1, "entry_version": 1, "region_version": 1, "timestamp": 1614444953534, "member_id": "558a65d2-3299-4404-946b-a081e7591d4e", "prev_member_id": null }, "interest_list": true, "cqs": false, "event": { "member": { "id": "Native_dgfccgfbad1", "name": "default_GeodeDS", "type": "MemberType.LONER_DM_TYPE", "flags": "PARTIAL_ID_BIT", "address": "::", "port": 2, "uuid": "00000000-0000-0000-0000-000000000000", "weight": 0 }, "thread_id": 1, "sequence_id": 3, "bucket_id": -1, "breadcrumb": 0 } } GF_CACHE_CONCURRENT_MODIFICATION_EXCEPTION is supposed to be raised if 2 events for the same key are received at the same time. But in this case, they are received at two distant points in time (before the restart and after the restart). Why is this happening? As you can see both versions have the same "entry_version" (1), so in this case for both Java and native clients what determines whether the exception is thrown is the result of the memberIDs comparison. So, this explains why randomly some entries are allowed and some are not under this scenario. Because depending on which ID have both members involved with both events the comparison yields a different result. Now here are my questions: * Shouldn't be better in this case to use the timestamp to determine which one is older/newer whenever both have the same entry_version? * Wouldn't be in this scenario inaccurate to throw ConcurrentCacheModificationException if the check fails? I mean, given the fact that there is no concurrent modification. Thanks for all, Mario. ________________________________ From: Mike Martell <marte...@vmware.com> Sent: Wednesday, March 3, 2021 8:10 PM To: dev@geode.apache.org <dev@geode.apache.org> Subject: Re: Cached regions are not synchronized after restore Hi Mario, Thanks for discovering this difference in behavior between the geode-native clients (C++ and .NET) and geode Java client regarding the sync'ing of local caches after a restore operation. It looks like geode-native has no tests for the exception type you're seeing: GF_CACHE_CONCURRENT_MODIFICATION_EXCEPTION We definitely should write a test around this functionality using our new testing framework and then fix the new failing test. If you're not familiar with our new test framework, I'd be happy to walk you through it. It's much nicer! Thanks, Mike. ________________________________ From: Mario Salazar de Torres <mario.salazar.de.tor...@est.tech> Sent: Monday, March 1, 2021 6:38 PM To: dev@geode.apache.org <dev@geode.apache.org> Subject: Re: Cached regions are not synchronized after restore Hi everyone, After replicating the test with both clients being the Java implementation, I could verify that what @Dan Smith<mailto:dasm...@vmware.com> pointed out about the documentation is happening. Every time subscription redundancy is lost, cached entries are erased. This is clearly not happening in geode-native. So, I will further investigate to bring this functionality into the native client. Thanks for the help ๐ BR/ Mario ________________________________ From: Dan Smith <dasm...@vmware.com> Sent: Monday, March 1, 2021 5:27 PM To: dev@geode.apache.org <dev@geode.apache.org> Subject: Re: Cached regions are not synchronized after restore The java client at least should automatically drop its cache when it loses and restores connectivity to all the servers. See https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgeode.apache.org%2Fdocs%2Fguide%2F12%2Fdeveloping%2Fevents%2Fhow_client_server_distribution_works.html%23how_client_server_distribution_works__section_928BB60066414BEB9FAA7FB3120334A3&data=04%7C01%7Cmartellm%40vmware.com%7C1c816655778943f0046d08d8def0ce9b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C1%7C0%7C637504473135136189%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=RC6pDjAJXaSvuIm8euBXe64claJWgJ7hAXCKTx4YhNw%3D&reserved=0 -Dan ________________________________ From: Jacob Barrett <jabarr...@vmware.com> Sent: Friday, February 26, 2021 9:29 AM To: dev@geode.apache.org <dev@geode.apache.org> Subject: Re: Cached regions are not synchronized after restore For clarification, does the a Java client show the same behavior in missing the events after restore or is this something you are saying is unique to the native clients? > On Feb 26, 2021, at 4:48 AM, Mario Salazar de Torres > <mario.salazar.de.tor...@est.tech> wrote: > > Hi, > These months I have been tackling a series of issues having to do with > several inconsistencies in the geode-native client after a cluster restore. > The later one has to do with subscription notification and cached regions. > The scenario is as follows: > > 1. Start the cluster and the clients. For clarification purposes let's say > we have a NC and a Java client. Being the NC the local client, and Java > client the external one. > Also note that NC client has subscription notification enabled for its pool > and the region has caching enabled. > 2. Register interest for all the region entries. > 3. Write some entries into the region from the Java client. Notifications > are received in the NC client and entries cached. > 4. Take a backup of the disk-stores. > 5. Write/modify some entries with the Java client. Notifications are > received in the NC client and entries cached. > 6. Restore the previous backup. > 7. Write/modify some entries with the Java client. Some of the > notifications are discarded and some others not. > Note that all the entries which notifications were ignored did not exist in > the step 4 of the scenario. > > The reason why notifications mentioned in step 7 are ignored is due to the > following log: > "Region::localUpdate: updateNoThrow<Region::put> for key [<redacted>] failed > because the cache already contains an entry with higher version. The cache > listener will not be invoked" > > So, first of, I wanted to ask: > > * Any of you have encountered this issue before? How did you tackle it? > * Is there any mechanism in the Java client to avoid this kind of issues > with caching de-sync? Note that I did not found any > * Maybe we should add an option to clear local cached regions after > connection is lost towards the cluster in the same way is done with > PdxTypeRegistry? > * Maybe any other solution having to do with cluster versioning? > > BR, > Mario.