Re: Cached regions are not synchronized after restore

[Dan Smith]

The java client at least should automatically drop its cache when it loses and 
restores connectivity to all the servers. See 
https://geode.apache.org/docs/guide/12/developing/events/how_client_server_distribution_works.html#how_client_server_distribution_works__section_928BB60066414BEB9FAA7FB3120334A3

-Dan


From: Jacob Barrett 
Sent: Friday, February 26, 2021 9:29 AM
To: dev@geode.apache.org 
Subject: Re: Cached regions are not synchronized after restore

For clarification, does the a Java client show the same behavior in missing the 
events after restore or is this something you are saying is unique to the 
native clients?

> On Feb 26, 2021, at 4:48 AM, Mario Salazar de Torres 
>  wrote:
>
> Hi,
> These months I have been tackling a series of issues having to do with 
> several inconsistencies in the geode-native client after a cluster restore.
> The later one has to do with subscription notification and cached regions. 
> The scenario is as follows:
>
>  1.  Start the cluster and the clients. For clarification purposes let's say 
> we have a NC and a Java client. Being the NC the local client, and Java 
> client the external one.
> Also note that NC client has subscription notification enabled for its pool 
> and the region has caching enabled.
>  2.  Register interest for all the region entries.
>  3.  Write some entries into the region from the Java client. Notifications 
> are received in the NC client and entries cached.
>  4.  Take a backup of the disk-stores.
>  5.  Write/modify some entries with the Java client. Notifications are 
> received in the NC client and entries cached.
>  6.  Restore the previous backup.
>  7.  Write/modify some entries with the Java client. Some of the 
> notifications are discarded and some others not.
> Note that all the entries which notifications were ignored did not exist in 
> the step 4 of the scenario.
>
> The reason why notifications mentioned in step 7 are ignored is due to the 
> following log:
> "Region::localUpdate: updateNoThrow for key [] failed 
> because the cache already contains an entry with higher version. The cache 
> listener will not be invoked"
>
> So, first of, I wanted to ask:
>
>  *   Any of you have encountered this issue before? How did you tackle it?
>  *   Is there any mechanism in the Java client to avoid this kind of issues 
> with caching de-sync? Note that I did not found any
>  *   Maybe we should add an option to clear local cached regions after 
> connection is lost towards the cluster in the same way is done with 
> PdxTypeRegistry?
>  *   Maybe any other solution having to do with cluster versioning?
>
> BR,
> Mario.

[PROPOSAL] backport fix for GEODE-8919 to 1.14.0

[Ernie Burghardt]

I would like to include the change for GEODE-8919 to 1.14.0, it is a valuable 
and low risk change.

Thanks,
EB

Re: [PROPOSAL] backport fix for GEODE-8919 to 1.14.0

[Owen Nichols]

+1

I agree this looks low-risk and valuable for debugging.  Added to 1.14.0 
blocker board. 

On 3/1/21, 10:06 AM, "Ernie Burghardt"  wrote:

I would like to include the change for GEODE-8919 to 1.14.0, it is a 
valuable and low risk change.

Thanks,
EB

Re: [DISCUSS] Cutting a Geode 1.14.0 release branch

[Raymond Ingles]

We're actually hoping to get a few things into 1.14 to help make the 
compatibility with Redis to be useful:

Merged to develop:
GEODE-8933: Report max memory setting in Geode Redis statistics
GEODE-8894: Allow individual deltas to trigger bucket size recalculation
GEDOE-8865: Create additional dunit and integration tests for Redis HMGET
GEODE-8624: Improve INCRBYFLOAT accuracy for very large values

Not accepted yet:
GEODE-8864: finish implementation of Redis HSCAN Command
GEODE-8859: Redis data structures may not accurately reflect their size in 
Geode stats
GEODE-8965: Implement Redis “noevict” policy for Geode Redis

On 2/26/21, 1:46 PM, "Raymond Ingles"  wrote:

The Redis Compatibility effort has a few things we're hoping to backport to 
1.14. I am putting together a list of tickets now and will report them later 
today. (Many of the tickets are done, and one or two are still in flight but 
close to completion.)

On 2/26/21, 12:09 PM, "Owen Nichols"  wrote:

It's been two weeks since support/1.14 was cut, and the 1.14.0 blocker 
board[1] still lists 5 issues (1 still Unassigned).  

Given the lack of activity on GEODE-8943, GEODE-8860, and GEODE-8809, 
should we defer them to 1.14.1 if we still "aim to ship on March 12th."?

On 2/18/21, 10:00 AM, "Owen Nichols"  wrote:

It's been about a week since support/1.14 was cut, and the 1.14.0 
blocker board [1] still lists 5 issues (two of which appear to be Unassigned).  
If a fix is taking longer than expected, please consider posting a status 
update in the ticket about once a week, and ask for help on the dev list if you 
need.

On 2/12/21, 1:20 AM, "Owen Nichols"  wrote:

support/1.14 has been cut.  Any fixes on develop from today 
onward should be marked as Fixed In 1.15.0.

The 1.14.0 blocker board [1] currently lists 5 issues.  RC1 
won't be created before this list gets to zero.  To add to the blocker board, 
please propose on the dev list.

[1] 
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fsecure%2FDashboard.jspa%3FselectPageId%3D12336052&data=04%7C01%7Cringles%40vmware.com%7C4bb0c63928e2424e64ce08d8da86cab3%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C1%7C0%7C637499619749705983%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=GUnp%2BlwJx63jfJNGUqBJPoHMS70kfTiBrbRKdCUwg5Y%3D&reserved=0

On 2/10/21, 8:49 AM, "Owen Nichols"  wrote:

Sounds good.

Once cut, only fixes from the blocker board may be brought 
to support/1.14.

After Feb 12, if you wish to add a critical issue to the 
blocker board, please propose it on the dev list.  If the community agrees it 
is critical, you may tag it as "blocks-1.14.0"

Thanks,
Owen Nichols
1.14.0 Release Manager

On 2/9/21, 11:54 AM, "Alexander Murmann" 
 wrote:

Hi everyone,

We aren't seeing many issues that would prevent a 
1.14.0 release on our blocker board < 
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fsecure%2FDashboard.jspa%3FselectPageId%3D12336052&data=04%7C01%7Cringles%40vmware.com%7C4bb0c63928e2424e64ce08d8da86cab3%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C1%7C0%7C637499619749715974%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=WBc%2FT8lqEvmLW0ACjYXcYBHzo0PxQrp3W%2Fzm2IeP%2B1U%3D&reserved=0>
 and all issues have an owner. No new issues seem to be coming in either. This 
seems like a good time to finally cut a 1.14 release branch and get us on track 
to ship Apache Geode 1.14.0 in early March.

I propose we cut the branch at the end of this week and 
aim to ship 4 weeks later, on March 12th.

Re: [DISCUSS] Cutting a Geode 1.14.0 release branch

[Owen Nichols]

Since redis effort has been a big theme for 1.14, it makes sense to polish 
what's there as best as possible for its first release.  Thanks for putting 
together this list, it feels like these are finishing touches and low risk, so 
I'll add them to the 1.14.0 blocker board.  

On 3/1/21, 11:57 AM, "Raymond Ingles"  wrote:

We're actually hoping to get a few things into 1.14 to help make the 
compatibility with Redis to be useful:

Merged to develop:
GEODE-8933: Report max memory setting in Geode Redis statistics
GEODE-8894: Allow individual deltas to trigger bucket size recalculation
GEDOE-8865: Create additional dunit and integration tests for Redis HMGET
GEODE-8624: Improve INCRBYFLOAT accuracy for very large values

Not accepted yet:
GEODE-8864: finish implementation of Redis HSCAN Command
GEODE-8859: Redis data structures may not accurately reflect their size in 
Geode stats
GEODE-8965: Implement Redis “noevict” policy for Geode Redis

On 2/26/21, 1:46 PM, "Raymond Ingles"  wrote:

The Redis Compatibility effort has a few things we're hoping to 
backport to 1.14. I am putting together a list of tickets now and will report 
them later today. (Many of the tickets are done, and one or two are still in 
flight but close to completion.)

On 2/26/21, 12:09 PM, "Owen Nichols"  wrote:

It's been two weeks since support/1.14 was cut, and the 1.14.0 
blocker board[1] still lists 5 issues (1 still Unassigned).  

Given the lack of activity on GEODE-8943, GEODE-8860, and 
GEODE-8809, should we defer them to 1.14.1 if we still "aim to ship on March 
12th."?

On 2/18/21, 10:00 AM, "Owen Nichols"  wrote:

It's been about a week since support/1.14 was cut, and the 
1.14.0 blocker board [1] still lists 5 issues (two of which appear to be 
Unassigned).  If a fix is taking longer than expected, please consider posting 
a status update in the ticket about once a week, and ask for help on the dev 
list if you need.

On 2/12/21, 1:20 AM, "Owen Nichols"  wrote:

support/1.14 has been cut.  Any fixes on develop from today 
onward should be marked as Fixed In 1.15.0.

The 1.14.0 blocker board [1] currently lists 5 issues.  RC1 
won't be created before this list gets to zero.  To add to the blocker board, 
please propose on the dev list.

[1] 
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fsecure%2FDashboard.jspa%3FselectPageId%3D12336052&data=04%7C01%7Conichols%40vmware.com%7Ca965eaaacbc44e3d475908d8dcec3628%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C1%7C0%7C637502254373782308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=488wtPLpuDl5YWN8l0v1qv6K3BdC9JGapcgfYBtfuNY%3D&reserved=0

On 2/10/21, 8:49 AM, "Owen Nichols"  
wrote:

Sounds good.

Once cut, only fixes from the blocker board may be 
brought to support/1.14.

After Feb 12, if you wish to add a critical issue to 
the blocker board, please propose it on the dev list.  If the community agrees 
it is critical, you may tag it as "blocks-1.14.0"

Thanks,
Owen Nichols
1.14.0 Release Manager

On 2/9/21, 11:54 AM, "Alexander Murmann" 
 wrote:

Hi everyone,

We aren't seeing many issues that would prevent a 
1.14.0 release on our blocker board < 
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fsecure%2FDashboard.jspa%3FselectPageId%3D12336052&data=04%7C01%7Conichols%40vmware.com%7Ca965eaaacbc44e3d475908d8dcec3628%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C1%7C0%7C637502254373782308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=488wtPLpuDl5YWN8l0v1qv6K3BdC9JGapcgfYBtfuNY%3D&reserved=0>
 and all issues have an owner. No new issues seem to be coming in either. This 
seems like a good time to finally cut a 1.14 release branch and get us on track 
to ship Apache Geode 1.14.0 in early March.

I propose we cut the branch at the end of this week 
and aim to ship 4 weeks later, on March 12th.

[PROPOSAL] backport fix for GEODE-8920 to 1.14.0

[Kamilla Aslami]

Hi all,

I would like to back-port the fix for GEODE-8920 to 1.14.0. It modified debug 
logging in DirectChannel to make it easier to trace a message. This change is 
low-risk and important for debugging.

Thanks,
Kamilla

Re: [PROPOSAL] backport fix for GEODE-8920 to 1.14.0

[Owen Nichols]

Sounds reasonable for same reasons as GEODE-8919.  I've added it to 1.14.0 
blocker board.

On 3/1/21, 1:20 PM, "Kamilla Aslami"  wrote:

Hi all,

I would like to back-port the fix for GEODE-8920 to 1.14.0. It modified 
debug logging in DirectChannel to make it easier to trace a message. This 
change is low-risk and important for debugging.

Thanks,
Kamilla

[Proposal] Backport GEODE-8958 into 1.14.x, 1.13.x, 1.12.x branches

[Mark Hanson]

I would like to backport GEODE-8958 into previous release branches to alleviate 
a problem with tombstones if timestamps become corrupt for some reason.

Thanks,
Mark

Re: Cached regions are not synchronized after restore

[Mario Salazar de Torres]

Hi everyone,

After replicating the test with both clients being the Java implementation, I 
could verify that what @Dan Smith pointed out about 
the documentation is happening. Every time subscription redundancy is lost, 
cached entries are erased. This is clearly not happening in geode-native.
So, I will further investigate to bring this functionality into the native 
client.

Thanks for the help 🙂
BR/
Mario

From: Dan Smith 
Sent: Monday, March 1, 2021 5:27 PM
To: dev@geode.apache.org 
Subject: Re: Cached regions are not synchronized after restore

The java client at least should automatically drop its cache when it loses and 
restores connectivity to all the servers. See 
https://geode.apache.org/docs/guide/12/developing/events/how_client_server_distribution_works.html#how_client_server_distribution_works__section_928BB60066414BEB9FAA7FB3120334A3

-Dan


From: Jacob Barrett 
Sent: Friday, February 26, 2021 9:29 AM
To: dev@geode.apache.org 
Subject: Re: Cached regions are not synchronized after restore

For clarification, does the a Java client show the same behavior in missing the 
events after restore or is this something you are saying is unique to the 
native clients?

> On Feb 26, 2021, at 4:48 AM, Mario Salazar de Torres 
>  wrote:
>
> Hi,
> These months I have been tackling a series of issues having to do with 
> several inconsistencies in the geode-native client after a cluster restore.
> The later one has to do with subscription notification and cached regions. 
> The scenario is as follows:
>
>  1.  Start the cluster and the clients. For clarification purposes let's say 
> we have a NC and a Java client. Being the NC the local client, and Java 
> client the external one.
> Also note that NC client has subscription notification enabled for its pool 
> and the region has caching enabled.
>  2.  Register interest for all the region entries.
>  3.  Write some entries into the region from the Java client. Notifications 
> are received in the NC client and entries cached.
>  4.  Take a backup of the disk-stores.
>  5.  Write/modify some entries with the Java client. Notifications are 
> received in the NC client and entries cached.
>  6.  Restore the previous backup.
>  7.  Write/modify some entries with the Java client. Some of the 
> notifications are discarded and some others not.
> Note that all the entries which notifications were ignored did not exist in 
> the step 4 of the scenario.
>
> The reason why notifications mentioned in step 7 are ignored is due to the 
> following log:
> "Region::localUpdate: updateNoThrow for key [] failed 
> because the cache already contains an entry with higher version. The cache 
> listener will not be invoked"
>
> So, first of, I wanted to ask:
>
>  *   Any of you have encountered this issue before? How did you tackle it?
>  *   Is there any mechanism in the Java client to avoid this kind of issues 
> with caching de-sync? Note that I did not found any
>  *   Maybe we should add an option to clear local cached regions after 
> connection is lost towards the cluster in the same way is done with 
> PdxTypeRegistry?
>  *   Maybe any other solution having to do with cluster versioning?
>
> BR,
> Mario.

Re: [Proposal] Backport GEODE-8958 into 1.14.x, 1.13.x, 1.12.x branches

[Owen Nichols]

That sounds a lot better than never expiring them if that does happen, I think 
this would be good to include.

On 3/1/21, 2:41 PM, "Mark Hanson"  wrote:

I would like to backport GEODE-8958 into previous release branches to 
alleviate a problem with tombstones if timestamps become corrupt for some 
reason.

Thanks,
Mark