I reproduced your issue with your scripts.
They do:
create gateway-receiver
create disk-store
create gateway-sender
create region
With that order, I see the hang you mentioned. I'm not 100% sure why that is
happening but you can prevent it by reordering these elements.
As Anil said, you should start your GatewayReceiver last like:
create disk-store
create gateway-sender
create region
create gateway-receiver
With that order, cluster1 restarts fine.
btw 1 - with the order you had regardless of the hang, you'll see lots of
dropped WAN events since the region doesn't exist yet when the receiver is
started:
[info 2021/07/28 17:02:39.795 PDT server1_1 <Function Execution Processor2>
tid=0x3c] The GatewayReceiver started on port : 5411
[warn 2021/07/28 17:02:39.883 PDT server1_1 <ServerConnection on port 5411
Thread 1> tid=0x4a] Server connection from
[identity(192.168.1.7(server2_2:25891)<v2>:41005,connection=1; port=52554]:
Caught exception processing batch create request 0 for 100 events
org.apache.geode.cache.RegionDestroyedException: Region /testregion was not
found during batch create request 0
btw 2 - I use CacheCreation.create to see the order that elements should be
started. Thats the object that the old GemFire cache xml uses to start things
in the right order.
Barry
________________________________
From: Anilkumar Gingade <aging...@vmware.com>
Sent: Wednesday, July 28, 2021 3:45 PM
To: dev@geode.apache.org <dev@geode.apache.org>
Subject: Re: "create region" cmd stuck on wan setup
The recommendation with WAN setup is:
- Create/start WAN Senders first
- Create Regions
- Create/Start WAN receivers last
That way when wan receiver is started; the regions are created on all the
sites. Sorry, I have not looked at your scripts...
-Anil.
On 7/28/21, 3:31 AM, "Alberto Bustamante Reyes"
<alberto.bustamante.re...@est.tech> wrote:
Hi Geode devs,
I have been analyzing an issue that occurs in the following scenario:
1) I start two Geode clusters (cluster1 & cluster2) with one locator and
two servers each.
Both clusters host a partitioned region called "testregion", which is
replicated using a parallel gateway sender and a gateway receiver.
These are the gfsh files I have been using for creating the clusters:
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2Falb3rtobr%2Fe230623255632937fa68265f31e97f3a&data=04%7C01%7Cboglesby%40vmware.com%7C6e6bff680f5d46c6bbcc08d952195ff7%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637631091210347322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=V%2Fnsqn8wiEnEpjf9GQZ4Ta38rPk5ha79RYqlZWZIXzY%3D&reserved=0
2) I run a client connected to cluster2 performing operations on testregion.
3) cluster1 is stopped and all persistent data is deleted. And then, I
create cluster1 again.
4) At this point, the command to create "testregion" get stuck.
After checking the thread stack and the code, I found that the problem is
the following.
This thread is trapped on an infinite loop waiting for a bucket primary
election at "PartitionedRegion.waitForNoStorageOrPrimary":
"Function Execution Processor4" tid=0x55
java.lang.Thread.State: TIMED_WAITING
at java.base@11.0.11/java.lang.Object.wait(Native Method)
- waiting on org.apache.geode.internal.cache.BucketAdvisor@28be7ae0
at
app//org.apache.geode.internal.cache.BucketAdvisor.waitForPrimaryMember(BucketAdvisor.java:1433)
at
app//org.apache.geode.internal.cache.BucketAdvisor.waitForNewPrimary(BucketAdvisor.java:825)
at
app//org.apache.geode.internal.cache.BucketAdvisor.getPrimary(BucketAdvisor.java:794)
at
app//org.apache.geode.internal.cache.partitioned.RegionAdvisor.getPrimaryMemberForBucket(RegionAdvisor.java:1032)
at
app//org.apache.geode.internal.cache.PartitionedRegion.getBucketPrimary(PartitionedRegion.java:9081)
at
app//org.apache.geode.internal.cache.PartitionedRegion.waitForNoStorageOrPrimary(PartitionedRegion.java:3249)
at
app//org.apache.geode.internal.cache.PartitionedRegion.getNodeForBucketWrite(PartitionedRegion.java:3234)
at
app//org.apache.geode.internal.cache.PartitionedRegion.shadowPRWaitForBucketRecovery(PartitionedRegion.java:10110)
at
app//org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderQueue.addShadowPartitionedRegionForUserPR(ParallelGatewaySenderQueue.java:564)
at
app//org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderQueue.addShadowPartitionedRegionForUserPR(ParallelGatewaySenderQueue.java:443)
at
app//org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderEventProcessor.addShadowPartitionedRegionForUserPR(ParallelGatewaySenderEventProcessor.java:195)
at
app//org.apache.geode.internal.cache.wan.parallel.ConcurrentParallelGatewaySenderQueue.addShadowPartitionedRegionForUserPR(ConcurrentParallelGatewaySenderQueue.java:183)
at
app//org.apache.geode.internal.cache.PartitionedRegion.postCreateRegion(PartitionedRegion.java:1177)
at
app//org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3050)
at
app//org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:2910)
at
app//org.apache.geode.internal.cache.GemFireCacheImpl.createRegion(GemFireCacheImpl.java:2894)
at
app//org.apache.geode.cache.RegionFactory.create(RegionFactory.java:773)
After creating testregion, the sender queue partitioned region is created.
While that region buckets are recovered the command is trapped on an infinite
loop waiting for a primary bucket election at
PartitionedRegion.waitForNoStorageOrPrimary.
This seems to be a known issue because in
PartitionedRegion.getNodeForBucketWrite, there is the following command before
calling waitForNoStorageOrPrimary (and the command has been there since Geode's
first commit!) :
// Possible race with loss of redundancy at this point.
// This loop can possibly create a soft hang if no primary is ever
selected.
// This is preferable to returning null since it will prevent obtaining
the
// bucket lock for bucket creation.
return waitForNoStorageOrPrimary(bucketId, "write");
Any idea about why the primary bucket is not elected?
It seems the failure is related with the fact that "testregion" is
receiving updates from the receiver before the "create region" command has
finished. If the test is repeated without traffic on cluster2 or if I create
the cluster1's receiver after creating "testregion", this problem is not
happening.
Is there any recommendation on the startup order of regions, senders and
receivers for an scenario like the one described?
Thanks in advance,
Alberto B.
