Our Slack channel

2020-07-02 Thread Alexander Murmann
Hi community,

For some of our coordination efforts, like release status updates,
coordination of flaky tests or collective debugging of some issues a more
synchronous medium might be useful. We've had an Apache Geode Slack channel
for a long time, but never used it much. I wonder if it would be worthwhile
to revive it.

I will start an effort to post updates on the current release status and
reliability of our test suite in the channel.

If you do have an Apache email address, you can follow this link
 and sign up with
your Apache email address by following the "create an account" flow below
the form.
If you don't have a Apache email address, you can use this link:
http://s.apache.org/slack-invite

See you all there!


Question about gateway sender stopped and memory consumption

2020-07-02 Thread Alberto Gomez
Hi,

We have observed that when a gateway sender is stopped in a site, all the 
events received while it is stopped are stored in the 
'AbstractGatewaySender.tmpDroppedEvents' queue of the primary sender. The 
elements of this queue are not removed from this queue until the sender is 
started back again.

This behavior implies that if the gateway sender is stopped for a long time, 
there is a risk of heap exhaustion in the members hosting primary senders.

Under split brain situations, if lasting long enough, there could be heap 
exhaustion problems in servers due to the memory used by the gateway sender 
queues, even if overflow to disk is used -given that part of the event is 
always stored in memory.
For those situations we had thought about stopping gateway senders when the 
memory used by the gateway sender queues reached a certain memory threshold. 
But according to the above, stopping the gateway senders would only make things 
worse.

Would it make sense for the gateway sender not to store the received events in 
tmpDroppedEvents while it is stopped?

Any suggestion on how to approach the problem of heap exhaustion due to the 
growth of gateway sender queues in long lasting split brain situations?

Thanks in advance,

Alberto G.




Re: Question about gateway sender stopped and memory consumption

2020-07-02 Thread Kirk Lund
I would have expected unsent events to be stored in a queue that is backed
by a persistent region or something on disk. If that's not currently true,
then it seems like a good direction might be to make tmpDroppedEvents use a
durable queue of some sort that overflows to disk.



On Thu, Jul 2, 2020 at 10:33 AM Alberto Gomez 
wrote:

> Hi,
>
> We have observed that when a gateway sender is stopped in a site, all the
> events received while it is stopped are stored in the
> 'AbstractGatewaySender.tmpDroppedEvents' queue of the primary sender. The
> elements of this queue are not removed from this queue until the sender is
> started back again.
>
> This behavior implies that if the gateway sender is stopped for a long
> time, there is a risk of heap exhaustion in the members hosting primary
> senders.
>
> Under split brain situations, if lasting long enough, there could be heap
> exhaustion problems in servers due to the memory used by the gateway sender
> queues, even if overflow to disk is used -given that part of the event is
> always stored in memory.
> For those situations we had thought about stopping gateway senders when
> the memory used by the gateway sender queues reached a certain memory
> threshold. But according to the above, stopping the gateway senders would
> only make things worse.
>
> Would it make sense for the gateway sender not to store the received
> events in tmpDroppedEvents while it is stopped?
>
> Any suggestion on how to approach the problem of heap exhaustion due to
> the growth of gateway sender queues in long lasting split brain situations?
>
> Thanks in advance,
>
> Alberto G.
>
>
>


Re: Question about gateway sender stopped and memory consumption

2020-07-02 Thread Ju@N
I recall some discussion about this in the past, there even was an "RFC"
that never got implemented:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80452478.
Best regards.

On Thu, 2 Jul 2020 at 18:41, Kirk Lund  wrote:

> I would have expected unsent events to be stored in a queue that is backed
> by a persistent region or something on disk. If that's not currently true,
> then it seems like a good direction might be to make tmpDroppedEvents use a
> durable queue of some sort that overflows to disk.
>
>
>
> On Thu, Jul 2, 2020 at 10:33 AM Alberto Gomez 
> wrote:
>
> > Hi,
> >
> > We have observed that when a gateway sender is stopped in a site, all the
> > events received while it is stopped are stored in the
> > 'AbstractGatewaySender.tmpDroppedEvents' queue of the primary sender. The
> > elements of this queue are not removed from this queue until the sender
> is
> > started back again.
> >
> > This behavior implies that if the gateway sender is stopped for a long
> > time, there is a risk of heap exhaustion in the members hosting primary
> > senders.
> >
> > Under split brain situations, if lasting long enough, there could be heap
> > exhaustion problems in servers due to the memory used by the gateway
> sender
> > queues, even if overflow to disk is used -given that part of the event is
> > always stored in memory.
> > For those situations we had thought about stopping gateway senders when
> > the memory used by the gateway sender queues reached a certain memory
> > threshold. But according to the above, stopping the gateway senders would
> > only make things worse.
> >
> > Would it make sense for the gateway sender not to store the received
> > events in tmpDroppedEvents while it is stopped?
> >
> > Any suggestion on how to approach the problem of heap exhaustion due to
> > the growth of gateway sender queues in long lasting split brain
> situations?
> >
> > Thanks in advance,
> >
> > Alberto G.
> >
> >
> >
>


-- 
Ju@N


Re: Question about gateway sender stopped and memory consumption

2020-07-02 Thread Alberto Gomez
Thanks for your answer, Kirk.

If we persist the unsent events in a persistent region then the memory consumed 
would not be as high but still it would not solve our problem with long lasting 
split brain as the persistent region would take some memory too to store those 
events even if they were overflown.

Ideally it should be backed up in a queue that does not use any memory.

Best regards,

Alberto


From: Kirk Lund 
Sent: Thursday, July 2, 2020 7:41 PM
To: dev@geode.apache.org 
Subject: Re: Question about gateway sender stopped and memory consumption

I would have expected unsent events to be stored in a queue that is backed
by a persistent region or something on disk. If that's not currently true,
then it seems like a good direction might be to make tmpDroppedEvents use a
durable queue of some sort that overflows to disk.



On Thu, Jul 2, 2020 at 10:33 AM Alberto Gomez 
wrote:

> Hi,
>
> We have observed that when a gateway sender is stopped in a site, all the
> events received while it is stopped are stored in the
> 'AbstractGatewaySender.tmpDroppedEvents' queue of the primary sender. The
> elements of this queue are not removed from this queue until the sender is
> started back again.
>
> This behavior implies that if the gateway sender is stopped for a long
> time, there is a risk of heap exhaustion in the members hosting primary
> senders.
>
> Under split brain situations, if lasting long enough, there could be heap
> exhaustion problems in servers due to the memory used by the gateway sender
> queues, even if overflow to disk is used -given that part of the event is
> always stored in memory.
> For those situations we had thought about stopping gateway senders when
> the memory used by the gateway sender queues reached a certain memory
> threshold. But according to the above, stopping the gateway senders would
> only make things worse.
>
> Would it make sense for the gateway sender not to store the received
> events in tmpDroppedEvents while it is stopped?
>
> Any suggestion on how to approach the problem of heap exhaustion due to
> the growth of gateway sender queues in long lasting split brain situations?
>
> Thanks in advance,
>
> Alberto G.
>
>
>


Re: Question about gateway sender stopped and memory consumption

2020-07-02 Thread Alberto Gomez
Thanks Juan!

I will check it.

Alberto

From: Ju@N 
Sent: Thursday, July 2, 2020 7:46 PM
To: dev@geode.apache.org 
Subject: Re: Question about gateway sender stopped and memory consumption

I recall some discussion about this in the past, there even was an "RFC"
that never got implemented:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80452478.
Best regards.

On Thu, 2 Jul 2020 at 18:41, Kirk Lund  wrote:

> I would have expected unsent events to be stored in a queue that is backed
> by a persistent region or something on disk. If that's not currently true,
> then it seems like a good direction might be to make tmpDroppedEvents use a
> durable queue of some sort that overflows to disk.
>
>
>
> On Thu, Jul 2, 2020 at 10:33 AM Alberto Gomez 
> wrote:
>
> > Hi,
> >
> > We have observed that when a gateway sender is stopped in a site, all the
> > events received while it is stopped are stored in the
> > 'AbstractGatewaySender.tmpDroppedEvents' queue of the primary sender. The
> > elements of this queue are not removed from this queue until the sender
> is
> > started back again.
> >
> > This behavior implies that if the gateway sender is stopped for a long
> > time, there is a risk of heap exhaustion in the members hosting primary
> > senders.
> >
> > Under split brain situations, if lasting long enough, there could be heap
> > exhaustion problems in servers due to the memory used by the gateway
> sender
> > queues, even if overflow to disk is used -given that part of the event is
> > always stored in memory.
> > For those situations we had thought about stopping gateway senders when
> > the memory used by the gateway sender queues reached a certain memory
> > threshold. But according to the above, stopping the gateway senders would
> > only make things worse.
> >
> > Would it make sense for the gateway sender not to store the received
> > events in tmpDroppedEvents while it is stopped?
> >
> > Any suggestion on how to approach the problem of heap exhaustion due to
> > the growth of gateway sender queues in long lasting split brain
> situations?
> >
> > Thanks in advance,
> >
> > Alberto G.
> >
> >
> >
>


--
Ju@N


PR submission and Commit message etiquet.

2020-07-02 Thread Udo Kohlmeyer
Hey there Geode-Devs,

It has come to my attention that there have been a few commits that have 
creeped into the `develop` branch that don’t follow a standard that we have set.

I would like to make every committer and contributor aware of the following 
code of conduct we have all agreed to: 
https://cwiki.apache.org/confluence/display/GEODE/Criteria+for+Code+Submissions 
and https://cwiki.apache.org/confluence/display/GEODE/Commit+Message+Format

I know it is sometimes just easier not to do it… but it does not help the 
community in the long run.

Can we all please be vigilant and proactive in keeping to, finding and 
addressing such simple things so that they don’t become a habit and a smell we 
cannot get rid of.

—Udo