I recall some discussion about this in the past, there even was an "RFC" that never got implemented: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80452478. Best regards.
On Thu, 2 Jul 2020 at 18:41, Kirk Lund <kl...@apache.org> wrote: > I would have expected unsent events to be stored in a queue that is backed > by a persistent region or something on disk. If that's not currently true, > then it seems like a good direction might be to make tmpDroppedEvents use a > durable queue of some sort that overflows to disk. > > > > On Thu, Jul 2, 2020 at 10:33 AM Alberto Gomez <alberto.go...@est.tech> > wrote: > > > Hi, > > > > We have observed that when a gateway sender is stopped in a site, all the > > events received while it is stopped are stored in the > > 'AbstractGatewaySender.tmpDroppedEvents' queue of the primary sender. The > > elements of this queue are not removed from this queue until the sender > is > > started back again. > > > > This behavior implies that if the gateway sender is stopped for a long > > time, there is a risk of heap exhaustion in the members hosting primary > > senders. > > > > Under split brain situations, if lasting long enough, there could be heap > > exhaustion problems in servers due to the memory used by the gateway > sender > > queues, even if overflow to disk is used -given that part of the event is > > always stored in memory. > > For those situations we had thought about stopping gateway senders when > > the memory used by the gateway sender queues reached a certain memory > > threshold. But according to the above, stopping the gateway senders would > > only make things worse. > > > > Would it make sense for the gateway sender not to store the received > > events in tmpDroppedEvents while it is stopped? > > > > Any suggestion on how to approach the problem of heap exhaustion due to > > the growth of gateway sender queues in long lasting split brain > situations? > > > > Thanks in advance, > > > > Alberto G. > > > > > > > -- Ju@N
