Yes. From our experiment that looked like a possibility. -Anil.
On Tue, Mar 26, 2019 at 9:59 AM Dan Smith <dsm...@pivotal.io> wrote: > Following up on the conflation thing - Anil and I did an experiment. > Conflation definitely *does* happen on everything in the queue, not just > the last batch. But we didn't see destroys get conflated with updates. > > So one thing that might make this use case work is to conflate the destroys > with the updates. Then the disk space would be freed up when the expiration > events are conflated in the queue. > > -Dan > > On Tue, Mar 26, 2019 at 8:19 AM Bruce Schuchardt <bschucha...@pivotal.io> > wrote: > > > I've been thinking along those lines as well Suranjan. Since conflation > > and expiry-forwarding don't solve the problem of running out of disk > > space the solution needs to involve the dispatch thread. > > > > For the session-state caching scenario that raised this whole issue I > > think what you've described will work. Looking at it with a wider lens > > I'm a little concerned about a TTL on the queue because multiple regions > > can feed into the same queue and you might not have the same TTL > > settings on all of those regions. > > > > On 3/25/19 4:53 PM, Suranjan Kumar wrote: > > > Hi, > > > I think the one approach for a user would be to 'filter' the events > > while > > > dispatching. If I remember correctly, we can attach a filter at > dispatch > > > time and filter the events based on creationTime of the GatewayEvent. > We > > > can provide a pre created filter and use it based on some so that user > > > doesn't have to write his/her own. > > > > > > Something like, > > > /** > > > All the events which spend timeToLive or more time in queue will be > > deleted > > > from the queue > > > and will not be sent to remote site. > > > Possible consequence is that two sites can be inconsistent in case > > > */ > > > public GatewaySenderFactory setEntryTimeToLive(long timeToLive); > > > > > > As queues will be read in LRU way this would be faster too. Only > drawback > > > is that there will be only one thread (not sure if we have concurrent > > > dispatcher yet) clearing the queue. > > > > > > As Udo/Dan mentioned above, user needs to be aware of the consequences. > > > > > > > > > On Tue, Mar 26, 2019 at 3:09 AM Bruce Schuchardt < > bschucha...@pivotal.io > > > > > > wrote: > > > > > >> I've walked through the code to forward expiration actions to async > > >> event listeners & don't see how to apply it to removal of queue > entries > > >> for WAN. The current implementation just queues the expiration > > >> actions. If we wanted to remove any queued events associated with the > > >> expired region entry we'd have to scan the whole queue, which would be > > >> too slow if we're overflowing the queue to disk. > > >> > > >> I've also walked through the conflation code. It applies only to the > > >> current batch being processed by the gateway sender. The data > structure > > >> used to perform conflation is just a Map that is created in the > sender's > > >> batch processing method and then thrown away. > > >> > > >> On 3/20/19 11:15 AM, Dan Smith wrote: > > >>>> 2) The developer wants to replicate _state_. This means that > implicit > > >>>> state changes (expiration or eviction w/ destroy) could allow us to > > >>>> optimize the queue size. This is very similar to conflation, just a > > >>>> different kind of optimization. > > >>>> > > >>>> For this second case, does it make sense to allow the user to > specify > > a > > >>>> different TTL than the underlying region? It seems like what the > user > > >>>> wants is to not replicate stale data and having an extra TTL > attribute > > >>>> would just be another value to mis-configure. What do you think > about > > >> just > > >>>> providing a boolean flag? > > >>>> > > >>>> > > >>> This kinda jogged my memory. AsyncEventQueues actually *do* have a > > >> boolean > > >>> flag to allow you to forward expiration events to the queue. I have > no > > >> idea > > >>> how this interacts with conflation though - > > >>> > > >> > > > https://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/asyncqueue/AsyncEventQueueFactory.html#setForwardExpirationDestroy-boolean- > > >
