Yes. From our experiment that looked like a possibility.
-Anil.
On Tue, Mar 26, 2019 at 9:59 AM Dan Smith wrote:
> Following up on the conflation thing - Anil and I did an experiment.
> Conflation definitely *does* happen on everything in the queue, not just
> the last batch. But we didn't see
Following up on the conflation thing - Anil and I did an experiment.
Conflation definitely *does* happen on everything in the queue, not just
the last batch. But we didn't see destroys get conflated with updates.
So one thing that might make this use case work is to conflate the destroys
with the
I've been thinking along those lines as well Suranjan. Since conflation
and expiry-forwarding don't solve the problem of running out of disk
space the solution needs to involve the dispatch thread.
For the session-state caching scenario that raised this whole issue I
think what you've describ
Hi,
I think the one approach for a user would be to 'filter' the events while
dispatching. If I remember correctly, we can attach a filter at dispatch
time and filter the events based on creationTime of the GatewayEvent. We
can provide a pre created filter and use it based on some so that user
doe
I've walked through the code to forward expiration actions to async
event listeners & don't see how to apply it to removal of queue entries
for WAN. The current implementation just queues the expiration
actions. If we wanted to remove any queued events associated with the
expired region entry
An important use case for this is session caching. Obviously it’s pointless to
replicate an expired session—the user has already gone away. Copying the bits
to the remote cluster is just creating unnecessary work.
> On Mar 20, 2019, at 11:22 AM, Bruce Schuchardt wrote:
>
> I don't know why t
I don't know why the users didn't use conflation but I suspect they're
generating events with new keys. Conflation only applies if you're
operating on the same keys all the time. If you're generating new keys
it doesn't help at all.
On 3/20/19 11:10 AM, Udo Kohlmeyer wrote:
If all that the c
> 2) The developer wants to replicate _state_. This means that implicit
> state changes (expiration or eviction w/ destroy) could allow us to
> optimize the queue size. This is very similar to conflation, just a
> different kind of optimization.
>
> For this second case, does it make sense to all
If all that the customer is concerned about is that the receiving side
gets the "latest" data, conflation is definitely the best bet. How do we
classify old? The only classification that I have of old (in this
context) is that there is a newer version of a data entry. This
classification is not
I think there are two modes:
1) The developer wants to replicate _events_. This means all changes need to
be sent to the remote site regardless of the state in the local cluster. Most
likely in order :-)
2) The developer wants to replicate _state_. This means that implicit state
changes (ex
Udo, in the cases I've looked at the user is okay with inconsistency
because they don't really care about the old data. They're most
interested in getting the newest data and keeping the sending site from
going down. I guess the docs for TTL should make it very clear that it
will cause inconsi
-1, I don't believe this is a feature that we should support. IF a
client is experiencing slow WAN replication and users only care about
the "latest" data, then maybe the user should use "conflation".
With a TTL model, we are messing with our consistency tenet. I'm am NOT
in support of a setti
Dan, I agree - there shouldn't be a choice of action here. We can
change it to just take an int and specify in the docs that it is in
seconds and will cause timed-out events to not be transmitted to the
other site.
I think we want to guarantee that time-out happens in queue order.
On 3/20/19
IDK Anil, we'll figure that out in the implementation. I was thinking
it would be in the dispatch threads, so if distribution is need that
will happen as it does now. I'm hopeful that this won't perturb the
code too much.
One thing that was brought up to me in person was the Dead Letter Queu
Sounds like a good feature. I'm interested to see what ordering guarantees
we want to implement - if we can expire things in the order they were added
to the queue that seems like a good way to go.
As Anil pointed out - do you really want to let the user pass in an
ExpirationAttributes object? Tha
+1. Will the expiration (destroy) be applied on local queues or the
expiration will be replicated (for both serial and parallel)?
-Anil.
On Wed, Mar 20, 2019 at 8:59 AM Bruce Schuchardt
wrote:
> We've seen situations where the receiving side of a WAN gateway is slow
> to accept data or is not
16 matches
Mail list logo
