Re: Processing time series data in order

2016-12-21 Thread Ali Akhtar
The batch size can be large, so in memory ordering isn't an option, unfortunately. On Thu, Dec 22, 2016 at 7:09 AM, Jesse Hodges wrote: > Depending on the expected max out of order window, why not order them in > memory? Then you don't need to reread from Cassandra, in case of a problem > you ca

Re: Processing time series data in order

2016-12-21 Thread Jesse Hodges
Depending on the expected max out of order window, why not order them in memory? Then you don't need to reread from Cassandra, in case of a problem you can reread data from Kafka. -Jesse > On Dec 21, 2016, at 7:24 PM, Ali Akhtar wrote: > > - I'm receiving a batch of messages to a Kafka topi

Processing time series data in order

2016-12-21 Thread Ali Akhtar
- I'm receiving a batch of messages to a Kafka topic. Each message has a timestamp, however the messages can arrive / get processed out of order. I.e event 1's timestamp could've been a few seconds before event 2, and event 2 could still get processed before event 1. - I know the number of messag