Hmm that is a lot of messages to try to retain the order of. Your buffer time will need to be larger than the clock drift between publishers. Even you manage to keep your clocks within 1 second accuracy and use a 1 second buffer, that's still a million buffered packets.
Maybe you can shard processing so that no single worker needs a buffer that large. But in any case you're probably still looking at thousands of buffered messages per worker. It might be possible to optimize using a fixed size ring buffer (another pattern common in multimedia :)) but I can't say for sure. Good luck with the optimizing. On Wed, Dec 6, 2017, at 01:38 PM, Francesco wrote: > Hi Justin, > you're absolutely right, I want to do something very very similar to a > jitter buffer.> > I 've found some implementations: > https://github.com/icefreedom/jitter_buffer/tree/master/include/jitter_buffer> > https://github.com/alpartis/rtp.jitter > but honestly they are so much focused on RTP that are not > useful to me.> > It should be easy to write a generic message-oriented re-order > container using a std::map having the timestamp as key and the > message as value but honestly that does not look something high > performance (I expect to push about 1Mpps in and get the same rate in > output, at least in average).> Perhaps a possible solution is to couple > std::map with Boost memory > pool allocator > (http://www.boost.org/doc/libs/1_65_1/libs/pool/doc/html/boost_pool/pool/interfaces.html) > to decrease memory fragmentation and reduce to zero malloc/free calls> > Thanks for the reply > Francesco > > > > 2017-12-06 18:54 GMT+01:00 Justin Karneges <[email protected]>: >> __ >> Hi, >> >> It sounds like you need logic similar to a jitter buffer, which is >> commonly used in RTP media streaming (voip / TV airwaves).>> >> Basically, you have a time buffer (say 300 milliseconds) that >> timestamped messages are held in before processing. Each message sits >> in this queue for the defined time limit, and if any other messages >> arrive during that time then the messages are sorted by their >> timestamps. Once the time limit elapses for a message, it becomes >> available for reading. Note that this approach introduces a fixed >> amount of latency (e.g. in the case of a 300ms buffer, messages will >> run 300ms behind).>> >> I don't have any libraries to suggest, but you might look around for >> jitter buffer implementations as inspiration.>> >> >> Justin >> >> >> On Wed, Dec 6, 2017, at 03:55 AM, Francesco wrote: >>> Hi all, >>> As you know when using PUB/SUB model the order of arrival of >>> messages at SUB-side cannot be garantueed and generally speaking, >>> the SUB will receive messages out of order. This is a well known >>> problem, see e.g.:>>> https://cloud.google.com/pubsub/docs/ordering >>> >>> My question is: in case in the SUB you need strict time ordering, >>> are you aware of any good C++ implementation of a reordering queue?>>> >>> I used for other purposes moodycamel's queue >>> (http://moodycamel.com/blog/2013/a-fast-lock-free-queue-for-c++) >>> which is thread-safe but it does not perform time-reordering.>>> >>> Thanks! >>> >>> Francesco >>> >>> >>> >>> _________________________________________________ >>> zeromq-dev mailing list >>> [email protected] >>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >> >> >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > _________________________________________________ > zeromq-dev mailing list > [email protected] > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
