Hi Ziang, When faced with similar patterns/needs in my own work I consider what might be done at different points in the data flow from upstream to downstream and what implications that might have. Here are my thought processes applied to the case you describe:
Would all subscribers want the exact same collection of individual messages? If so, then for simplicity, I might develop a publisher that did the collecting and then emitted larger messages less frequently. If this is a high rate stream and the individual messages prior to collecting were small then this strategy can also improve overall throughput (see the zeromq wiki for some throughput performance results as a function of message). Or, do the many subscribers need different collections? If so, the collecting needs to be downstream of the PUB sockets. There are two general locations I would consider to add collecting. The first is to place the collecting functionality into a zeromq "proxy" or "device" (as it's sometimes called). This may be a stand-alone program or an actor function running in a thread of a larger application. It has an input SUB and an output PUB and in between it performs the collection. This allows the publisher and the subscriber codes to stay simple. It allows for a variety of collection mechanism across the overall network. In the case of intensive data flows, it spreads the processing over more cores. If message size bandwidth limitations are an issue, these actors can run in the same application as the original PUBs and use inproc:// transport. Finally, to provide the benefits above comes with the complexity of more "moving parts" that may require configuration or deployment mechanisms. So, finally, I'd consider hard-wiring collection right in the application that holds the final SUB socket. What is best depends strongly on many details not yet described. In my own case where I had all the detailed requirements I could handle I found judging which strategy is best is not at all obvious, a'priori. Trial and testing was needed. Happy flows, -Brett. Ziang Gao via zeromq-dev <[email protected]> writes: > Hello, > > I'm new to ZMQ and I recently started a project using the publisher and > subscriber model. > However, in my application, the client who's subscribing is making decisions > based on the > data collected during a small amount of time. Ideally, it would collect data > for 1 sec then make > a decision(during the decision-making period it would still be collecting > data for the next > decision), so I'm thinking that I can publish data into a buffer then grab > all the data in there > at a certain rate. > > Does ZMQ provide a function or model that supports this kind of design? I > know that I can > create a buffer in a separate thread by myself and store the data there, but > I would like to > know if ZMQ had already provided a function for it before I'm heading that > way. > > The language I'm using is Python and OS is Unbuntu 16.04. > > Thank you very much! > > Best regards, > Ziang > > _______________________________________________ > zeromq-dev mailing list > [email protected] > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
signature.asc
Description: PGP signature
_______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
