Thank you guys for the answers! That's plenty of information to get me going.
Regards, Gyorgy Szekely On Thu, May 14, 2020 at 6:13 PM Brett Viren <[email protected]> wrote: > I was going to reply similarly. Once a message is outside a socket, > it's out of ZeroMQ's hands and the responsibility of the application. > > This aspect is somewhat fresh in my own learning. The core issue here > (in my mind) is that the broker (router) severs any direct communication > between producer and consumer. Thus any back pressure from consumer to > producer must be communicated through the broker application. It may > sound obvious to say it like this but it took me a bit to internalize > this basic idea. > > If it's any help to look at some examples, I've recently implemented a > form of credit-based flow control following the ZeroMQ guide > description. It adds a runtime choice to use ROUTER/DEALER or > SERVER/CLIENT. The implementation is somewhat enmeshed in other choices > in my library but it could still serve as a general example. > > Actually, I implmented it twice, once in PyZMQ and once in cppzmq. > Barring some bug, the two should inter-operate. > > PyZMQ: > > https://github.com/brettviren/zio/tree/master/python/zio/flow > > cppzmq: > > https://github.com/brettviren/zio/blob/master/inc/zio/flow.hpp > https://github.com/brettviren/zio/blob/master/src/flow.cpp > > (flow.cpp is kind of a mess due to me still trying to understand > boost.sml which I use for the FSM. I welcome any critiques if anyone > cares to offer them.) > > One caveat, this code is Free to use but I don't consider it released. > I may still end up making drastic changes. > > But, to understand (this implementation of) credit-based flow control > this documentation and diagram may be more useful than the code: > > https://brettviren.github.io/zio/flow.html > > > And, one last comment: The Python implementation includes a "broker" > that does a little dance in order to hook up the "producer" and the > "consumer" ends in a way that allows each to think they are simple > clients and thus they can reuse the same "flow protocol" code in a > symmetric manner regadless of which role each endpoint plays. I have > some rambling about how the broker does this dance: > > https://brettviren.github.io/zio/flow-broker.html > > There are likely other, and simpler ways to do this. I'd be interested > to hear any ideas on that. The main issue is the broker needs to spawn > a "back end" consumer (producer) to service the "front end" producer > (consumer). The broker has no a'priori info about its front end clients > and what they may want from a back end so this spawning must be done as > a dymamic reaction to the first message from a "front end". That and my > desire for symmetric "flow protocol" code leads to this somewhat > contorted dance. > > > Cheers, > -Brett. > > > Kevin Sapper <[email protected]> writes: > > > Hi Gyorgy, > > > > back-pressure is something very specific to your application. ZeroMQ > itself only implements blocking or dropping depending on the socket types > > you're using. > > > > To implement a credit-based flow have a look into the guide: > http://zguide.zeromq.org/page:all#Transferring-Files > > > > //Kevin > > > > Am Do., 14. Mai 2020 um 09:55 Uhr schrieb Gyorgy Szekely < > [email protected]>: > > > > Hi All, > > > > I have a message router that routes messages between different types of > producers and consumers. Socket types are router (message > > router), dealer (producer, consumer). Currently if a consumer is not > fast enough messages start to queue up (dealer input queue, router > > output queue), eventually the router starts to drop messages to avoid > blocking. > > > > I would like to implement some kind of flow control. With plain TCP I > can rely on the built-in flow control: write blocks if consumer is > > overloaded, and producers can be sanctioned by not reading their socket. > > > > With ZMQ router/dealer I can detect if a consumer is slow by receiving > EAGAIN on send, but as far as I understand I can't "slow down" a > > specific producer, because router socket does fair queuing. So I have > to do application layer "stop sending" and "continue" messages and > > send them to specific producers... > > > > Is there any better way to do this? I would rather not reinvent the > wheel, TCP already has a sophisticated mechanism for this. > > > > Regards, > > Gyorgy Szekely > > > > _______________________________________________ > > zeromq-dev mailing list > > [email protected] > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > _______________________________________________ > > zeromq-dev mailing list > > [email protected] > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ > zeromq-dev mailing list > [email protected] > https://lists.zeromq.org/mailman/listinfo/zeromq-dev >
_______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
