Hi Bill. Thank you for your email. See my answers below.
On Fri, May 29, 2020 at 11:13 PM Bill Torpey <[email protected]> wrote: > A big question is what the “workers” do? > Each worker is a separate logical unit that is responsible for its own logic. Usually, a worker will update some shared state based on the message it got from the broker. > > - It sounds like you want to process “transactions” — i.e., the workers > need to be able to reliably update some shared state. Is that true? > That is true. > - If so, which of the ACID properties do you care about? > All of them: atomicity, consistency, isolation, and durability > - Are your transactions “idempotent” — i.e., does it not matter if a > transaction is executed multiple times? > Some of the workers can process the same transaction more than once and that won't affect the system in a bad way. Some transactions cannot be processed more than once. It is all about processing the original message that originated from the publisher just once. - Do transactions need to ordered with respect to each other? > Yes. The order is important. > > Note that ZeroMQ is NOT a transaction processing system, nor is there any > sort of “guaranteed delivery” option in ZeroMQ. If you really are > processing “transactions” then the tricky parts are going to be in your > code — not ZeroMQ. > > It may make sense to flesh out that design before getting into details of > the network. > > HTH > > On May 29, 2020, at 3:46 PM, Doron Somech <[email protected]> wrote: > > I suggest not to use pub sub. Instead, if the number two is constant, you > can have two dealers, one for each, connect multiple brokers to those > dealers, now the broker are not a single point of failure. If you want to > make it more scalable, you can have a router socket instead of the pub > socket and smartly publish the message to multiple brokers, you will to > manage the data structure and distribution algorithm yourself. Continue to > read the guide, there multiple examples on how to use router together with > your own algorithm and data structure. Also read the pubsub chapter (I > think it is chapter 4), advanced request-response (5?) and look for the > binary star pattern. > > On Fri, May 29, 2020, 22:24 Nir Sharony <[email protected]> wrote: > >> Hi, >> >> While going over the guide, I ran across the section >> <http://zguide.zeromq.org/page:all#Shared-Queue-DEALER-and-ROUTER-sockets> >> that talks about managing a broker between clients and servers. >> What I need is something similar to the pattern below, except that >> instead of REQ clients, I am using PUB publishers and instead of REP >> services, I would like to use PULL workers. >> <Screen Shot 2020-05-29 at 21.48.25.png> >> This design attempts to explain what I want to achieve. >> The publisher broadcasts messages that are handled by two brokers. >> Each broker is in charge of its own group of workers (each group is >> enclosed in a box in the design). >> The brokers use the PUSH/PULL pattern to delegate the message to ONE of >> the workers in their group. >> I can bring as many workers as needed without changing the rest of the >> system. >> <Screen Shot 2020-05-29 at 22.13.09.png> >> The blue elements are the pieces of code that actually do the work and I >> can bring as many of them as needed. >> However, the introduction of a broker seems to bring a "single point of >> failure" into the system. >> If the application that runs the broker dies, the entire module in its >> box is no longer working... >> >> Is this a known issue or am I missing something out? >> What I want to make sure is that there are no single points of failure in >> the system. >> I thought of having two brokers per group (both will act the SUB >> consumers for the PUB message). >> However, I don't know if it is possible to split the work between such >> two brokers so that they will delegate the message to only one worker in >> the group. >> >> Any assistance would be greatly appreciated. >> Thanks, >> Nir >> >> >> >> >> On Fri, May 29, 2020 at 12:57 PM Doron Somech <[email protected]> wrote: >> >>> Hey, I suggest starting with the guide: http://zguide.zeromq.org/ >>> >>> If you then have any questions about specific pattern I would be able to >>> help. >>> >>> On Fri, May 29, 2020, 11:26 Nir Sharony <[email protected]> wrote: >>> >>>> Hello fellow developers. >>>> >>>> I am trying to develop a robust distributed system that will allow >>>> communication between Docker microservices, each being a separate process. >>>> The system needs to either run on the cloud or on-premise on dedicated >>>> hardware. >>>> >>>> In order to decouple dependency between the various microservices. I >>>> thought of using the Pub/Sub communication pattern. >>>> >>>> I am fairly new to ZeroMQ, so I am wondering how it handles failures, >>>> specifically in a >>>> >>>> >>>> - Does the queue itself constitute a single point of failure?` >>>> - Could the queue crash and the unsent payloads be lost? >>>> - How should the system recover from a crash? >>>> >>>> >>>> I would appreciate any references to documentation on this issue. >>>> >>>> Thanks in advance for your help. >>>> >>>> _______________________________________________ >>>> zeromq-dev mailing list >>>> [email protected] >>>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>> >>> _______________________________________________ >>> zeromq-dev mailing list >>> [email protected] >>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >> _______________________________________________ > zeromq-dev mailing list > [email protected] > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > >
_______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
