Hi Nir, we created the Dafka protocol which fulfills many of your requirements. Have a look at the reference implementation and the rfc.
https://github.com/zeromq/dafka https://rfc.zeromq.org/spec/46/ Cheers Kevin Nir Sharony <[email protected]> schrieb am Sa., 6. Juni 2020, 15:40: > Hi, guys. > > I wanted to share with you some design requirements and see if you could > point me in the right direction. > > I am developing a system that consists of multiple microservices that need > to communicate among themselves. Microservices belong in groups. In order > to not have a single point of failure and to handle a large volume of > messages, each group will contain at least two (probably more) > microservices. > > When a microservice publishes a message, it designated to a specific group > and should be processed by just one microservice within that group. Note > that if the processing fails (microservice dies in the middle), the same > message should be picked up some other microservice in the same group. > > An additional requirement in my system is that the order of the messages > that are being processing by a group, must be maintained because the > internal state relies on it. While this is true for all the message > consumers in the system, in practicality, the chance of messages getting > processed out of order highly depends on the pace that they are being > produced and consumed. > > Some messages are produced at a low rate (one message every 5-10 seconds). > As the processing time for each message is just a few milliseconds, the > chance that consumer A will finish processing message X *AFTER* consumer > B finished processing message X+1 is slim. > In such cases, we can probably lift that "maintain message order" > requirement and we will still be ok. > > However, some producers produce messages at a much higher pace (sometimes > more than 30 times a second). This means that the chance the consumers are > processing those jobs out of order increases. > In such cases, the "maintain message order" cannot be ignored. > > Does ZeroMQ address these issues at all? > > Thanks in advance. > _______________________________________________ > zeromq-dev mailing list > [email protected] > https://lists.zeromq.org/mailman/listinfo/zeromq-dev >
_______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
