Hi Mark, have a look at ZMQ_XPUB_NODROP socket option. Despite the name, it applies also to PUB sockets (not only XPUB) and allows you to have a PUB/SUB without drops. I'm using that in my applications all the times.
Btw I'm sending Mpps through PUB/SUB ZMQ sockets, so 100kpps should not be a big trouble for ZMQ. HTH, Francesco 2017-12-06 16:06 GMT+01:00 Mark Holbrook <[email protected]>: > Hi All, > > Slightly on and off topic at the same time: > > I am considering using PUB/SUB for distributing raw data packets from our > instrument. > > While it is not absolutely critical that the packets be perfectly ordered, > it is pretty important none are dropped. > > Our rate is fairly fast. I need to be able to publish 100,000 512 byte > messages per second and hopefully for some overhead extend this to say > 125,000. > > The instrument sends its data over a hard wire direct private LAN between > the instrument and the computer. So we have a second NIC which will > receive this high speed data from the instrument. > > My hope is to receive it, then publish it using ZeroMq both intranet > (inside the computer network stack to other programs running on the same > computer) and over the LAN. > > Understandably clients that connect say over a poor WiFi will not be able > to maintain that reception rate. > > But to test this concept I wrote a simple C# console app that generates > the 512 byte packets and sends them in a fast loop using ZeroMq PUB > socket. Each packet has an incrementing number: 0...1..2..3.. etc... > > I then wrote a client receiver C# console program that subscribes and > receives the packets. I am aware of the initial sync errors and I am not > worried about that issue. > > But I do see if I either have the computer get a little busy or I > subscribe more than say 3 receivers that some of the receivers will report > message loss. > > I am simply looking at the message received, extracting the number and > ensuring it is +1 from the last message received. > > I have increased the TX and RX buffers on the sockets to 1048576. I have > tried increasing the threads as well. I still will see these message > losses. > > I'm trying to figure out if I have just reached some limit or if there is > some setting I need to set or if I just have some bugs. > > If I do a single TX/RX pair then it pretty much handles 125,000 messages > per second all day long unless the computer really bogs down. > > Add a second RX and it will generally be ok but the TX will slow down a > little as I would expect since it is sending 2x messages now. Add a third > RX and I start to see message loss sporadically. > > I would be more than willing to post some code but wanted to just see if > I'm in the ballpark with this idea first. > > Thanks - Mark > > > > > ************************************************************ > **************************************** > This message contains information that may be privileged or confidential > and is the property of Propel Labs, Inc. It is intended only for the > person to whom it is addressed. If you are not the intended recipient, you > are not authorized to read, print, retain, copy, disseminate, distribute or > use this message or any part thereof. If you receive this message in error, > please notify the sender immediately and delete all copies of this message. > ************************************************************ > **************************************** > _______________________________________________ > zeromq-dev mailing list > [email protected] > https://lists.zeromq.org/mailman/listinfo/zeromq-dev >
_______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
