Hi All, Slightly on and off topic at the same time:
I am considering using PUB/SUB for distributing raw data packets from our instrument. While it is not absolutely critical that the packets be perfectly ordered, it is pretty important none are dropped. Our rate is fairly fast. I need to be able to publish 100,000 512 byte messages per second and hopefully for some overhead extend this to say 125,000. The instrument sends its data over a hard wire direct private LAN between the instrument and the computer. So we have a second NIC which will receive this high speed data from the instrument. My hope is to receive it, then publish it using ZeroMq both intranet (inside the computer network stack to other programs running on the same computer) and over the LAN. Understandably clients that connect say over a poor WiFi will not be able to maintain that reception rate. But to test this concept I wrote a simple C# console app that generates the 512 byte packets and sends them in a fast loop using ZeroMq PUB socket. Each packet has an incrementing number: 0...1..2..3.. etc... I then wrote a client receiver C# console program that subscribes and receives the packets. I am aware of the initial sync errors and I am not worried about that issue. But I do see if I either have the computer get a little busy or I subscribe more than say 3 receivers that some of the receivers will report message loss. I am simply looking at the message received, extracting the number and ensuring it is +1 from the last message received. I have increased the TX and RX buffers on the sockets to 1048576. I have tried increasing the threads as well. I still will see these message losses. I'm trying to figure out if I have just reached some limit or if there is some setting I need to set or if I just have some bugs. If I do a single TX/RX pair then it pretty much handles 125,000 messages per second all day long unless the computer really bogs down. Add a second RX and it will generally be ok but the TX will slow down a little as I would expect since it is sending 2x messages now. Add a third RX and I start to see message loss sporadically. I would be more than willing to post some code but wanted to just see if I'm in the ballpark with this idea first. Thanks - Mark **************************************************************************************************** This message contains information that may be privileged or confidential and is the property of Propel Labs, Inc. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message. **************************************************************************************************** _______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
