HI,
to add some more interesting facts, I decided to pin threads to input/output by using the thread affinity option. After assigning one thread to the input socket and a set of 3 threads for output connection, the situation improved and the blocking calls got less, but I can still see them. Any idea what this means? Cheers, Jens Von: zeromq-dev [mailto:[email protected]] Im Auftrag von Jens Auer Gesendet: Dienstag, 21. Juni 2016 22:00 An: 'ZeroMQ development list' Betreff: Re: [zeromq-dev] EAGAIN on ZMQ_STREAM Hi all, I did some more investigations on this, but I still don’t understand the behavior I see. I have added some debugging output in ZeroMQ to print the size of the inbox when reaching the watermark, and I can confirm that this is actually happening. Everything works as expected when I set the HWM to 0, but I cannot do this because then a slow client would overflow the system. Just let me recapitulate my setting. I send data with ~8500 messages per second, all of size 1129 bytes. This should be ~100000bps. From time to time, the sending process generates a burst where it sends an additional package of 5s * 8500 messages from a buffer. My process receives the messages just fine on a SUB socket, and forwards them to external clients on a ZMQ_STREAM socket. I have set the HWM to a ridiculously large value of 100 * rate, so it should be large enough to compensate the burst. Even when taking into account that the actual HWM can be smaller (the doc says 60%) and that I need to send the id message for every message, I should be able to buffer 25*rate messages. However, I sometimes see that sending would block and I lose the data because I send non-blocking. I think that 100 * rate should be basically the same as HWM=0, but this does not seem to be the case. When running without HWM, I expected to see some increase in memory consumption compared to the other settings, but that was not the case. This is strange, because now it can store as many frames as needed, and since the other settings block repeatedly, this should be more and probably grow over time. The memory consumption is the same regardless of the buffer settings. Cheers, Jens Von: zeromq-dev [mailto:[email protected]] Im Auftrag von Doron Somech Gesendet: Samstag, 18. Juni 2016 19:04 An: ZeroMQ development list Betreff: Re: [zeromq-dev] EAGAIN on ZMQ_STREAM It is correct, however when the socket is in shutdown process you can also get an EAGAIN. Which of the high watermark do you set, send or receive? Try to set both, it should be the send. On Sat, Jun 18, 2016 at 7:54 PM, Jens Auer <[email protected]> wrote: Hi, I’m using the C++ binding from https://github.com/zeromq/cppzmq/blob/master/zmq.hpp. It is quite simple. The send method I use (there are several depending on the parameters) is: inline bool send (message_t &msg_, int flags_ = 0) { int nbytes = zmq_msg_send (&(msg_.msg), ptr, flags_); if (nbytes >= 0) return true; if (zmq_errno () == EAGAIN) return false; throw error_t (); } In this case, the return value is false for EAGAIN and true otherwise. Other errors are handled as exceptions. I jut to ask under which circumstances EAGAIN is return for ZMQ_STREAM sockets. The HWM should be large enough for anything I send, but I will investigate more on Monday. Is it correct that HWM=0 disabled the limit completely and I should not expect any EAGAIN ever? Cheers, Jens Von: zeromq-dev [mailto:[email protected]] Im Auftrag von Doron Somech Gesendet: Samstag, 18. Juni 2016 18:47 An: ZeroMQ development list Betreff: Re: [zeromq-dev] EAGAIN on ZMQ_STREAM Which language and binding are you using? you are checking the return number from the send method, in most binding that return the number of bytes send and not the error code. It might be different with binding you are using but I'm not sure. Anyway if that is the case it might be that the number of bytes just equal EAGAIN? On Sat, Jun 18, 2016 at 11:59 AM, Jens Auer <[email protected]> wrote: Hi Doron, I don’t think it is disconnect. I am testing it on the local network, so there is not much internet infrastructure in between. I am using a pseudo client which logs socket errors, connects and disconnects and I don’t see anything. On the server side, I have a socket monitor which logs socket events but this does not show anything. Finally, if the TCP connection disconnects the ZMQ_STREAM socket shall create the empty disconnect message which I do not receive, neither before or after the EAGAIN. Best wishes, Jens Von: zeromq-dev [mailto:[email protected]] Im Auftrag von Doron Somech Gesendet: Freitag, 17. Juni 2016 22:14 An: ZeroMQ development list Betreff: Re: [zeromq-dev] EAGAIN on ZMQ_STREAM Why you can exclude the first case? TCP connection can get disconnect quite often. Anyway from what you are describing it seems the connection got disconnected. On Fri, Jun 17, 2016 at 8:05 PM, Jens Auer <[email protected]> wrote: Hi, I am witnessing something in my system that I didn't expect. I am processing data at a data rate of 100Mbps, with occasional peaks of 5s of data (=5 * 100Mbps) delivered as fast as possible because an internal buffer is flushed. I use ZeroMQ internally, but I have to deliver the data on a TCP plain socket at the endpoint of my system. Here, I use a ZMQ_STREAM socket where clients connect. Basically, I keep track of connected clients and send the data to all clients as a copy of the message. The main loop here looks like for(auto const& c: clientIdMessages) { bool eagain = !socket.send(c.copy(), ZMQ_DONTWAIT | ZMQ_SNDMORE); if (!eagain) { eagain = !socket.send(msg.copy(), ZMQ_DONTWAIT | ZMQ_SNDMORE); } } This usually works fine, but from time to time a message is skipped because send returns with EAGAIN. If I understand the docs correctly, this happens if the peer is unknown (ZMQ_STREAM behavior) or when the message cannot be queued. I can exclude the first case, so I must a high HWM for sending messages. However, I tried to increase the HWM to a huge value of 50 times the expected message rate and it did not solve the problem. I also tried to disable the HWM by setting it to 0, which means "no limit". Still, I am getting an EAGAIN which I don't expect. I can exclude any previous loss because I have counters showing me that the full data stream is delivered to the ZMQ_STREAM socket code above. I also added an assertion on !eagain after sending, and this fires. Are there any other conditions when I can get EAGAIN from a ZMQ_STREAM socket? Best wishes, Jens _______________________________________________ zeromq-dev mailing list <mailto:[email protected]> [email protected] <http://lists.zeromq.org/mailman/listinfo/zeromq-dev> http://lists.zeromq.org/mailman/listinfo/zeromq-dev --- Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. https://www.avast.com/antivirus
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
