I am not a maintainer of the project so it's probably best if someone else (who properly understands) guides you further. There may be another way to trigger the cleanup but I don’t know what it is.
All I am saying when you have ZMQ_CONFLATE set, the incoming message (and their associated memory) are only cleared after you have called zmq_recv() 100 times. Until you have the correct answer I would just have a 1 second timeout on your keyboard read, if the keyboard is not hit in one second do a zmq_read() and throw away the message. That will keep the memory in check. -----Original Message----- From: zeromq-dev <[email protected]> On Behalf Of Kevin Wang Sent: 21 June 2018 15:07 To: ZeroMQ development list <[email protected]> Subject: Re: [zeromq-dev] Memory leak on SUB set with CONFLATE? I see, thanks James. I had no idea config.hpp existed. Why would the queue grow if I have conflate set? Doesn’t that mean it should only queue one value? I’m doing this because there is some data that I would like client programs to always have access to, but only the most recent one. So my solution is to use CONFLATE and always send it out. If you have a better solution to that, I’d be glad to hear it. But until then, maybe I should change that value to 1. What other penalties/trade offs are there for changing it to 1? > On Jun 21, 2018, at 4:39 AM, James Harvey <[email protected]> wrote: > > Hi Kevin, > > I ran your test code and saw the same issue. I also created the tests with > straight libzmq (not czmq) and the issue still occurs. > > The memory is not lost (valgrind confirms) but it does grow. > > Eventually I noticed its proportional to the number of times you call > zmq_recv(). > > Adding code to force heap cleanup and print the stats shows that every 100 > times zmq_recv is called, the queue is purged. > > malloc_trim(0); > malloc_stats(); > > Finally I found inbound_poll_rate which is used by socket_base to determine > how often to check for signals and commands. > https://github.com/zeromq/libzmq/blob/b77d7610cd91dbe25096b804c77b2706 > 5b7dd1fd/src/config.hpp#L53 > > Reducing this 1 (as a test) fixed the issue and memory never rose above 4Mb. > > Is this a real issue? I am not sure. > I guess for some edge cases where you receive a truck load of data and don’t > process it. But on the other hand why would you do that. > > Possible fixes I can think of would be a socket_option to modify > inbound_poll_rate OR Force process_commands() on a timer if no > activity is occurring. > > Thanks > > James > > > > > > -----Original Message----- > From: zeromq-dev <[email protected]> On Behalf Of > Kevin Wang > Sent: 20 June 2018 22:12 > To: [email protected] > Subject: Re: [zeromq-dev] Memory leak on SUB set with CONFLATE? > > The memory of the SUB process goes up. > > If I remove the getline() call and the SUB process keeps up, the RAM stays > still for both. Those programs compile with gcc pretty easily, if you feel > like trying it out. Maybe my Ubuntu is just messed up. > > I assume since its CONFLATE is on, maybe the PUB doesn't queue it, but SUB > still does? In that case, I also would assume that replacing the old message > with the new one would free the memory held by the old message. > > >> On 06/20/2018 01:38 PM, James Harvey wrote: >> When you say “the system memory skyrockets” are you talking about the memory >> used by the SUB process or PUB process? >> >> I would expect the memory of the PUB process to increase as it will be >> queueing messages. >> >>> On 20 Jun 2018, at 21:02, Kevin Wang <[email protected]> wrote: >>> >>> the system memory skyrockets, >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > _______________________________________________ > zeromq-dev mailing list > [email protected] > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ > zeromq-dev mailing list > [email protected] > https://lists.zeromq.org/mailman/listinfo/zeromq-dev _______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev _______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
