Hi Michal, I am not happy with limits, they are hard to get right and will probably break legal cases. My proposal is to
a) Fix fty-rest to use malamute client pool https://github.com/42ity/fty-rest/blob/master/src/shared/tntmlm.cc exclusively. This way you will have fixed number of mlm clients, which will regularly read their mailboxes. Theoretically it is possible that their mailbox will grow indefinitely, but that'd be very pathological case - like only one thread issuing long replies and timeouts all the time. b) Extend malamute protocol, so clients can mark themselves as "ephemeral". Adding broker based configuration just brings additional hidden protocol. Client always knows if he wants to have persistent mailbox or not. Don't forget this does not work with mlm client pool :-) See following example (and please come with better names :-) // Set client's mailbox as temporary, so messages won't be queued if client is not connected. // Return 0 if successful, -1 otherwise (interrupted, not supported) int mlm_client_set_temporary_mailbox (mlm_client_t *); //... mlm_client_t *client = mlm_client_new (); mlm_client_set_temporary_malbox (client); mlm_client_connect (...); The limits inside broker looks really complicated and impossible to setup in real use cases. Plus if you really know that if client is gone you're not interested in messages, "ephemeral" clients is what you want to. Bye Michal On Mon, Dec 11, 2017 at 12:23 PM, Osiris Pedroso <[email protected]> wrote: > Sounds like a good improvement on Malamute's current implementation. > Go ahead a submit your PR wit these changes. > If someone deems them improvable, they will do so by doing a new PR. > > PS: About the limits, I suggest a much smaller limit for the warning: > size-limit-warn = 65536 > > > > On Fri, Dec 8, 2017 at 8:07 AM Marek, Michal <[email protected]> > wrote: > >> Hi, >> >> I've been debugging issues in our project, where malamute's memory usage >> increases due to undelivered mailbox messages. The mailbox/size-limit >> config option helps in some cases, but it does not cover everything. >> E.g., we have client connections that are ephemeral -- they connect with >> a randomly generated address, send a request to some other client's >> mailbox and either receive a response or disconnect after a timeout. The >> result is that if the system is under higher load, malamute ends up with >> bunch of queues holding one message each. Another issue is that it's >> difficult to debug such cases, because the mailbox engine is rather >> silent. Would you guys take patches that >> >> A) add some "soft" limit for the queue size to log a warning when the >> mailbox grows too much? I'm thinking of something like >> mlm_server >> mailbox >> size-limit = 524288 >> size-limit-warn = 262144 >> >> B) add an aging mechanism for messages? The idea is that a message >> undelivered for a long period of time can a sign of a problem, either >> on the sender or the receiver. In fact, in our use case, all the >> clients are supposed to be connected, so actually any queued message >> is an anomaly. >> >> C) implement some special handling of ephemeral clients, so that >> messages directed to them are either delivered directly or dropped. >> Now the problem is that if the clients are to advertise their >> ephemeral nature, malamute would have to keep track of past clients, >> so the memory usage would just accumulate elsewhere :). How about >> keying this off the client address somehow, e.g. via a configurable >> regex: >> mlm_server >> mailbox >> no-queue-pattern = ^temp\..* >> and the clients would set their address to "temp.XXXXXX". >> >> The above also applies to service queues to some extent, we just do not >> use this model in our project. What do you think? >> >> Thanks, >> Michal >> >> -- >> Michal Marek Eaton European Innovation Center >> Open Source Developer Bořivojova 2380 >> 42ity.org 252 63 Roztoky >> Tel: +420 211 151 532 <+420%20211%20151%20532> www.eaton.cz >> >> >> >> >> >> ----------------------------- >> Eaton Elektrotechnika s.r.o. ~ Sídlo společnosti, jak je zapsáno v >> rejstříku: Komárovská 2406, Praha >> <https://maps.google.com/?q=Kom%C3%A1rovsk%C3%A1+2406,+Praha&entry=gmail&source=g> >> 9 - Horní Počernice, 193 00, Česká Republika ~ Jméno, místo, kde byla >> společnost zaregistrována: Praha ~ Identifikační číslo (IČO): 498 11 894 >> >> ----------------------------- >> >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > > _______________________________________________ > zeromq-dev mailing list > [email protected] > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > -- best regards Michal Vyskocil
_______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
