Hi Andy, I remember writing a faq about process counts growing.
https://cyrusimap.org/imap/faqs/o-toomanyprocesses.html I'm not actually sure it's applicable in your situation, but just in case... Cheers, Nicola ----- Original message ----- From: "Andy Dorman via Info-cyrus" <info-cyrus@lists.andrew.cmu.edu> To: info-cyrus@lists.andrew.cmu.edu Subject: Re: Cyrus 2.5.9 imapd children and tcp_timeout questions Date: Mon, 12 Sep 2016 07:40:28 -0500 Eliie, thank you for that information about imapd usage. I have only been managing our imap for a few months and am still learning. We do not have a lot of active users on each server (probably under 20), but enough that what you describe could be happening. As for how I am determining they are reaching their usage limit, all I have are the process start times. When we hit the 100 limit I can see that the preforked 5 are still running as are many that were started many hours ago. I could be crazy wrong (probably am ;-) but it seemed to me that if processes were being killed and recycled I should not see a lot of old processes sitting around. And I owe an apology to Konrad...I originally wrote this email a week ago and then set it aside while I tried setting tcp_timeout=1. Then when that did not help, I sent the email and forgot to update my imapd.conf. This is what I have in my imapd.conf. tcp_keepalive: 1 tcp_keepalive_idle: 900 tcp_keepalive_cnt: 5 tcp_keepalive_intvl: 75 It would be great if anyone who is using tcp_keepalive could share their settings in case I completely misunderstood how these work. In the meantime I am going to bump my max processes up to 200 and my monitoring warning level to 150 see how that works. Thanks for all the help everyone. Andy On 09/11/2016 11:05 PM, ellie timoney via Info-cyrus wrote: >> We now have 4 of the 14 servers where the number of imapd processes >> slowly grows and usually within 12-14 hours reaches the imapd process >> limit (currently set to 100). > > How many user accounts are on these servers? Each client connection > constitutes one imapd process on the server. > > It's my understanding that Thunderbird (for example) in its default > configuration holds 5 connections open at once, and tries to keep them > open (either with IDLE, if enabled, or NOOP), and will reconnect them > after a while if they disconnect. (I would expect most other IMAP > clients of a similar maturity to behave similarly.) So it would only > take 20 users leaving their mail client open most of the time to hit > your limit of 100 concurrent imapd processes. > > So 100 seems like a very small number, unless you have a very small > number of user accounts. > >> From what we can tell watching the >> process list, the old processes are never shut down after >> reaching their >> usage limit (which is the default). > > Do you have more information about how you determined that their usage > limit had been reached? > >> While we had this problem intermittently with 2.4.18 (once or twice a >> month on a random server), it is much worse since the update >> to 2.5.9. > > This might simply be 2.5.9 doing a better job of helping clients keep > open connections open? > > Cheers, > > ellie > > On Mon, Sep 12, 2016, at 01:01 PM, Andy Dorman via Info-cyrus wrote: >> Hi everyone. We have 14 Debian servers running the current Debian >> release of Cyrus imap 2.5.9. >> >> We upgraded from 2.4.18 to 2.5.9 on Aug 28, upgraded the cyrus.conf & >> imapd.conf to rename the deprecated options and switch from >> skiplist to >> twoskip for all our dbs. Our cyrus.conf and imapd.con (minus >> comments) >> is at the end of this email. >> >> We now have 4 of the 14 servers where the number of imapd processes >> slowly grows and usually within 12-14 hours reaches the imapd process >> limit (currently set to 100). From what we can tell watching the >> process list, the old processes are never shut down after >> reaching their >> usage limit (which is the default). >> >> The difference between the 4 with the problem and the others >> have to be >> the mail clients. Each server supports a different group of mail >> domains and users, so I suspect some of the users on the problem >> servers >> are using one or more mail clients that are not well behaved, >> but I have >> no indication of what they are doing wrong. >> >> While we had this problem intermittently with 2.4.18 (once or twice a >> month on a random server), it is much worse since the update >> to 2.5.9. >> We now have to restart imapd on all the problem servers almost >> twice a >> day. >> >> If it matters, we have tried both idled and polling and have not seen >> any difference. >> >> So has anyone else seen an issue like this? Does anyone have any >> suggestions on how to debug? >> >> We have examined syslog and do not see any strange reports outside of >> the normal berkeley DBERROR messages we always see due to bdb >> compiled >> into the debian Cyrus release. >> >> ===== cyrus.conf ===== >> START { >> recover cmd="/usr/sbin/cyrus ctl_cyrusdb -r" >> idle cmd="idled" >> delprune cmd="/usr/sbin/cyrus expire -E 3" >> tlsprune cmd="/usr/sbin/cyrus tls_prune" >> } >> >> SERVICES { >> imap cmd="imapd" listen="*:imap" prefork=5 maxchild=100 >> imaps cmd="imapd -s" listen="*:imaps" prefork=5 maxchild=100 >> lmtp cmd="lmtpd" listen="*:lmtp" prefork=5 maxchild=20 >> sieve cmd="timsieved" listen="*:sieve" prefork=0 maxchild=100 >> } >> >> EVENTS { >> checkpoint cmd="/usr/sbin/cyrus ctl_cyrusdb -c" period=30 >> delprune cmd="/usr/sbin/cyrus expire -E 3" period=120 >> tlsprune cmd="/usr/sbin/cyrus tls_prune" period=60 >> resquat cmd="/usr/sbin/cyrus squatter -s -r -i *" at=0437 >> } >> >> ===== imapd.conf ===== >> admins: cyrus >> >> allowallsubscribe: on >> >> allowanonymouslogin: no >> >> allowplaintext: on >> >> altnamespace: on >> >> annotation_db: twoskip >> >> anyoneuseracl: off >> >> autocreate_quota: 512000 >> >> configdirectory: /var/lib/cyrus >> >> defaultdomain: ironicdesign.com >> >> defaultpartition: default >> >> duplicate_db: twoskip >> >> expunge_mode: delayed >> >> fulldirhash: on >> >> guid_mode: sha1 >> >> hashimapspool: on >> >> idlesocket: /var/run/cyrus/socket/idle >> >> improved_mboxlist_sort: on >> >> lmtp_downcase_rcpt: on >> >> lmtpsocket: /var/run/cyrus/socket/lmtp >> >> mboxname_lockpath: /run/cyrus/lock >> >> notifysocket: /var/run/cyrus/socket/notify >> >> partition-default: /var/spool/cyrus/mail >> >> popminpoll: 1 >> >> proc_path: /run/cyrus/proc >> >> quota_db: twoskip >> >> sasl_log_level: 7 >> sasl_mech_list: PLAIN >> sasl_pwcheck_method: saslauthd >> sasl_minimum_layer: 0 >> allowapop: no >> >> sieveusehomedir: false >> sievedir: /var/spool/cyrus/sieve >> >> statuscache_db: twoskip >> >> syslog_prefix: cyrus >> >> tls_sessions_db: twoskip >> >> tls_client_ca_file: /etc/ssl/certs/mail.ironicdesign.com.pem >> >> tls_client_ca_dir: /etc/ssl/certs >> >> # The rest of our TLS setup >> tls_server_cert: /etc/ssl/certs/mail.ironicdesign.com.pem >> tls_server_key: /etc/ssl/certs/mail.ironicdesign.com.pem >> tls_session_timeout: 1440 >> >> tls_ciphers: TLSv1+HIGH:!aNULL:@STRENGTH >> >> tls_versions: tls1_0 tls1_1 tls1_2 >> >> umask: 077 >> >> unix_group_enable: off >> >> unixhierarchysep: on >> >> userdeny_db: twoskip >> >> virtdomains: userid >> >> sieve_extensions: fileinto reject vacation imapflags notify envelope >> body relational regex subaddress copy >> ===== end imapd.conf ===== >> >> Thanks for any ideas on what to look for or how to debug this. >> >> -- >> Andy Dorman >> Ironic Design, Inc. >> AnteSpam.com >> >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > ---- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
---- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
