expunge_days and cyr_expire
How do the 'expunge_days' option in imapd.conf and the -X flag to cyr_expire play together? Currently we're using this cyrus.conf entry: delprune cmd="cyr_expire -E 3 -X 3 -D 3" at=0400 I'm not sure if we should use 'expunge_days: 3' from now on or if it doesn't matter? Cheers, Sebastian -- .:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:. p7sJ49CMCW9Sp.p7s Description: S/MIME cryptographic signature Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
tcp_keepalive option
Hi, I've got a question regarding the tcp_keepalive option that's new in Cyrus 2.4: we've always had problems with POP and IMAP processes that got stuck when a connection was dropped in the "right" way. It's gotten better over the years, but it still happens with 2.3.14. I'm thinking that tcp_keepalive might get rid of this problem for good. Thoughts? Is there a downside I'm not seeing? Thanks, Sebastian -- .:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:. p7scEQPVTUnVv.p7s Description: S/MIME cryptographic signature Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Questions before upgrading vom 2.3 to 2.4
Hi Pascal, thanks for your reply! I decided to ask my more specific questions in separate mails, but I have a few follow-up questions for you. --On 5. Januar 2012 19:50:28 +0100 Pascal Gienger wrote: In principle it is simple, but I hit a hard ground today. I did the mistake to miscalculate the "converting indexes" Our storage was perfectly sized to handle the needed IOPS for cyrus operation, but having 8000 index files being rebuild simultaneosuly... I am very aware of that issue, but we haven't yet decided how to handle it. I'm actually considering a scheduled downtime for the conversion. I'm planning on performing the actual upgrade on a backup of our mail store to have an estimate of how long it's going to take. Important: Set lmtp_mail_timeout (when you use postfix to deliver mail) to a low value, like 2seconds, because also cyrus lmtpd triggers the index rebuild of a INBOX. So people with less mail in their INBOX (which already constructed index and cache) still get their mail in their INBOX, and people having MANY mails so that the index rebuild takes time will get the mail later due to the early deferred state (after lmtp_mail_timeout). Otherwise you'll have many blocked Postfix lmtp and no mail gets transferred. That's a Postfix option, I suppose? We use Sendmail. I don't think there's an exact equivalent for that option. Do you know after which LMTP command the rebuild takes place? I would assume that it's after "rcpt to:"? So the LMTP session has to wait for that before it can issue the "data" command? In that case I suppose I could use 'confTO_RCPT'. And be familiar with expunge_mode: delete_mode: expunge_days: and the consequences on cyr_expire! You mean that with "delayed" mode there's a lot to do for cyr_expire? But that hasn't changed between 2.3. and 2.4, right? I asked about expunge_days in a separate message. With meta-partition on a fast SSD device this even would not have occured (500 GB SSD needed in our case) We don't have any of those, unfortunately ... Sebastian -- .:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:. p7sfPQGdTIThQ.p7s Description: S/MIME cryptographic signature Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: expunge_days and cyr_expire
On Fri, 2012-01-06 at 10:22 +0100, Sebastian Hagedorn wrote: > How do the 'expunge_days' option in imapd.conf and the -X flag to > cyr_expire play together? Currently we're using this cyrus.conf entry: > delprunecmd="cyr_expire -E 3 -X 3 -D 3" at=040 > I'm not sure if we should use 'expunge_days: 3' from now on or if it > doesn't matter? Good question! I hadn't noticed that one. I wonder when messages qualifying under "expunge_days" get expunged if cyr_expire is not run or run with a different value. Also does expunge_days calculate from when the message was *expunged* or when the message was *received*. I know there has been questions about that concerning cyr_expire in the past. The documentation of expunge_days seem clearer than the text for cyr_expire [of the previous text for cyr_expire]. man cyr_expire: Expunge previously deleted messages older than expunge-duration (when using the "delayed" expunge mode). Format is the same as delete-duration. NOTE: "older" than "expunge-duration". At least to me the phrase "older" seems ambiguous. The documentation for delete-duration uses the same phrasing. man imapd.conf Number of days to retain expunged messages before cleaning up their index records. The default is 7. NOTE: "number of days to retain" seems [to me] to clearly mean X number of days since the message was expunged. signature.asc Description: This is a digitally signed message part Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Questions before upgrading vom 2.3 to 2.4
On Fri, January 6, 2012 11:04 am, Sebastian Hagedorn wrote: > Hi Pascal, > > > thanks for your reply! I decided to ask my more specific questions in separate > mails, but I have a few follow-up questions for you. > > --On 5. Januar 2012 19:50:28 +0100 Pascal Gienger > wrote: > > >> In principle it is simple, but I hit a hard ground today. I did the >> mistake to miscalculate the "converting indexes" >> >> Our storage was perfectly sized to handle the needed IOPS for cyrus >> operation, but having 8000 index files being rebuild simultaneosuly... > > I am very aware of that issue, but we haven't yet decided how to handle it. Sebastian, Pascal, Neither have we, and I know for sure my I/O subsystem is unable to handle the extra metadata reconstruction load. To give you an idea : 73k users, 508k mailboxes, 4.5 TB of messages, 48M messages. In an additional effort to grasp the dynamics of our central mail system (when hosting over 10k users it pays to have regular reporting and statistics produced) I discovered that no less than 66% of messages reside in Inboxes. Which is extraordinary, given the fact that POP3 users number less than 10% of the total. (We are a mixed IMAP/POP shop, for historical reasons.) If more messages were to be found in sub-mailboxes, metadata conversion load would have been more distributed in time, as mailboxes are accessed. Regards, Eric Luyten, Computing Centre VUB/ULB. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Questions before upgrading vom 2.3 to 2.4
Hi Eric, --On 6. Januar 2012 15:00:44 +0100 Eric Luyten wrote: To give you an idea : 73k users, 508k mailboxes, 4.5 TB of messages, 48M messages. In an additional effort to grasp the dynamics of our central mail system (when hosting over 10k users it pays to have regular reporting and statistics produced) I discovered that no less than 66% of messages reside in Inboxes. Which is extraordinary, given the fact that POP3 users number less than 10% of the total. (We are a mixed IMAP/POP shop, for historical reasons.) If more messages were to be found in sub-mailboxes, metadata conversion load would have been more distributed in time, as mailboxes are accessed. that's interesting. I guess we're pretty much in the same boat, but I don't have the numbers to prove it. Can you share the scripts you use for reporting? Cheers, Sebastian -- .:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:. p7s2NmQewu3gE.p7s Description: S/MIME cryptographic signature Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: anysievefolder and autosievefolders gone in 2.4?
Ok, I didn't notice that it is a separate patch, because the cyrus-imapd package of RHEL/CentOS was compiled with that patch, in Ubuntu it isn't. The patch works fine for several years now and I don't think that any new features will be added. Why hasn't it been incorporated and merged into the main tree of cyrus? Kind regards Marten Lehmann Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: migration and seen flag
> Do you know what format the seen files are in on the old server? I think > you should be able to copy them across with the rest of the mail spool > using rsync, but the default database format for seen might have changed. > The default in 2.4 appears to be skiplist. > > Check the setting seenstate_db in imapd.conf. > Hello, On the old server there is no setting for seenstate_db actually specified in the config, however it seems to be skiplist since on a restart it logs; imapd[27028]: skiplist: recovered /var/lib/imap/user/v/viktort.seen The thing is on the 2.4.13 server I have configured; seenstate_db: skiplist however it doesn't create any .seen files like the 2.1.x version, it seems to store seen information in the "cyrus.index" file in each users mailspool directory. I am using the invoca rpm build, is this a particularity or is there another setting I have to set/unset to imitate the previous behaviour? Thanks, Ron Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Questions before upgrading vom 2.3 to 2.4
On Fri, January 6, 2012 3:05 pm, Sebastian Hagedorn wrote: > Hi Eric, > > > --On 6. Januar 2012 15:00:44 +0100 Eric Luyten > wrote: > > >> To give you an idea : 73k users, 508k mailboxes, 4.5 TB of messages, 48M >> messages. In an additional effort to grasp the dynamics of our central mail >> system (when hosting over 10k users it pays to have regular reporting and >> statistics produced) I discovered that no less than 66% of messages reside in >> Inboxes. Which is extraordinary, given the fact that POP3 users >> number less than 10% of the total. (We are a mixed IMAP/POP shop, for >> historical reasons.) If more messages were to be found in sub-mailboxes, >> metadata conversion load would have been more distributed in time, as >> mailboxes are accessed. > > that's interesting. I guess we're pretty much in the same boat, but I don't > have the numbers to prove it. Can you share the scripts you use for reporting? Sebastian, They're obviously very site specific but this is more or less what we do. Two nested "for" loops, the first over our nine datastores, the second over all users in that datastore (list obtained by `ls /cyr.../user`) invoke 'mbexamine' on every Inbox and pipe its output through 'egrep '(Number of Messages|Examining user)' This produces 26 files which are then concatenated, run through #!/bin/sh while read line1 do read line2 echo $line1 $line2 done and finally (apply cosmetics) sed -e 's/^Examining //' -e 's/ Number of Messages: / Messages: /' -e 's/ Mailbox Size: / Size: /' which, at the end, produces user/aauser1... Messages: 171 Size: 22483962 bytes user/aauser1/Out... Messages: 3 Size: 1279 bytes user/aauser1/Spam... Messages: 0 Size: 0 bytes user/aauser1/Trash... Messages: 0 Size: 0 bytes user/aauser2... Messages: 391 Size: 16056164 bytes user/aauser2/Spam... Messages: 0 Size: 0 bytes <...> This process runs every Saturday morning and takes between 7 and 8 hours. grep "^user\/[a-z0-9]*\.\.\. " extracts the Inboxes. Eric Luyten. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: anysievefolder and autosievefolders gone in 2.4?
On Fri, Jan 06, 2012 at 03:20:06PM +0100, Marten Lehmann wrote: > Ok, > > I didn't notice that it is a separate patch, because the cyrus-imapd > package of RHEL/CentOS was compiled with that patch, in Ubuntu it isn't. > > The patch works fine for several years now and I don't think that any > new features will be added. Why hasn't it been incorporated and merged > into the main tree of cyrus? Sorry - I've been planning to do it for ages, and it just hasn't happened due to other things always being more pressing. It's on the "MUST HAVE" list for 2.5. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Questions before upgrading vom 2.3 to 2.4
On Fri, Jan 06, 2012 at 11:04:26AM +0100, Sebastian Hagedorn wrote: > >With meta-partition on a fast SSD device this even would not have > >occured (500 GB SSD needed in our case) > > We don't have any of those, unfortunately ... As nice as SSDs are, they wouldn't have helped - the IO hit is reading every single message file from the spools to rebuild cache and index files with accurate checksums. I'm not convinced it was the best idea (after the troubles people have had with it)... but the flip side would have been much more complex code paths for handling incomplete checksum data :( Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: migration and seen flag
On Fri, Jan 06, 2012 at 10:17:26AM -0500, Ron Vachiyer wrote: > however it doesn't create any .seen files like the 2.1.x version, it seems to > store seen information in the "cyrus.index" file in each users mailspool > directory. I am using the invoca rpm build, is this a particularity or is > there another setting I have to set/unset to imitate the previous behaviour? Seen data is only stored in .seen files for shared or other user mailboxes. The seen data for the mailbox owner is stored in the cyrus.index. This has two benefits: 1) lower IO for the most common case. We need to update the cyrus.index file anyway to record the new MODSEQ. 2) owner seen becomes shared seen if you switch to shared seen mode on the mailbox, so that you don't wind up with one huge file full of shared seen data if you have a lot of shared mailboxes. Seen files were actually a lot of extra IO on 2.3, small files with multiple fsyncs! Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: expunge_days and cyr_expire
On Fri, Jan 06, 2012 at 07:36:15AM -0500, Adam Tauno Williams wrote: > On Fri, 2012-01-06 at 10:22 +0100, Sebastian Hagedorn wrote: > > How do the 'expunge_days' option in imapd.conf and the -X flag to > > cyr_expire play together? Currently we're using this cyrus.conf entry: > > delprune cmd="cyr_expire -E 3 -X 3 -D 3" at=040 > > I'm not sure if we should use 'expunge_days: 3' from now on or if it > > doesn't matter? > > Good question! I hadn't noticed that one. I wonder when messages > qualifying under "expunge_days" get expunged if cyr_expire is not run or > run with a different value. > > Also does expunge_days calculate from when the message was *expunged* or > when the message was *received*. I know there has been questions about > that concerning cyr_expire in the past. The documentation of > expunge_days seem clearer than the text for cyr_expire [of the previous > text for cyr_expire]. > > man cyr_expire: > Expunge previously deleted messages older than expunge-duration > (when using the "delayed" expunge mode). Format is the same as > delete-duration. > > NOTE: "older" than "expunge-duration". At least to me the phrase > "older" seems ambiguous. The documentation for delete-duration > uses the same phrasing. > > man imapd.conf > Number of days to retain expunged messages before cleaning up > their index records. The default is 7. > > NOTE: "number of days to retain" seems [to me] to clearly mean X > number of days since the message was expunged. > Man - I'd better go write some nicer docs! expunge_days is kind of a "fallback" if you don't run cyr_expire it will still clean up eventually. So long as you set it longer than your cyr_expire time then it's fine - otherwise your other processes will do the cleanup when they feel like it. It's purely a "clean out expunged records" value, and it calculates from when they were expunged. Nothing about the expunge cleanup cares when the message was received. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: migration and seen flag
> Seen data is only stored in .seen files for shared or other user > mailboxes. The seen data for the mailbox owner is stored in the > cyrus.index. This has two benefits: > > 1) lower IO for the most common case. We need to update the >cyrus.index file anyway to record the new MODSEQ. > 2) owner seen becomes shared seen if you switch to shared seen >mode on the mailbox, so that you don't wind up with one huge >file full of shared seen data if you have a lot of shared >mailboxes. > > Seen files were actually a lot of extra IO on 2.3, small files > with multiple fsyncs! > > Bron. Hello, I am beginning to see this. Is there a mecanism to read from these .seen files to recover this data during a migration? Because from what I am seeing, if I move a .seen file from the old server, the new server never reads from it and all my seen data is lost. Thanks in advance, Ron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: anysievefolder and autosievefolders gone in 2.4?
Hello, > Sorry - I've been planning to do it for ages, and it just hasn't happened > due to other things always being more pressing. It's on the "MUST HAVE" > list for 2.5. ok, but where can I find a patch for the current 2.4.9 release of Ubuntu 11.10? The latest release I can find at http://email.uoa.gr/projects/cyrus/autosievefolder/ is from 2009 for the 2.3.16 release. Kind regards Marten Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Questions before upgrading vom 2.3 to 2.4
On Fri, January 6, 2012 5:50 pm, Bron Gondwana wrote: > I'm not convinced it was the best idea (after the > troubles people have had with it)... Hey, don't worry too much. We'll survive :-) Eric (taking the opportunity to thank you for your continuing efforts in developing and supporting a robust and solid productivity tool which I really love to administer) Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Questions before upgrading vom 2.3 to 2.4
On 01/06/2012 01:05 PM, Eric Luyten wrote: On Fri, January 6, 2012 5:50 pm, Bron Gondwana wrote: I'm not convinced it was the best idea (after the troubles people have had with it)... Hey, don't worry too much. We'll survive :-) Agreed. Two hours of high load on a Sunday morning was worth it to get a better performing Cyrus. Eric (taking the opportunity to thank you for your continuing efforts in developing and supporting a robust and solid productivity tool which I really love to administer) Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <> Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: anysievefolder and autosievefolders gone in 2.4?
> Hello, > >> Sorry - I've been planning to do it for ages, and it just hasn't >> happened >> due to other things always being more pressing. It's on the "MUST HAVE" >> list for 2.5. > > ok, but where can I find a patch for the current 2.4.9 release of Ubuntu > 11.10? The latest release I can find at > http://email.uoa.gr/projects/cyrus/autosievefolder/ is from 2009 for the > 2.3.16 release. Check here: http://www.vx.sk/download/patches/cyrus-imapd/ If they don't apply, then you can extract the version in our current rpm: http://www.invoca.ch/pub/packages/cyrus-imapd/RPMS/ils-5/SRPMS/cyrus-imapd-2.4.13-1.el5.src.rpm I think I had to rework them. Simon > > Kind regards > Marten > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: migration and seen flag
On Fri, Jan 06, 2012 at 11:58:40AM -0500, Ron Vachiyer wrote: > I am beginning to see this. Is there a mecanism to read from these .seen > files to recover this data during a migration? Because from what I am > seeing, if I move a .seen file from the old server, the new server never > reads from it and all my seen data is lost. Oh, if you've already moved the mailbox and the seen file hasn't been moved yet... yeah, that's messy :( The main problem is that the seen file is indexed by UNIQUEID rather than mailbox name, otherwise you could just copy-paste the sequence out and run TAG UID STORE +Flags \Seen via IMAP after selecting each folder. But I suspect you have hundreds of these folders? You can get the UNIQUEID from the cyrus.header file to script it. You can use cyr_dbtool from the new cyrus to dump the file and then use perl or something to munge the data. NOTE: you need to be able to log in to imap as the user themselves, otherwise you'll be setting the wrong seen flags. Possible cheap-n-nasty workaround is to use an admin connection and switch each mailbox to sharedseen, apply the flags, then switch it back. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: migration and seen flag
> Oh, if you've already moved the mailbox and the seen file hasn't > been moved yet... yeah, that's messy :( > > The main problem is that the seen file is indexed by UNIQUEID > rather than mailbox name, otherwise you could just copy-paste > the sequence out and run > > TAG UID STORE +Flags \Seen > > via IMAP after selecting each folder. > > But I suspect you have hundreds of these folders? You can get > the UNIQUEID from the cyrus.header file to script it. You can > use cyr_dbtool from the new cyrus to dump the file and then > use perl or something to munge the data. NOTE: you need to be > able to log in to imap as the user themselves, otherwise you'll > be setting the wrong seen flags. > > Possible cheap-n-nasty workaround is to use an admin connection > and switch each mailbox to sharedseen, apply the flags, then > switch it back. To be clear, here is my scenario; I have 8000 user accounts, 40+Gigs of email on a 2.1.17 server. They all seem to be using .seen files stored in /var/lib/imap/user/u/user.seen files. The mailspool is in /var/spool/imap/u/user/user If I copy the contents of /var/lib/imap/user/[a-z]/* and /var/spool/imap/[a-z]/* to a new server, create accounts beforehand on the new server, then reconstruct -r -f each account, on a 2.3.16 installation this works. I continue to read the .seen files as I am learning that this mecanism was still present in 2.3.x If I do this on a 2.4.13 server, however, the seen data is lost. Perhaps I have to upgrade to 2.3 first and then to 2.4? Ron Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: migration and seen flag
On Fri, Jan 06, 2012 at 01:40:10PM -0500, Ron Vachiyer wrote: > To be clear, here is my scenario; > > I have 8000 user accounts, 40+Gigs of email on a 2.1.17 server. They all > seem to be using .seen files stored in /var/lib/imap/user/u/user.seen files. > The mailspool is in /var/spool/imap/u/user/user > > If I copy the contents of /var/lib/imap/user/[a-z]/* and > /var/spool/imap/[a-z]/* to a new server, create accounts beforehand on the > new server, then reconstruct -r -f each account, on a 2.3.16 installation > this works. I continue to read the .seen files as I am learning that this > mecanism was still present in 2.3.x Far out. Bloody reconstruct. Don't do that. Copy /var/lib/imap with the databases as well, and don't reconstruct. > If I do this on a 2.4.13 server, however, the seen data is lost. Probably because you're missing other stuff :( > Perhaps I have to upgrade to 2.3 first and then to 2.4? It should just find the seen data during the upgrade. I'd be interesting in knowing how it didn't... can you send me an example .seen file from someone who doesn't mind it? Thanks, Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
