Cyrus IMAP 2.5.7 released

[ellie timoney via Info-cyrus]

The Cyrus team is proud to announce the immediate availability of a new 
version of Cyrus IMAP: 2.5.7

This is a bug fix release for the 2.5 series.

Download URLs:

  http://www.cyrusimap.org/releases/cyrus-imapd-2.5.7.tar.gz
  http://www.cyrusimap.org/releases/cyrus-imapd-2.5.7.tar.gz.sig

  ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-2.5.7.tar.gz
  ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-2.5.7.tar.gz.sig

Please consult the release notes before upgrading to 2.5.7:

  https://docs.cyrus.foundation/imap/release-notes/2.5-current.html

And join us at https://git.cyrus.foundation/ to report issues, join in
the deliberations of new features for the next Cyrus IMAP release, and
to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

Ellie Timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Security release for 2.4 series?

[ellie timoney via Info-cyrus]

Hi Tim,

Yeah, those are the correct two commits.

>> My other concern was that, honestly, I'm not all that sure what the
>> true risk and capability to exploit is for these bugs.  I've read the
>> CVE's, and associated discussions on the a few lists - but it hasn't
>> enlightened me as much as I've hoped.
>>

As I understand it, the vulnerability was that someone (with enough
access to make a urlfetch request and not have it refused for lack of
auth) could trick the server into returning more data than the requested
resource actually contained, with contents of memory leaking into the
extra.  I don't know whether it could contain anything predictable,
useful or "interesting".  If the range they asked for crossed an
allocation boundary, I think they'd probably just trigger a SEGV in the
process serving their connection, and be disconnected.

I don't know much about urlfetch in practice, or this implementation in
particular, so I too don't really know how exploitable this is.  The bug
just happened to cross my radar, so it got whacked.

Cheers,

ellie

On Tue, Dec 15, 2015, at 03:36 AM, Tim Champ via Info-cyrus wrote:
> Hello all.  Sorry about following up to my own email, but I think I
> understand the changes now to 2.4.18 in order to make it CVE
> compliant.  As best I can understand, if I apply the two commits from
> Ellie Timoney on 10/26/2015, 2.4.18 would be "secure" once recompiled.
> These two commits appear to be these:
>
> https://cyrus.foundation/cyrus-imapd/commit/?h=cyrus-imapd-2.4&id=538359e5a7c978e2f27c80124c8bd1282c7661a9
>
> https://cyrus.foundation/cyrus-imapd/commit/?h=cyrus-imapd-2.4&id=0142e98fa90f02a030f93469523ac64f91ae7a9f
>
> If someone can confirm that I'm correct on this, it would be very
> appreciated!  Thanks in advance.
>
> Tim
>
> On Mon, Dec 14, 2015 at 11:04 AM, Tim Champ  wrote:
>> Hello all.
>>
>> We're trying to sort through our path here with patching for the
>> CVE/commits that were released in 2.5.7, but also relevant to 2.4.18.
>> We're currently on 2.4 series, and I was wondering what the plans
>> were for a 2.4 release to address these security fixes.  While moving
>> to 2.5 is in the plans, I always despise a quick upgrade of anything
>> before major holiday periods!
>>
>> My other concern was that, honestly, I'm not all that sure what the
>> true risk and capability to exploit is for these bugs.  I've read the
>> CVE's, and associated discussions on the a few lists - but it hasn't
>> enlightened me as much as I've hoped.
>>
>> Any help, or answers, for either issue is appreciated.  Thanks!
>>
>> Tim
>>
>> --
>> Tim Champ Coordinator of Unix Infrastructure UMBC - Division of
>> Information Technology
>>
>
>
>
> --
> Tim Champ Coordinator of Unix Infrastructure UMBC - Division of
> Information Technology
> 
> Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info:
> http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: IPv6

[ellie timoney via Info-cyrus]

> Ellie, can you please fix the listen statement to accept correctly
> bracketed ipv6 and backport to at least 2.5 and 2.4, shouldn't be many
> changes in that code.

I don't think there's anything to fix here.  The code looks fine as is,
just docs missing.  Unless we want to explicitly *not* accepted
un-bracketed "ip6-address ':' port"?  But forbidding that seems silly,
considering it requires adding code to remove functionality.

This code hasn't changed since 2012, when it was refactored by 306099b. 
It's in the same state (modulo tabs/spaces change) on 2.5 and master
branches.  The 2.4 version is significantly different, as it didn't get
the refactor (but that doesn't appear to be a problem here).

I'm scouring the thread trying to figure out if there's even a problem
being reported (other than lack of docs) and I can't see it.  It seems
to have gone like:

Sebastian: How do I do this?
Various: It's not in docs, but try this...
Sebastian: I tried this and it worked

Sebastian, is there anything you tried that *didn't* work, and if so,
what happened?

Cheers,

ellie

On Mon, Mar 28, 2016, at 04:24 PM, Bron Gondwana wrote:
> Ellie, can you please fix the listen statement to accept correctly
> bracketed ipv6 and backport to at least 2.5 and 2.4, shouldn't be many
> changes in that code.
> 
> Nicola, can you check that we document how it's working now, and of
> course the fixed version :)
> 
> Bron (on phone in bush somewhere)
> 
> On Thu, Mar 24, 2016, at 05:10, Hajimu UMEMOTO via Info-cyrus wrote:
> > Hi,
> > 
> > > On Wed, 23 Mar 2016 17:43:30 +0100
> > > Sebastian Hagedorn  said:
> > 
> > >  * Parse the "listen" parameter as one of the forms:
> > >  *
> > >  * hostname
> > >  * hostname ':' port
> > >  * ipv4-address
> > >  * ipv4-address ':' port
> > >  * '[' ipv4-address ']'
> > >  * '[' ipv4-address ']' ':' port
> > >  * '[' ipv6-address ']'
> > >  * '[' ipv6-address ']' ':' port
> > 
> > The `ipv6-address ':' port' notation is also accepted.  However, it is
> > not mentioned even in the comment.
> > 
> > Hagedorn> Thank you! Perhaps that should go into the manpage of cyrus.conf?
> > 
> > I'm not sure but I think the bracket notation is recommended.
> > 
> > Sincerely,
> > 
> > --
> > Hajimu UMEMOTO
> > u...@mahoroba.org  u...@freebsd.org
> > http://www.mahoroba.org/~ume/
> > 
> > Cyrus Home Page: http://www.cyrusimap.org/
> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > To Unsubscribe:
> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> 
> -- 
>   Bron Gondwana
>   br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: IPv6

[ellie timoney via Info-cyrus]

Hi Sebastian,

> > Sebastian, is there anything you tried that *didn't* work, and if so,
> > what happened?
> 
> The only thing I tried that didn't work was to add a IPv6 listener and to 
> HUP the master process. The manpage for master reads (in my version):
> 
>Cyrus-master rereads its configuration file when it receives a 
> hangup signal, SIGHUP.   Services  and
>events  may be added, deleted or modified when the configuration 
> file is reread.  Any active services
>removed from the configuration file will be allowed to run until 
> completion.
> 
> From that it isn't obvious that some class of changes to cyrus.conf 
> apparently require a restart of the service. 

I've been looking through master/master.c to see what it actually does,
and it looks like it matches this documentation.

It does have some commentary in reread_conf() about recycling services
that have not been removed nor were newly added, which almost sounds as
if it might have this sort of effect... except that, digging into
add_service(), it will only reuse entries if their name, listen and
proto all match (which if you've changed one to IPv6, it won't), and
otherwise it will be added as a new service (and so reread_conf() will
treat it as a newly added service, not an existing one to recycle).

I'm pretty tired, and so probably not reading it as closely as I could
otherwise -- maybe there's a bug or subtlety I've missed -- but: it at
least /looks like it intends to/ do what the documentation says.  So
it's interesting that it didn't.

Cheers,

ellie

On Tue, Mar 29, 2016, at 11:45 PM, Sebastian Hagedorn wrote:
> Hi Ellie,
> 
> --On 29. März 2016 um 12:30:34 +1100 ellie timoney  
> wrote:
> 
> >> Ellie, can you please fix the listen statement to accept correctly
> >> bracketed ipv6 and backport to at least 2.5 and 2.4, shouldn't be many
> >> changes in that code.
> >
> > I don't think there's anything to fix here.  The code looks fine as is,
> > just docs missing.  Unless we want to explicitly *not* accepted
> > un-bracketed "ip6-address ':' port"?  But forbidding that seems silly,
> > considering it requires adding code to remove functionality.
> 
> I agree.
> 
> > This code hasn't changed since 2012, when it was refactored by 306099b.
> > It's in the same state (modulo tabs/spaces change) on 2.5 and master
> > branches.  The 2.4 version is significantly different, as it didn't get
> > the refactor (but that doesn't appear to be a problem here).
> >
> > I'm scouring the thread trying to figure out if there's even a problem
> > being reported (other than lack of docs) and I can't see it.  It seems
> > to have gone like:
> >
> > Sebastian: How do I do this?
> > Various: It's not in docs, but try this...
> > Sebastian: I tried this and it worked
> >
> > Sebastian, is there anything you tried that *didn't* work, and if so,
> > what happened?
> 
> The only thing I tried that didn't work was to add a IPv6 listener and to 
> HUP the master process. The manpage for master reads (in my version):
> 
>Cyrus-master rereads its configuration file when it receives a 
> hangup signal, SIGHUP.   Services  and
>events  may be added, deleted or modified when the configuration 
> file is reread.  Any active services
>removed from the configuration file will be allowed to run until 
> completion.
> 
> From that it isn't obvious that some class of changes to cyrus.conf 
> apparently require a restart of the service. So I'm mainly asking for 
> documentation fixes:
> 
> • clarify the allowed IPv6 address formats
> • clarify that SIGHUP isn't enough for all (which?) config changes
> 
> Thanks, Sebastian´
> -- 
> .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:.
>  .:.Regionales Rechenzentrum (RRZK).:.
>.:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:.
> Email had 1 attachment:
> + Attachment2
>   1k (application/pgp-signature)

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: IPv6

[ellie timoney via Info-cyrus]

Hi Sebastian,

> after thinking about it, I think it's like this: I added a service that
> was 
> configured to listen on a privileged port. But master has dropped 
> privileges by that point, so it can't add such a listener. I just 
> double-checked that I *can* add a listener at run-time if it's set up to 
> listen to a non-privileged port. Obvious in hindsight, but perhaps worth
> a 
> note anyway.

Oh yeah, of course.  I've added the following to man/master.8 for future
releases:

> Services added or modified to listen on a privileged port may not
> be able to bind the port, depending on your system configuration.
> In this case a full restart is needed.

I'm not entirely sold on the wording, but it's better than the nothing
we had.

"depending on your system configuration", because looking at the code,
if you are running Cyrus on Linux, and if you have compiled it with
--with-libcap=yes, then master will actually drop its privileges
*before* spawning any services at all, but in such a way that it
preserves the capability to bind privileged ports.  Assuming that this
actually works, then it should also be able to start up new/modified
services on privileged ports upon receipt of a SIGHUP.  So that's pretty
cool.  But it's not default: you must be on Linux, have libcap, and
explicitly request it at compile time.

Cheers,

ellie

On Tue, Apr 5, 2016, at 04:22 PM, Sebastian Hagedorn wrote:
> Hi Ellie,
> 
> --On 5. April 2016 um 14:33:46 +1000 ellie timoney  
> wrote:
> 
> >> > Sebastian, is there anything you tried that *didn't* work, and if so,
> >> > what happened?
> >>
> >> The only thing I tried that didn't work was to add a IPv6 listener and
> >> to  HUP the master process. The manpage for master reads (in my version):
> >>
> >>Cyrus-master rereads its configuration file when it receives a
> >> hangup signal, SIGHUP.   Services  and
> >>events  may be added, deleted or modified when the configuration
> >> file is reread.  Any active services
> >>removed from the configuration file will be allowed to run until
> >> completion.
> >>
> >> From that it isn't obvious that some class of changes to cyrus.conf
> >> apparently require a restart of the service.
> >
> > I've been looking through master/master.c to see what it actually does,
> > and it looks like it matches this documentation.
> >
> > It does have some commentary in reread_conf() about recycling services
> > that have not been removed nor were newly added, which almost sounds as
> > if it might have this sort of effect... except that, digging into
> > add_service(), it will only reuse entries if their name, listen and
> > proto all match (which if you've changed one to IPv6, it won't), and
> > otherwise it will be added as a new service (and so reread_conf() will
> > treat it as a newly added service, not an existing one to recycle).
> >
> > I'm pretty tired, and so probably not reading it as closely as I could
> > otherwise -- maybe there's a bug or subtlety I've missed -- but: it at
> > least /looks like it intends to/ do what the documentation says.  So
> > it's interesting that it didn't.
> 
> after thinking about it, I think it's like this: I added a service that
> was 
> configured to listen on a privileged port. But master has dropped 
> privileges by that point, so it can't add such a listener. I just 
> double-checked that I *can* add a listener at run-time if it's set up to 
> listen to a non-privileged port. Obvious in hindsight, but perhaps worth
> a 
> note anyway.
> 
> Cheers
> Sebastian
> -- 
> .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:.
>  .:.Regionales Rechenzentrum (RRZK).:.
>.:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:.
> Email had 1 attachment:
> + Attachment2
>   1k (application/pgp-signature)

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: @ in mailboxname: 2.4 vs. 2.5

[ellie timoney via Info-cyrus]

On Sat, Apr 9, 2016, at 08:04 AM, Wolfgang Breyha via Info-cyrus wrote:
> On 08/04/16 23:28, Wolfgang Breyha via Info-cyrus wrote:
> > Is it save to remove this limitation from mboxname.c?
> 
> https://bugzilla.cyrusimap.org/show_bug.cgi?id=3693
> 
> This patch was reverted in "some" releases, but obviously not in 2.5.
> Please do so.

Thanks for tracking this down, the patch has been reverted.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.0.0-beta2 released

[ellie timoney via Info-cyrus]

The Cyrus team is proud to announce the immediate availability of the
second beta from the Cyrus IMAP 3.0 series: 3.0.0-beta2.

As this is a beta release it is not considered stable for production
usage.  Interfaces, APIs, features, etc are likely to change.

Download URLs:

http://www.cyrusimap.org/releases/cyrus-imapd-3.0.0-beta2.tar.gz
http://www.cyrusimap.org/releases/cyrus-imapd-3.0.0-beta2.tar.gz.sig

ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-3.0.0-beta2.tar.gz
ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-3.0.0-beta2.tar.gz.sig

Please consult the release notes before upgrading to 3.0.0-beta2:

http://cyrusimap.org/imap/release-notes/3.0/x/3.0.0-beta2.html

And join us at https://git.cyrus.foundation/ to report issues, join in
the deliberations of new features for the next Cyrus IMAP release, and
to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

Ellie Timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: 2.5.7 caldav interoperability

[ellie timoney via Info-cyrus]

Thanks for tracking that down.  It's now on the cyrus-imapd-2.5 branch,
and will be in the next release.

Cheers,

ellie

On Fri, Apr 22, 2016, at 11:11 PM, Wolfgang Breyha via Info-cyrus wrote:
> rsto--- via Info-cyrus wrote on 21/04/16 22:51:
> > P.S.: I am based in Vienna as well, so feel free to reach out to me :)
> 
> Thanks for your offlist help. I think you pointed me in the right
> direction.
> 
> 2.5.7 announces VPOLL for all except iOS/8 devices. HEAD doesn't announce
> VPOLL at all
> 
> HEAD
> http_caldav.c:4520
> /* Apple clients don't like VPOLL */
> types &= ~CAL_COMP_VPOLL;
> vs. 2.5.7
> http_caldav.c:3512
> /* XXX  iOS/8 doesn't like VPOLL */
> if (hdr && strstr(hdr[0], "iOS/8")) types &= ~CAL_COMP_VPOLL;
> 
> May I ask to backport this change to 2.5.x? I patched it locally and can
> confirm that our iOS/7 and 9 devices are able to access the Default
> Calendar now.
> 
> Greetings, Wolfgang
> -- 
> Wolfgang Breyha  | http://www.blafasel.at/
> Vienna University Computer Center | Austria
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 2.5.8 released

[ellie timoney via Info-cyrus]

The Cyrus team is proud to announce the immediate availability of a new
version of Cyrus IMAP: 2.5.8

This is a bug fix release for the 2.5 series.

Download URLs:

  http://www.cyrusimap.org/releases/cyrus-imapd-2.5.8.tar.gz
  http://www.cyrusimap.org/releases/cyrus-imapd-2.5.8.tar.gz.sig

  ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-2.5.8.tar.gz
  ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-2.5.8.tar.gz.sig

Please consult the release notes before upgrading to 2.5.8:

  http://cyrusimap.org/imap/release-notes/2.5/x/2.5.8.html

And join us at https://git.cyrus.foundation/ to report issues, join in
the deliberations of new features for the next Cyrus IMAP release, and
to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

Ellie Timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: httpd/carddav from git sources

[ellie timoney via Info-cyrus]

Hi Johan,

What revision are these patches against?

The Makefile.am patch is unnecessary, and doesn't apply cleanly anyway,
so I suspect you're looking at an old revision.

I haven't studied the httpd patch in depth yet.

Cheers,

ellie

On Thu, Jun 23, 2016, at 12:59 AM, Johan Hattne via Info-cyrus wrote:
> Dear all;
> 
> To make carddav run from git sources
> (git://git.cyrusimap.org/cyrus-imapd) I had to apply attached tiny
> patches; the Makefile.am patch addresses a use-before-definition issue
> which causes automake-1.14.1 to fall over, the httpd patch ensures that
> the sasl_http_request_t structure which is set as a property of
> httpd_saslconn does not go out of scope before it is used (I suppose this
> behavior is a tad system-dependent).  Apologies if both these issues have
> already been addressed elsewhere.
> 
> // Best wishes; Johan
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> Email had 2 attachments:
> + imap-httpd.patch
>   1k (text/plain)
> + imap-makefile.patch
>   1k (text/plain)

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: httpd/carddav from git sources

[ellie timoney via Info-cyrus]

Hi Johan,

I don't know how your git repository got into that state...

>   $ git describe
>   cyrus-imapd-2.5-snapshot-autoconf-and-automake-3857-g4049dd2

This says that the nearest tag your git can find is
"cyrus-imapd-2.5-snapshot-autoconf-and-automake"(which is over four
years old), and that your branch is 3857 commits ahead of that tag (but
the "cyrus-imapd-2.5.0" release tag is only 1796 commits ahead, so why
didn't describe find that, instead?), and that the commit id of your
branch tip is "4049dd2" (which I don't have, presumably because it's
your local commit containing your changes).

You probably want to refetch entirely I suspect [but keep reading first]

>   * remote origin
> Fetch URL: git://git.cyrusimap.org/cyrus-imapd
> Push  URL: git://git.cyrusimap.org/cyrus-imapd
> HEAD branch: master

I think these URLs are wrong/old.  If you can even fetch from them
anymore, do they update?  Look at the dates on the recent commits -- are
they ancient?  ('git log --format=fuller' to see commit dates).

But anyway, we just moved our repositories to github last week, so
anything you get from your old origin is going to be stale now
regardless.  https://github.com/cyrusimap/cyrus-imapd is the github page
with the clone/fork/etc links.

> I got this all from
> https://cyrusimap.org/mediawiki/index.php/Contribute#Anonymous_GIT_Access;
> the text indicates that development is actually happening elsewhere, but
> it has never been clear to me what the relationship between the
> repositories is.

Okay, that mediawiki is ancient -- I actually thought it was gone
already -- so that explains the old info.   I guess you got there from
Google, because it hasn't been linked from cyrusimap.org for ages.

Anyway, I suspect once you set up the correct git remote, and fetch from
that, your build troubles will go away.

Cheers,

ellie

On Fri, Jun 24, 2016, at 11:33 PM, Johan Hattne wrote:
> Hi Ellie;
> 
>   $ git remote show origin | head -n 4
>   * remote origin
> Fetch URL: git://git.cyrusimap.org/cyrus-imapd
> Push  URL: git://git.cyrusimap.org/cyrus-imapd
> HEAD branch: master
>   $ git describe
>   cyrus-imapd-2.5-snapshot-autoconf-and-automake-3857-g4049dd2
> 
> I got this all from
> https://cyrusimap.org/mediawiki/index.php/Contribute#Anonymous_GIT_Access;
> the text indicates that development is actually happening elsewhere, but
> it has never been clear to me what the relationship between the
> repositories is.
> 
> // Best wishes; Johan
> 
> > On Jun 23, 2016, at 19:08, ellie timoney via Info-cyrus 
> >  wrote:
> > 
> > Hi Johan,
> > 
> > What revision are these patches against?
> > 
> > The Makefile.am patch is unnecessary, and doesn't apply cleanly anyway,
> > so I suspect you're looking at an old revision.
> > 
> > I haven't studied the httpd patch in depth yet.
> > 
> > Cheers,
> > 
> > ellie
> > 
> > On Thu, Jun 23, 2016, at 12:59 AM, Johan Hattne via Info-cyrus wrote:
> >> Dear all;
> >> 
> >> To make carddav run from git sources
> >> (git://git.cyrusimap.org/cyrus-imapd) I had to apply attached tiny
> >> patches; the Makefile.am patch addresses a use-before-definition issue
> >> which causes automake-1.14.1 to fall over, the httpd patch ensures that
> >> the sasl_http_request_t structure which is set as a property of
> >> httpd_saslconn does not go out of scope before it is used (I suppose this
> >> behavior is a tad system-dependent).  Apologies if both these issues have
> >> already been addressed elsewhere.
> >> 
> >> // Best wishes; Johan
> >> 
> >> 
> >> Cyrus Home Page: http://www.cyrusimap.org/
> >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> >> To Unsubscribe:
> >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> >> Email had 2 attachments:
> >> + imap-httpd.patch
> >>  1k (text/plain)
> >> + imap-makefile.patch
> >>  1k (text/plain)
> > 
> > Cyrus Home Page: http://www.cyrusimap.org/
> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > To Unsubscribe:
> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: httpd/carddav from git sources

[ellie timoney via Info-cyrus]

> >>Fetch URL: git://git.cyrusimap.org/cyrus-imapd
> >>    Push  URL: git://git.cyrusimap.org/cyrus-imapd
> >>HEAD branch: master
> >>  $ git describe
> >>  cyrus-imapd-2.5-snapshot-autoconf-and-automake-3857-g4049dd2
> >> 
> >> I got this all from
> >> https://cyrusimap.org/mediawiki/index.php/Contribute#Anonymous_GIT_Access;
> >> the text indicates that development is actually happening elsewhere, but
> >> it has never been clear to me what the relationship between the
> >> repositories is.
> >> 
> >> // Best wishes; Johan
> >> 
> >>> On Jun 23, 2016, at 19:08, ellie timoney via Info-cyrus 
> >>>  wrote:
> >>> 
> >>> Hi Johan,
> >>> 
> >>> What revision are these patches against?
> >>> 
> >>> The Makefile.am patch is unnecessary, and doesn't apply cleanly anyway,
> >>> so I suspect you're looking at an old revision.
> >>> 
> >>> I haven't studied the httpd patch in depth yet.
> >>> 
> >>> Cheers,
> >>> 
> >>> ellie
> >>> 
> >>> On Thu, Jun 23, 2016, at 12:59 AM, Johan Hattne via Info-cyrus wrote:
> >>>> Dear all;
> >>>> 
> >>>> To make carddav run from git sources
> >>>> (git://git.cyrusimap.org/cyrus-imapd) I had to apply attached tiny
> >>>> patches; the Makefile.am patch addresses a use-before-definition issue
> >>>> which causes automake-1.14.1 to fall over, the httpd patch ensures that
> >>>> the sasl_http_request_t structure which is set as a property of
> >>>> httpd_saslconn does not go out of scope before it is used (I suppose this
> >>>> behavior is a tad system-dependent).  Apologies if both these issues have
> >>>> already been addressed elsewhere.
> >>>> 
> >>>> // Best wishes; Johan
> >>>> 
> >>>> 
> >>>> Cyrus Home Page: http://www.cyrusimap.org/
> >>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> >>>> To Unsubscribe:
> >>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> >>>> Email had 2 attachments:
> >>>> + imap-httpd.patch
> >>>> 1k (text/plain)
> >>>> + imap-makefile.patch
> >>>> 1k (text/plain)
> >>> 
> >>> Cyrus Home Page: http://www.cyrusimap.org/
> >>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> >>> To Unsubscribe:
> >>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> >> 
> 

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.0.0-beta3 released

[ellie timoney via Info-cyrus]

The Cyrus team is proud to announce the immediate availability of the
third beta from the Cyrus IMAP 3.0 series: 3.0.0-beta3.

As this is a beta release it is not considered stable for production
usage.  Interfaces, APIs, features, etc are likely to change.

Download URLs:

http://www.cyrusimap.org/releases/cyrus-imapd-3.0.0-beta3.tar.gz
http://www.cyrusimap.org/releases/cyrus-imapd-3.0.0-beta3.tar.gz.sig

ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-3.0.0-beta3.tar.gz
ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-3.0.0-beta3.tar.gz.sig

Please consult the release notes before upgrading to 3.0.0-beta3:

http://cyrusimap.org/imap/release-notes/3.0/x/3.0.0-beta3.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to
report issues, join in the deliberations of new features for the next
Cyrus IMAP release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

Ellie Timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Huge performance problems after updating from 2.4 to 2.5.8

[ellie timoney via Info-cyrus]

On Fri, Jul 15, 2016, at 11:08 PM, Bron Gondwana via Info-cyrus wrote:
> 
> It's in the cyrus-imapd-2.5 branch on git, so it will be in 2.5.9.  I'll
> talk with Ellie on Monday about releasing that.  I hope to have a fix for
> the other issue as well.
> 
> Bron.

I was already planning on releasing 2.5.9 later this week or early next
:)

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 2.5.9 released

[ellie timoney via Info-cyrus]

The Cyrus team is proud to announce the immediate availability of a new
version of Cyrus IMAP: 2.5.9

This is a bug fix release for the 2.5 series.

Download URLs:

  http://www.cyrusimap.org/releases/cyrus-imapd-2.5.9.tar.gz
  http://www.cyrusimap.org/releases/cyrus-imapd-2.5.9.tar.gz.sig

  ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-2.5.9.tar.gz
  ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-2.5.9.tar.gz.sig

Please consult the release notes before upgrading to 2.5.9:

  http://cyrusimap.org/imap/release-notes/2.5/x/2.5.9.html

And join us at https://github.com/cyrusimap/cyrus-imapd to report
issues, join in the deliberations of new features for the next Cyrus
IMAP release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

Ellie Timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Problem moving between folders during 2.3.16 -> 2.5.9 upgrade

[ellie timoney via Info-cyrus]

I don't have a 2.3.x environment handy to try to reproduce this from the
outside in, so I'm starting at the error code and working outwards...

> > <1469384361 > >1469384362>a0080 NO Mailbox format corruption detected

"Mailbox format corruption detected" corresponds with
IMAP_MAILBOX_CHECKSUM

This error code is exclusively returned by functions in imap/mailbox.c,
for index/record crc mismatches.  Given that the mailbox is selectable
and such, I'm supposing that the whole-file/whole-record crc checks are
okay...

It looks like 2.3.16 had MAILBOX_MINOR_VERSION 10, and in particular
doesn't have the record->cache_crc field, which many of the
IMAP_MAILBOX_CHECKSUM checks are for.  When reading mailboxes < 12 in
newer Cyrus versions, this field is initialised to zero.

Some of the places that want to do a crc check against record->cache_crc
first make sure record->cache_crc is non-zero before doing so: 
cache_append_record(), mailbox_cacherecord().

But some don't, or at least don't appear to: mailbox_append_cache()
initially uses cache_append_record() (good), but then reads it straight
back in to ensure freshness -- and then having done so, appears to do an
unconditional check against record->cache_crc (dodgy?)

Actually that seems like all the spots this check happens -- so the fix
might be as simple as the attached patch?

Kenneth, are you able to try this out?

Cheers,

ellie

On Mon, Jul 25, 2016, at 02:02 PM, Bron Gondwana wrote:
> Thanks for reporting this.  Ellie, if you get a chance can you look at
> it?  I'm in the middle of the the FastMail security rollout right now, so
> I can't do anything today.
> 
> Bron.
> 
> On Mon, Jul 25, 2016, at 11:45, Kenneth Marshall wrote:
> > Hi,
> > 
> > I accidentally hijacked another thread. So here is a new message. I am
> > testing a Cyrus IMAP 2.3.16 to 2.5.9 upgrade and I am having problems
> > moving messages between folders before the 'reconstruct -V max' has been
> > run on the mailboxes. Here is the error from the IMAP client:
> > 
> > imap_copy_messages [a0080 NO Mailbox format corruption detected]?
> > 
> > and the details from the IMAP logs:
> > 
> > <1469384341 > >1469384341>a0077 OK Completed
> > <1469384354 > >1469384355>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> > * LIST (\HasChildren) "/" DSPAM_notspam
> > * LIST (\HasChildren) "/" DSPAM_spam
> > * LIST (\HasNoChildren) "/" Drafts
> > * LIST (\HasChildren) "/" Mail
> > * LIST (\HasNoChildren) "/" Sent
> > * LIST (\HasNoChildren) "/" Spam
> > * LIST (\HasNoChildren) "/" Trash
> > a0078 OK Completed (0.240 secs 356 calls)
> > <1469384361 > >1469384361>* STATUS Drafts (UIDVALIDITY 1345619846)
> > a0079 OK Completed
> > <1469384361 > >1469384362>a0080 NO Mailbox format corruption detected
> > <1469384370 > >1469384370>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> > * LIST (\HasChildren) "/" DSPAM_notspam
> > * LIST (\HasChildren) "/" DSPAM_spam
> > * LIST (\HasNoChildren) "/" Drafts
> > * LIST (\HasChildren) "/" Mail
> > * LIST (\HasNoChildren) "/" Sent
> > * LIST (\HasNoChildren) "/" Spam
> > * LIST (\HasNoChildren) "/" Trash
> > a0081 OK Completed (0.230 secs 356 calls)
> > <1469384372 > a0083 MYRIGHTS "Drafts"
> > a0084 SELECT "Drafts"
> > >1469384372>a0082 OK Completed
> > >1469384372>* MYRIGHTS Drafts lrswipkxtecda
> > a0083 OK Completed
> > >1469384372>* 0 EXISTS
> > * 0 RECENT
> > * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old)
> > * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old 
> > \*)] Ok
> > * OK [UIDVALIDITY 1345619846] Ok
> > * OK [UIDNEXT 6] Ok
> > * OK [HIGHESTMODSEQ 1] Ok
> > * OK [URLMECH INTERNAL] Ok
> > * OK [ANNOTATIONS 65536] Ok
> > a0084 OK [READ-WRITE] Completed
> > >1469384355>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> > * LIST (\HasChildren) "/" DSPAM_notspam
> > * LIST (\HasChildren) "/" DSPAM_spam
> > * LIST (\HasNoChildren) "/" Drafts
> > * LIST (\HasChildren) "/" Mail
> > * LIST (\HasNoChildren) "/" Sent
> > * LIST (\HasNoChildren) "/" Spam
> > * LIST (\HasNoChildren) "/" Trash
> > a0078 OK Completed (0.240 secs 356 calls)
> > <1469384361 > >1469384361>* STATUS Drafts (UIDVALIDITY 1345619846)
> > a0079 OK Completed
> > <1469384361 > >1469384362>a0080 NO Mailbox format corruption detected
> > <1469384370 > >1469384370>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> > * LIST (\HasChildren) "/" DSPAM_notspam
> > * LIST (\HasChildren) "/" DSPAM_spam
> > * LIST (\HasNoChildren) "/" Drafts
> > * LIST (\HasChildren) "/" Mail
> > * LIST (\HasNoChildren) "/" Sent
> > * LIST (\HasNoChildren) "/" Spam
> > * LIST (\HasNoChildren) "/" Trash
> > a0081 OK Completed (0.230 secs 356 calls)
> > <1469384372 > a0083 MYRIGHTS "Drafts"
> > a0084 SELECT "Drafts"
> > >1469384372>a0082 OK Completed
> > >1469384372>* MYRIGHTS Drafts lrswipkxtecda
> > a0083 OK Completed
> > >1469384372>* 0 EXISTS
> > * 0 RECENT
> > * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old)
> > * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Se

Re: Problem moving between folders during 2.3.16 -> 2.5.9 upgrade

[ellie timoney via Info-cyrus]

On Mon, Jul 25, 2016, at 10:53 PM, Kenneth Marshall wrote:
> On Mon, Jul 25, 2016 at 03:53:43PM +1000, ellie timoney wrote:
> > I don't have a 2.3.x environment handy to try to reproduce this from the
> > outside in, so I'm starting at the error code and working outwards...
> > 
> > > > <1469384361 > > > >1469384362>a0080 NO Mailbox format corruption detected
> > 
> > "Mailbox format corruption detected" corresponds with
> > IMAP_MAILBOX_CHECKSUM
> > 
> > This error code is exclusively returned by functions in imap/mailbox.c,
> > for index/record crc mismatches.  Given that the mailbox is selectable
> > and such, I'm supposing that the whole-file/whole-record crc checks are
> > okay...
> > 
> > It looks like 2.3.16 had MAILBOX_MINOR_VERSION 10, and in particular
> > doesn't have the record->cache_crc field, which many of the
> > IMAP_MAILBOX_CHECKSUM checks are for.  When reading mailboxes < 12 in
> > newer Cyrus versions, this field is initialised to zero.
> > 
> > Some of the places that want to do a crc check against record->cache_crc
> > first make sure record->cache_crc is non-zero before doing so: 
> > cache_append_record(), mailbox_cacherecord().
> > 
> > But some don't, or at least don't appear to: mailbox_append_cache()
> > initially uses cache_append_record() (good), but then reads it straight
> > back in to ensure freshness -- and then having done so, appears to do an
> > unconditional check against record->cache_crc (dodgy?)
> > 
> > Actually that seems like all the spots this check happens -- so the fix
> > might be as simple as the attached patch?
> > 
> > Kenneth, are you able to try this out?
> > 
> > Cheers,
> > 
> > ellie
> > 
> 
> Hi Ellie,
> 
> That fixes the problem. Thank you so much for the quick response.
> 
> Regards,
> Ken

That's great to hear, thanks for trying it out.  This patch will be
included in 2.5.10 when it comes out.

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Patch against 2.5.9 for OpenSSL in lib64, pkg-config

[ellie timoney via Info-cyrus]

Hi Jason,

That sounds pretty reasonable, thanks for the patch.  I'll try to get it
tested and merged this week.

Cheers,

ellie

On Tue, Jul 26, 2016, at 06:25 AM, Jason Englander via Info-cyrus wrote:
> If you have a source installed OpenSSL under /usr/local/ssl on a 64-bit 
> system, you probably have /usr/local/ssl/lib64 instead of 
> /usr/local/ssl/lib
> 
> In order for Cyrus IMAP configure to find the libraries, even passing 
> --with-openssl=/usr/local/ssl to configure, you would need to use 
> LDFLAGS=-L/usr/local/ssl/lib64 and also add /usr/local/ssl/lib64 to LIBS 
> in perl/imap/Makefile.PL and perl/sieve/managesieve/Makefile.PL
> 
> I am no autoconf guru by any means, but I made two changes to
> configure.ac 
> that enable (after 'autoreconf -f -i') you to use either 
> --with-ssl=/usr/local/ssl and have that check both lib and lib64 under 
> there, and also if you use --with-ssl with no path, for it to try 
> pkg-config before system defaults (e.g. /usr/include and /usr/lib).
> 
> If you go the pkg-config way, as long as your PKG_CONFIG_PATH variable 
> (possibly set in /etc/profile) includes /usr/local/ssl/lib64/pkgconfig or 
> wherever yours is, it should work.  I believe they added support for it
> to 
> OpenSSL > 0.9.6
> 
> So far only tested on one Slackware Linux machine with source Cyrus IMAP 
> 2.5.9 and source OpenSSL 1.0.2h
> 
> 
> --- configure.ac.orig   2016-07-21 22:01:21.0 -0400
> +++ configure.ac2016-07-25 14:25:55.432452824 -0400
> @@ -984,10 +984,16 @@
>   ;;
>   x|xyes)
>   # default args or --with-openssl
> -# try to enable, search in default system directories
>   #
> -ssl_cppflags=
> -ssl_ldflags=
> +# Try pkg-config - OpenSSL >= 0.9.6 has openssl.pc
> +PKG_CHECK_MODULES(OPENSSL, openssl)
> +ssl_cppflags="$OPENSSL_CFLAGS"
> +ssl_ldflags="$OPENSSL_LIBS"
> +if test -n "$ssl_cppflags" -a -n "$ssl_ldflags"; then
> +  with_ssl=yes
> +fi
> +#
> +# If pkg-config doesn't work, search in default system directories
>   ;;
>   *)
>   # --with-openssl=DIR
> @@ -995,7 +1001,11 @@
>   if test -d "$with_ssl"; then
>  ssl_cppflags="-I${with_ssl}/include"
>  ssl_ldflags=
> -   CMU_ADD_LIBPATH_TO(${with_ssl}/lib, ssl_ldflags)
> +   if test -d "${with_ssl}/lib64"; then
> +   CMU_ADD_LIBPATH_TO(${with_ssl}/lib64, ssl_ldflags)
> +   else
> +   CMU_ADD_LIBPATH_TO(${with_ssl}/lib, ssl_ldflags)
> +   fi
>  with_ssl=yes
>   else
>  AC_WARN([Disabling OpenSSL - no include files found])
> 
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Patch against 2.5.9 for OpenSSL in lib64, pkg-config

[ellie timoney via Info-cyrus]

This is now on the cyrus-imapd-2.5 and master branches, thanks :)

I ended up tweaking it a little bit:

> > +if test -n "$ssl_cppflags" -a -n "$ssl_ldflags"; then
> > +  with_ssl=yes
> > +fi

with_ssl=yes is now set by the PKG_CONFIG_MODULES macro's
action-if-found.  This seems like it will be more reliable in the case
where pkg-config successfully finds the library, but determines that it
doesn't actually need any special cflags and/or ldflags.

The committed version is here: 
https://github.com/cyrusimap/cyrus-imapd/commit/e84d2f58afac0ff2e7d71bb0efb2a5ad7cb04a93

Cheers,

ellie

On Tue, Jul 26, 2016, at 09:54 AM, ellie timoney via Info-cyrus wrote:
> Hi Jason,
> 
> That sounds pretty reasonable, thanks for the patch.  I'll try to get it
> tested and merged this week.
> 
> Cheers,
> 
> ellie
> 
> On Tue, Jul 26, 2016, at 06:25 AM, Jason Englander via Info-cyrus wrote:
> > If you have a source installed OpenSSL under /usr/local/ssl on a 64-bit 
> > system, you probably have /usr/local/ssl/lib64 instead of 
> > /usr/local/ssl/lib
> > 
> > In order for Cyrus IMAP configure to find the libraries, even passing 
> > --with-openssl=/usr/local/ssl to configure, you would need to use 
> > LDFLAGS=-L/usr/local/ssl/lib64 and also add /usr/local/ssl/lib64 to LIBS 
> > in perl/imap/Makefile.PL and perl/sieve/managesieve/Makefile.PL
> > 
> > I am no autoconf guru by any means, but I made two changes to
> > configure.ac 
> > that enable (after 'autoreconf -f -i') you to use either 
> > --with-ssl=/usr/local/ssl and have that check both lib and lib64 under 
> > there, and also if you use --with-ssl with no path, for it to try 
> > pkg-config before system defaults (e.g. /usr/include and /usr/lib).
> > 
> > If you go the pkg-config way, as long as your PKG_CONFIG_PATH variable 
> > (possibly set in /etc/profile) includes /usr/local/ssl/lib64/pkgconfig or 
> > wherever yours is, it should work.  I believe they added support for it
> > to 
> > OpenSSL > 0.9.6
> > 
> > So far only tested on one Slackware Linux machine with source Cyrus IMAP 
> > 2.5.9 and source OpenSSL 1.0.2h
> > 
> > 
> > --- configure.ac.orig   2016-07-21 22:01:21.0 -0400
> > +++ configure.ac2016-07-25 14:25:55.432452824 -0400
> > @@ -984,10 +984,16 @@
> >   ;;
> >   x|xyes)
> >   # default args or --with-openssl
> > -# try to enable, search in default system directories
> >   #
> > -ssl_cppflags=
> > -ssl_ldflags=
> > +# Try pkg-config - OpenSSL >= 0.9.6 has openssl.pc
> > +PKG_CHECK_MODULES(OPENSSL, openssl)
> > +ssl_cppflags="$OPENSSL_CFLAGS"
> > +ssl_ldflags="$OPENSSL_LIBS"
> > +if test -n "$ssl_cppflags" -a -n "$ssl_ldflags"; then
> > +  with_ssl=yes
> > +fi
> > +#
> > +# If pkg-config doesn't work, search in default system directories
> >   ;;
> >   *)
> >   # --with-openssl=DIR
> > @@ -995,7 +1001,11 @@
> >   if test -d "$with_ssl"; then
> >  ssl_cppflags="-I${with_ssl}/include"
> >  ssl_ldflags=
> > -   CMU_ADD_LIBPATH_TO(${with_ssl}/lib, ssl_ldflags)
> > +   if test -d "${with_ssl}/lib64"; then
> > +   CMU_ADD_LIBPATH_TO(${with_ssl}/lib64, ssl_ldflags)
> > +   else
> > +   CMU_ADD_LIBPATH_TO(${with_ssl}/lib, ssl_ldflags)
> > +   fi
> >  with_ssl=yes
> >   else
> >  AC_WARN([Disabling OpenSSL - no include files found])
> > 
> > 
> > 
> > Cyrus Home Page: http://www.cyrusimap.org/
> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > To Unsubscribe:
> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: imapd dumps core on APPEND URL with invalid section

[ellie timoney via Info-cyrus]

The master branch has a commit to address this exact (like, literally
"TEXT.MIME") issue:

https://github.com/cyrusimap/cyrus-imapd/commit/f7b95b93

I've just cherry-picked it onto the cyrus-imapd-2.4 branch and it
applied cleanly first time, so it's upstream now.  (It's already in 2.5,
and has been since the 2.5.0 release.)

Though your email raises another question -- what do we want to do about
other junk section values?

On Sat, Jul 30, 2016, at 11:18 PM, Edda via Info-cyrus wrote:
> Hi,
> 
> we get core dumps of imapd on commands like this:
> 
> A7 APPEND "INBOX/Junk E-mail" () "29-Jul-2016 07:17:38 +" CATENATE 
> (URL "/INBOX/;uid=44335/;section=TEXT.MIME" URL 
> "/INBOX/;uid=44335/;section=TEXT")
> Connection closed by foreign host.
> 
> Tested with:
> Cyrus 2.4.18 on Solaris 11
> Cyrus 2.4.17 on CentOS 7
> 
> section=MIME instead of section=TEXT.MIME (which I think is not a valid 
> section) works for the message:
> 
> A7 APPEND "INBOX/Junk E-mail" () "29-Jul-2016 07:17:38 +" CATENATE 
> (URL "/INBOX/;uid=44335/;section=MIME" URL 
> "/INBOX/;uid=44335/;section=TEXT")
> A7 OK [APPENDUID 1469792687 169] Completed
> 
> To illustrate the issue we produced core dumps with some nonsense 
> sections, example:
> 
> A7 APPEND "INBOX/Junk E-mail" () "29-Jul-2016 07:17:38 +" CATENATE 
> (URL "/INBOX/;uid=44335/;section=CATS_AND_DOGS" URL 
> "/INBOX/;uid=44335/;section=TEXT")
> Connection closed by foreign host.
> 
> 
> This is the stacktrace of the corresponding core file (produced with 
> Cyrus 2.4.17):
> 
> (gdb) bt full
> #0  __bswap_32 (__bsx= address 0x7f6211818650>) at /usr/include/bits/byteswap.h:47
> No locals.
> #1  index_urlfetch (state=, msgno=, 
> params=0, section=, start_octet=0, octet_count=0,
>  pout=0x7f612b939610, outsize=0x7fff44d3ce80) at index.c:2785
>  num_parts = 2
>  p = 0x7f612b9292fb "CATS_AND_DOGS"
>  data = 0x7f6129f41000 
>  msg_base = 0x7f6129f41000 
>  msg_size = 4812
>  cacheitem = 0x7f6211818650   bounds>
>  fetchmime = 1
>  domain = 0
>  size = 4812
>  skip = 1697477688
>  n = 
>  r = 
>  decbuf = 0x0
>  mailbox = 0x7f612b929878
>  im = 0x7f612b92a7b0
> […]
> (gdb) where
> #0  __bswap_32 (__bsx= address 0x7f62a7ebe650>) at /usr/include/bits/byteswap.h:47
> #1  index_urlfetch (state=, msgno=, 
> params=0, section=, start_octet=0, octet_count=0, 
> pout=0x7f61c12d4600, outsize=0x7ffcec9b1fc0)
>  at index.c:2785
> #2  0x7f61c06d0277 in cmd_append (tag=, 
> name=, cur_name=) at imapd.c:3121
> #3  0x7f61c06d5f2c in cmdloop () at imapd.c:1279
> #4  0x7f61c06d7759 in service_main (argc=, 
> argv=, envp=) at imapd.c:946
> #5  0x7f61c06c0875 in main (argc=, argv= out>, envp=0x7ffcec9b7a88) at service.c:582
> 
> 
> I don’t know where to fix it best in order to get BADURL or something 
> instead of a core dump, so any help would be highly appreciated.
> 
> Regards,
> Edda
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: sync_client errors out after 2.3.16 -> 2.5.9 upgrade

[ellie timoney via Info-cyrus]

Thanks for passing these reports on!

Initial impression, I don't think this one is as straightforward as the
last one was, unfortunately. :(

Here's the chunk of code that produces those "guid mismatch" SYNCERRORs:
 https://git.io/v6JWK
(wait a moment for it to load and it will jump to the lines I've
highlighted)

We can see from the order in which those mismatching guids are reported
that, for these messages, the "master" server (i.e. the server on which
sync_client is running) has a guid with a value, whereas the "replica"
server (i.e the other end) has a guid with a null value.  And then the
assertion fails in sync_client -- i.e. the "master" server.  So this is
the split-brain codepath detecting the guid mismatch, understanding that
as "these are two distinct messages that have accidentally wound up with
the same uid at each end", and so rewriting both messages with whole new
uids -- which then fails, because one of the guids is zero. (Note that
the side that *has guids* is trying to write a message without a guid.)

One possible solution that comes to mind is to not treat it as a
split-brain situation if one of the mismatching guids is null -- but I
don't understand the ramifications of that.  I have not yet looked at
the master branch to see if it has improvements in this area.

The other aspect: here's the code that reads the guid from the index
record in the first place: https://git.io/v6JlR

If I'm reading that correctly, we expect version 10 mailboxes to have a
guid in them.  I got the impression from the last problem we looked at
that your mailboxes were version 10, but maybe I was wrong and they're
even older?  Can you check the version of the cyrus.index for
user.robot, specifically on the server you were trying to replicate to? 
There was a thread about how to check that on this list the other day.

I'm not sure whether the assert in mailbox_append_index_record is overly
aggressive or not.  We do expect new records, constructed on the current
version, to have a guid in them.  Maybe, rather than asserting for that,
it should try recalculate the guid if the provided one is null?  Or
maybe something further upstream should have done this?

mailbox_append_index_record eventually calls
mailbox_inbox_record_to_buf, which is responsible for producing a
correct output for the index version it's actually writing to.  It's
here that the outward version detection exists, and it looks consistent
with the inward code linked above: the guid is omitted if the mailbox
version is < 10. (But it still expects it to have been set in the record
it was handed -- it just sometimes doesn't use it.)

Bron, your input would be really appreciated here :)

ellie

On Tue, Aug 2, 2016, at 07:56 AM, Kenneth Marshall via Info-cyrus wrote:
> Hi Cyrus Developers,
>
> Thank you for your patch to address the folder move problem between
> un-reconstructed mailboxes after the 2.3.16 -> 2.5.9 upgrade. I am
> not sure, but it looks like there may be another overly aggressive
> check. I keep getting these fatal errors from sync_client:
>
> Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch
> user.robot 1696 (56412de8678bfb53f6cdb5fe4502031af5484056
> )
> Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch
> user.robot 1697 (1b0024218a4419973b83ae3e84ac7133a4ab7d40
> )
> Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch
> user.robot 1698 (f17084425d83bccb28a4dfa195846c7ef88c7567
> )
> Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch
> user.robot 1699 (7a751e41e1d3a58e541298ab724be4c29d96e49d
> )
> Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch
> user.robot 1700 (724a013d0ae97d27a1da33832487df1719681659
> )
> Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: Fatal error: Internal
> error: assertion failed: imap/mailbox.c: 2850:
> !message_guid_isnull(&record->guid)
>
> And then the sync_client has to be run manually and if lucky, it will
> process the full log successfully. I was looking in imap/mailbox.c
> and it looks like the assert at line 2850 may need a similar override
> for non-upgraded folders:
>
> ---imap/mailbox.c---
> /* append a single message to a mailbox - also updates everything
>  * automatically.  These two functions are the ONLY way to modify
>  * the contents or tracking fields of a message */
> EXPORTED int mailbox_append_index_record(struct mailbox *mailbox,
> struct index_record *record)
> {
> indexbuffer_t ibuf;
> unsigned char *buf = ibuf.buf;
> size_t offset;
> int r;
> int n;
> struct utimbuf settime;
> uint32_t recno;
>
> assert(mailbox_index_islocked(mailbox, 1));
>
> /* Append MUST be a higher UID than a

Re: sync_client errors out after 2.3.16 -> 2.5.9 upgrade

[ellie timoney via Info-cyrus]

I've been under the impression that Ken's mailboxes were version 10,
which seems like they *should* have guids in them.   If this is the
case, then it's interesting that the replica is coming up with zeroed
ones.

If his mailboxes are older than version 10, then it all makes sense,
nothing to see here... though it would be good if the sync code detected
mailboxes being too old to replicate and reported that, instead of
naively trying to sync them and crashing out on the guid check.

I guess it's plausible that version 10 mailboxes had a field in the
index for the guid, but that it wasn't necessarily populated?  That
might complicate things?  I don't know the history here.

On Tue, Aug 2, 2016, at 10:01 AM, Bron Gondwana via Info-cyrus wrote:
> You can't sync a mailbox without GUID for messages.  You need to upgrade
> the mailboxes before you can use them for replication.  The GUID is used
> for replication - if we allowed zero GUIDs, then every message would
> deduplicate to the same message!
> 
> On Tue, Aug 2, 2016, at 07:56, Kenneth Marshall via Info-cyrus wrote:
> > Hi Cyrus Developers,
> > 
> > Thank you for your patch to address the folder move problem between
> > un-reconstructed mailboxes after the 2.3.16 -> 2.5.9 upgrade. I am
> > not sure, but it looks like there may be another overly aggressive
> > check. I keep getting these fatal errors from sync_client:
> > 
> > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch 
> > user.robot 1696 (56412de8678bfb53f6cdb5fe4502031af5484056 
> > )
> > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch 
> > user.robot 1697 (1b0024218a4419973b83ae3e84ac7133a4ab7d40 
> > )
> > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch 
> > user.robot 1698 (f17084425d83bccb28a4dfa195846c7ef88c7567 
> > )
> > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch 
> > user.robot 1699 (7a751e41e1d3a58e541298ab724be4c29d96e49d 
> > )
> > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch 
> > user.robot 1700 (724a013d0ae97d27a1da33832487df1719681659 
> > )
> > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: Fatal error: Internal 
> > error: assertion failed: imap/mailbox.c: 2850: 
> > !message_guid_isnull(&record->guid)
> > 
> > And then the sync_client has to be run manually and if lucky, it will
> > process the full log successfully. I was looking in imap/mailbox.c
> > and it looks like the assert at line 2850 may need a similar override
> > for non-upgraded folders:
> > 
> > ---imap/mailbox.c---
> > /* append a single message to a mailbox - also updates everything
> >  * automatically.  These two functions are the ONLY way to modify
> >  * the contents or tracking fields of a message */
> > EXPORTED int mailbox_append_index_record(struct mailbox *mailbox,
> > struct index_record *record)
> > {
> > indexbuffer_t ibuf;
> > unsigned char *buf = ibuf.buf;
> > size_t offset;
> > int r;
> > int n;
> > struct utimbuf settime;
> > uint32_t recno;
> > 
> > assert(mailbox_index_islocked(mailbox, 1));
> > 
> > /* Append MUST be a higher UID than any we've yet seen */
> > assert(record->uid > mailbox->i.last_uid)
> > 
> > /* Append MUST have a message with data */
> > assert(record->size);
> > 
> > =>/* GUID must not be null */
> > =>assert(!message_guid_isnull(&record->guid));
> > 
> > /* belt AND suspenders - check the previous record too */
> > if (mailbox->i.num_records) {
> > struct index_record prev;
> > 
> > ---imap/mailbox.c---
> > 
> > What do you think?
> > 
> > Regards,
> > Ken
> > 
> > Cyrus Home Page: http://www.cyrusimap.org/
> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > To Unsubscribe:
> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> 
> -- 
>   Bron Gondwana
>   br...@fastmail.fm
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: sync_client errors out after 2.3.16 -> 2.5.9 upgrade

[ellie timoney via Info-cyrus]

It's fiddly to tell from git history -- there's a right mess somewhere
during the 2.3 series where it looks like the branch got rebased, or
trashed and reimported, or something like that -- the earlier 2.3
release tags are on commits that are not ancestors of the later ones.  I
keep tripping over this when tracking down this sort of thing...

Anyway, this is the commit that bumped the mailbox version from 9-10:
https://github.com/cyrusimap/cyrus-imapd/commit/3fcdb14

And this is its alternate-universe friend:
https://github.com/cyrusimap/cyrus-imapd/commit/e462a8b

... which looks like it was first released in 2.3.10.

This commit bumped it from 6 to 9:
https://github.com/cyrusimap/cyrus-imapd/commit/9c5e3ee

... which looks like it was first released in 2.3.8.

So given you started at 2.3.2, your oldest mailboxes must have been
upgraded a bunch of times by now(!)

On Tue, Aug 2, 2016, at 12:22 PM, Kenneth Marshall via Info-cyrus wrote:
> Hi Bron and Ellie,
> 
> I think that is what happened. Replication was working and the
> 'reconstruct -G'
> was optional in terms of working, so some of the oldest mailboxes could
> be in
> that state. I will check the versions and see if there are any older than
> 10,
> but we started with version 2.3.2 so unless the version changed within
> the
> patch series, they will all be 10.
> 
> Regards,
> Ken
> 
> On Tue, Aug 02, 2016 at 11:39:01AM +1000, Bron Gondwana via Info-cyrus
> wrote:
> > Yeah, could be.  In that case then the mailbox needs a reconstruct -G first 
> > :(
> > Version 10 mailboxes have a GUID space available, but I guess they could
> > wind up zero depending on how they got upgraded in the past.
> > 
> > Bron.
> > 
> > On Tue, Aug 2, 2016, at 11:26, ellie timoney via Info-cyrus wrote:
> > > I've been under the impression that Ken's mailboxes were version 10,
> > > which seems like they *should* have guids in them.   If this is the
> > > case, then it's interesting that the replica is coming up with zeroed
> > > ones.
> > > 
> > > If his mailboxes are older than version 10, then it all makes sense,
> > > nothing to see here... though it would be good if the sync code detected
> > > mailboxes being too old to replicate and reported that, instead of
> > > naively trying to sync them and crashing out on the guid check.
> > > 
> > > I guess it's plausible that version 10 mailboxes had a field in the
> > > index for the guid, but that it wasn't necessarily populated?  That
> > > might complicate things?  I don't know the history here.
> > > 
> > > On Tue, Aug 2, 2016, at 10:01 AM, Bron Gondwana via Info-cyrus wrote:
> > > > You can't sync a mailbox without GUID for messages.  You need to upgrade
> > > > the mailboxes before you can use them for replication.  The GUID is used
> > > > for replication - if we allowed zero GUIDs, then every message would
> > > > deduplicate to the same message!
> > > > 
> > > > On Tue, Aug 2, 2016, at 07:56, Kenneth Marshall via Info-cyrus wrote:
> > > > > Hi Cyrus Developers,
> > > > > 
> > > > > Thank you for your patch to address the folder move problem between
> > > > > un-reconstructed mailboxes after the 2.3.16 -> 2.5.9 upgrade. I am
> > > > > not sure, but it looks like there may be another overly aggressive
> > > > > check. I keep getting these fatal errors from sync_client:
> > > > > 
> > > > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid 
> > > > > mismatch user.robot 1696 (56412de8678bfb53f6cdb5fe4502031af5484056 
> > > > > )
> > > > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid 
> > > > > mismatch user.robot 1697 (1b0024218a4419973b83ae3e84ac7133a4ab7d40 
> > > > > )
> > > > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid 
> > > > > mismatch user.robot 1698 (f17084425d83bccb28a4dfa195846c7ef88c7567 
> > > > > )
> > > > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid 
> > > > > mismatch user.robot 1699 (7a751e41e1d3a58e541298ab724be4c29d96e49d 
> > > > > )
> > > > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid 
> > > > > mismatch user.robot 1700 (724a013d0ae97d27a1da33832487df1719681659 
> &g

Re: sync_client errors out after 2.3.16 -> 2.5.9 upgrade

[ellie timoney via Info-cyrus]

On Tue, Aug 2, 2016, at 02:14 PM, Kenneth Marshall via Info-cyrus wrote:
> On Tue, Aug 02, 2016 at 12:34:13PM +1000, Bron Gondwana wrote:
> > 2.3.2 wasn't version 10, it was version 7!  It would have upgraded through 
> > 8, 9, 10 - and maybe you needed to reconstruct to get GUIDs for those 
> > versions - that sounds familiar.
> > 
> > Bron.
> > 
> 
> Hi Bron,
> 
> Is there a way to identify mailboxes with no GUID? Then I could target
> them
> for reconstruction first.
> 
> Regards,
> Ken

Hi Ken,

I don't suppose you still have mailboxes with no GUID?  Or have you
already found and "reconstruct -G"ed everything?

I've attached a patch that does two things, which are kind of the same
thing:

* If the mailbox on either end of the replication is of a version < 10,
then the operation will fail cleanly and early with an "Operation is not
supported on mailbox" error (rather than trying to do the replication,
then crashing out like it currently does) -- though I don't believe this
case will affect you, as I believe your mailboxes are all at least
version 10.

* If the mailbox on either end of the replication contains any index
records that do not have a GUID set, then the same will occur, and (in
addition to the above error) a warning will be logged to syslog like:
: missing guid for record  -- needs 'reconstruct
-G'?

So with this patch, you should be able to avoid these crashes, and also
identify mailboxes that need "reconstruct -G" just by checking for the
warning in syslog.

Are you able to try this out?

Cheers,

ellie
diff --git a/imap/sync_client.c b/imap/sync_client.c
index 2663301..8408380 100644
--- a/imap/sync_client.c
+++ b/imap/sync_client.c
@@ -212,6 +212,7 @@ static int find_reserve_all(struct sync_name_list *mboxname_list,
 	 * USE the value... the whole "add to master folders" actually
 	 * looks a bit pointless... */
 	r = mailbox_open_iwl(mbox->name, &mailbox);
+	if (!r) r = sync_mailbox_version_check(&mailbox);
 
 	/* Quietly skip over folders which have been deleted since we
 	   started working (but record fact in case caller cares) */
@@ -1298,6 +1299,7 @@ static int mailbox_full_update(const char *mboxname)
 
 /* we'll be updating it! */
 r = mailbox_open_iwl(mboxname, &mailbox);
+if (!r) r = sync_mailbox_version_check(&mailbox);
 if (r) goto done;
 
 /* if local UIDVALIDITY is lower, copy from remote, otherwise
@@ -1468,6 +1470,7 @@ static int update_mailbox_once(struct sync_folder *local,
 annotate_state_t *astate = NULL;
 
 r = mailbox_open_iwl(local->name, &mailbox);
+if (!r) r = sync_mailbox_version_check(&mailbox);
 if (r == IMAP_MAILBOX_NONEXISTENT) {
 	/* been deleted in the meanwhile... it will get picked up by the
 	 * delete call later */
@@ -1946,6 +1949,7 @@ static int do_mailbox_info(void *rock,
 /* XXX - check for deleted? */
 
 r = mailbox_open_irl(name, &mailbox);
+if (!r) r = sync_mailbox_version_check(&mailbox);
 /* doesn't exist?  Probably not finished creating or removing yet */
 if (r == IMAP_MAILBOX_NONEXISTENT) {
 	r = 0;
diff --git a/imap/sync_server.c b/imap/sync_server.c
index c49df9f..51daf94 100644
--- a/imap/sync_server.c
+++ b/imap/sync_server.c
@@ -1052,7 +1052,7 @@ static void reserve_folder(const char *part, const char *mboxname,
 
 /* Open and lock mailbox */
 r = mailbox_open_irl(mboxname, &mailbox);
-
+if (!r) r = sync_mailbox_version_check(&mailbox);
 if (r) return;
 
 for (recno = 1; recno <= mailbox->i.num_records; recno++) {
@@ -1437,6 +1437,7 @@ static int do_mailbox(struct dlist *kin)
 mbtype = mboxlist_string_to_mbtype(mboxtype);
 
 r = mailbox_open_iwl(mboxname, &mailbox);
+if (!r) r = sync_mailbox_version_check(&mailbox);
 if (r == IMAP_MAILBOX_NONEXISTENT) {
 	r = mboxlist_createsync(mboxname, mbtype, partition,
 sync_userid, sync_authstate,
@@ -1668,6 +1669,7 @@ static int mailbox_cb(char *name,
  * to safely get read-only access to the annotation and
  * other "side" databases here */
 r = mailbox_open_iwl(name, &mailbox);
+if (!r) r = sync_mailbox_version_check(&mailbox);
 /* doesn't exist?  Probably not finished creating or removing yet */
 if (r == IMAP_MAILBOX_NONEXISTENT ||
 	r == IMAP_MAILBOX_RESERVED) {
@@ -1705,6 +1707,7 @@ static int do_getfullmailbox(struct dlist *kin)
  * don't have a good way to express that, so we use
  * write locks anyway */
 r = mailbox_open_iwl(kin->sval, &mailbox);
+if (!r) r = sync_mailbox_version_check(&mailbox);
 if (r) goto out;
 
 r = sync_mailbox(mailbox, NULL, NULL, kl, NULL, 1);
@@ -1955,6 +1958,7 @@ static int do_annotation(struct dlist *kin)
 mboxname_hiersep_toexternal(sync_namespacep, name, 0);
 
 r = mailbox_open_iwl(name, &mailbox);
+if (!r) r = sync_mailbox_version_check(&mailbox);
 if (r) goto done;
 
 appendattvalue(&attvalues,
@@ -2010,6 +2014,8 @@ static int do_unannotation(struct dlist *kin)
 mboxname_hiersep_toe

Re: sync_client errors out after 2.3.16 -> 2.5.9 upgrade

[ellie timoney via Info-cyrus]

On Sat, Aug 13, 2016, at 03:12 AM, Kenneth Marshall wrote:
> On Wed, Aug 10, 2016 at 12:30:03PM +1000, ellie timoney wrote:
> > On Tue, Aug 2, 2016, at 02:14 PM, Kenneth Marshall via Info-cyrus wrote:
> > > On Tue, Aug 02, 2016 at 12:34:13PM +1000, Bron Gondwana wrote:
> > > > 2.3.2 wasn't version 10, it was version 7!  It would have upgraded 
> > > > through 8, 9, 10 - and maybe you needed to reconstruct to get GUIDs for 
> > > > those versions - that sounds familiar.
> > > > 
> > > > Bron.
> > > > 
> > > 
> > > Hi Bron,
> > > 
> > > Is there a way to identify mailboxes with no GUID? Then I could target
> > > them
> > > for reconstruction first.
> > > 
> > > Regards,
> > > Ken
> > 
> > Hi Ken,
> > 
> > I don't suppose you still have mailboxes with no GUID?  Or have you
> > already found and "reconstruct -G"ed everything?
> > 
> > I've attached a patch that does two things, which are kind of the same
> > thing:
> > 
> > * If the mailbox on either end of the replication is of a version < 10,
> > then the operation will fail cleanly and early with an "Operation is not
> > supported on mailbox" error (rather than trying to do the replication,
> > then crashing out like it currently does) -- though I don't believe this
> > case will affect you, as I believe your mailboxes are all at least
> > version 10.
> > 
> > * If the mailbox on either end of the replication contains any index
> > records that do not have a GUID set, then the same will occur, and (in
> > addition to the above error) a warning will be logged to syslog like:
> > : missing guid for record  -- needs 'reconstruct
> > -G'?
> > 
> > So with this patch, you should be able to avoid these crashes, and also
> > identify mailboxes that need "reconstruct -G" just by checking for the
> > warning in syslog.
> > 
> > Are you able to try this out?
> > 
> > Cheers,
> > 
> > ellie
> 
> 
> Hi Ellie,
> 
> I tried the patch and it did print the warnings on the replica.
> Unfortunately,
> it looked like it was looping on the master (sync_client) and the updates
> that
> could be made were not. I had to rollback to the dying version.
> 
> Regards,
> Ken

What do you mean by "looping"?  sync_client in rolling mode (-r option)
will keep reprocessing the same sync log (from the top) until it
finishes it successfully, which in this case if it contains operations
on messages that aren't replicable, will be the case.  Even on a failed
run, mailboxes that did get replicated will have been replicated -- they
just won't be removed from the sync log, so the next time it tries they
will be replicated again (which does nothing if they are already up to
date from having just been replicated).  This is how it's supposed to
work.

It wouldn't repeat like this if it were crashing out, obviously -- but
in rolling mode, that would be a problem, because it's supposed to keep
going and try again, not crash out.

If you're not using rolling mode, or if the looping behaviour you
describe is something other than this, are you able to provide logs of
what it's doing?  If you run it with '-R' instead of '-r', it will be
the same as rolling mode but it won't put itself in the background, then
you can use the -v option several times to increase the verbosity of its
output, and shut it down easily with ^C.

Obviously there's privacy concerns with sending sync_client output --
even at its lowest verbosity it exposes users' mailbox names, and at
higher verbosities also exposes message content, flags, and everything
else, so don't send this output directly.  But maybe if you could look
over it and summarise it?

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: 2.5.9 build issues

[ellie timoney via Info-cyrus]

Isn't this already fixed on the 2.5 branch?

On Mon, Aug 22, 2016, at 02:17 AM, Ken Murchison via Info-cyrus wrote:
> I'm guessing we should steal json_support.h from master and add it to 
> the 2.5 branch.
> 
> 
> On 08/20/2016 10:40 AM, Wolfgang Breyha via Info-cyrus wrote:
> > After modifying my spec file from 2.5.8 to 2.5.9 I ended up with two new
> > issues:
> >
> > *) jcal.c can't be compiled with jansson < 2.7 (eg. every RHEL) because
> > json_boolean_value() is missing. I fixed it with:
> > -
> > --- cyrus-imapd-2.5.9/imap/jcal.h.orig  2016-08-20 16:20:52.049359274 
> > +0200
> > +++ cyrus-imapd-2.5.9/imap/jcal.h   2016-08-20 16:21:03.719723892 +0200
> > @@ -50,6 +50,10 @@
> >
> >   #include "util.h"
> >
> > +#if (JANSSON_MINOR_VERSION < 7)
> > +#define json_boolean_value json_is_true
> > +#endif
> > +
> >   extern char *icalcomponent_as_jcal_string(icalcomponent* comp);
> >   extern icalcomponent *jcal_string_as_icalcomponent(const char *str);
> >   extern const char *begin_jcal(struct buf *buf);
> > 
> >
> > *) htmlstrip.c gets installed as source to /usr/lib/cyrus-imapd. It seems
> > it doesn't get compiled at all and the source gets installed instead.
> >
> > Greetings, Wolfgang
> 
> -- 
> Kenneth Murchison
> Principal Systems Software Engineer
> Carnegie Mellon University
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: 2.5.9 build issues

[ellie timoney via Info-cyrus]

> Isn't this already fixed on the 2.5 branch?

Yep, commit 8a41e54

As for this:

> > > *) htmlstrip.c gets installed as source to /usr/lib/cyrus-imapd. It seems
> > > it doesn't get compiled at all and the source gets installed instead.

That's weird. htmlstrip is a tool used in the build process only, it
shouldn't get installed anywhere (nor should its source).  Will see if I
can reproduce it and work out why it happens.

Cheers,

ellie

On Mon, Aug 22, 2016, at 09:14 AM, ellie timoney via Info-cyrus wrote:
> Isn't this already fixed on the 2.5 branch?
>
> On Mon, Aug 22, 2016, at 02:17 AM, Ken Murchison via Info-cyrus wrote:
> > I'm guessing we should steal json_support.h from master and add it to
> > the 2.5 branch.
> >
> >
> > On 08/20/2016 10:40 AM, Wolfgang Breyha via Info-cyrus wrote:
> > > After modifying my spec file from 2.5.8 to 2.5.9 I ended up with two new
> > > issues:
> > >
> > > *) jcal.c can't be compiled with jansson < 2.7 (eg. every RHEL) because
> > > json_boolean_value() is missing. I fixed it with:
> > > -
> > > --- cyrus-imapd-2.5.9/imap/jcal.h.orig2016-08-20 16:20:52.049359274 
> > > +0200
> > > +++ cyrus-imapd-2.5.9/imap/jcal.h 2016-08-20 16:21:03.719723892 +0200
> > > @@ -50,6 +50,10 @@
> > >
> > >   #include "util.h"
> > >
> > > +#if (JANSSON_MINOR_VERSION < 7)
> > > +#define json_boolean_value json_is_true
> > > +#endif
> > > +
> > >   extern char *icalcomponent_as_jcal_string(icalcomponent* comp);
> > >   extern icalcomponent *jcal_string_as_icalcomponent(const char *str);
> > >   extern const char *begin_jcal(struct buf *buf);
> > > 
> > >
> > > *) htmlstrip.c gets installed as source to /usr/lib/cyrus-imapd. It seems
> > > it doesn't get compiled at all and the source gets installed instead.
> > >
> > > Greetings, Wolfgang
> >
> > --
> > Kenneth Murchison
> > Principal Systems Software Engineer
> > Carnegie Mellon University
> >
> > 
> > Cyrus Home Page: http://www.cyrusimap.org/
> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > To Unsubscribe:
> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: 2.5.9 build issues

[ellie timoney via Info-cyrus]

> > > > *) htmlstrip.c gets installed as source to /usr/lib/cyrus-imapd. It 
> > > > seems
> > > > it doesn't get compiled at all and the source gets installed instead.
> 
> That's weird. htmlstrip is a tool used in the build process only, it
> shouldn't get installed anywhere (nor should its source).  Will see if I
> can reproduce it and work out why it happens.

I can't reproduce this using either 2.5.8, 2.5.9, or the current
cyrus-imapd-2.5 from git.  The former two I testing using the released
distribution tarballs from cyrusimap.org/releases, and also the tagged
git sources.  I used the usual configure/make/make install procedure. 
In no case did I end up with a "htmlstrip.c" file existing anywhere on
my system other than the respective source directories.

Is it possible this file on your system is hanging around from some
earlier version?  Before it got moved to tools/, the htmlstrip.c file
used to live in doc/text/ -- the same directory as the plain text docs
that it was used to generate (from html sources).  It's entirely
plausible that some previous version would then install it along with
those docs.

On Mon, Aug 22, 2016, at 09:25 AM, ellie timoney via Info-cyrus wrote:
> > Isn't this already fixed on the 2.5 branch?
> 
> Yep, commit 8a41e54
> 
> As for this:
> 
> > > > *) htmlstrip.c gets installed as source to /usr/lib/cyrus-imapd. It 
> > > > seems
> > > > it doesn't get compiled at all and the source gets installed instead.
> 
> That's weird. htmlstrip is a tool used in the build process only, it
> shouldn't get installed anywhere (nor should its source).  Will see if I
> can reproduce it and work out why it happens.
> 
> Cheers,
> 
> ellie
> 
> On Mon, Aug 22, 2016, at 09:14 AM, ellie timoney via Info-cyrus wrote:
> > Isn't this already fixed on the 2.5 branch?
> >
> > On Mon, Aug 22, 2016, at 02:17 AM, Ken Murchison via Info-cyrus wrote:
> > > I'm guessing we should steal json_support.h from master and add it to
> > > the 2.5 branch.
> > >
> > >
> > > On 08/20/2016 10:40 AM, Wolfgang Breyha via Info-cyrus wrote:
> > > > After modifying my spec file from 2.5.8 to 2.5.9 I ended up with two new
> > > > issues:
> > > >
> > > > *) jcal.c can't be compiled with jansson < 2.7 (eg. every RHEL) because
> > > > json_boolean_value() is missing. I fixed it with:
> > > > -
> > > > --- cyrus-imapd-2.5.9/imap/jcal.h.orig  2016-08-20 16:20:52.049359274 
> > > > +0200
> > > > +++ cyrus-imapd-2.5.9/imap/jcal.h   2016-08-20 16:21:03.719723892 
> > > > +0200
> > > > @@ -50,6 +50,10 @@
> > > >
> > > >   #include "util.h"
> > > >
> > > > +#if (JANSSON_MINOR_VERSION < 7)
> > > > +#define json_boolean_value json_is_true
> > > > +#endif
> > > > +
> > > >   extern char *icalcomponent_as_jcal_string(icalcomponent* comp);
> > > >   extern icalcomponent *jcal_string_as_icalcomponent(const char *str);
> > > >   extern const char *begin_jcal(struct buf *buf);
> > > > 
> > > >
> > > > *) htmlstrip.c gets installed as source to /usr/lib/cyrus-imapd. It 
> > > > seems
> > > > it doesn't get compiled at all and the source gets installed instead.
> > > >
> > > > Greetings, Wolfgang
> > >
> > > --
> > > Kenneth Murchison
> > > Principal Systems Software Engineer
> > > Carnegie Mellon University
> > >
> > > 
> > > Cyrus Home Page: http://www.cyrusimap.org/
> > > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > > To Unsubscribe:
> > > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> > 
> > Cyrus Home Page: http://www.cyrusimap.org/
> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > To Unsubscribe:
> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: sync_client errors out after 2.3.16 -> 2.5.9 upgrade

[ellie timoney via Info-cyrus]

On Sat, Aug 13, 2016, at 03:12 AM, Kenneth Marshall wrote:
> Hi Ellie,
> 
> I tried the patch and it did print the warnings on the replica.
> Unfortunately,
> it looked like it was looping on the master (sync_client) and the updates
> that
> could be made were not. I had to rollback to the dying version.
> 
> Regards,
> Ken

I've managed to reproduce this (more or less) and I can see what's going
on now -- with my patch, if a mailbox on the replica is too old (or
contains messages without guids), it won't be able to replicate that
whole user.  Or at least, it won't be able to replicate that user /at a
user level/ -- though their other mailboxes might be replicable if
targeted individually with sync_client -m (but I haven't tested this).

It fails like this because the sync_server in 2.5 treats any mailbox
error as a failure of the entire "GET USER" command.  

This is different on master: there, mailboxes that have errors during
"GET USER" are just omitted from the response, and thus replication
proceeds successfully with at least some of the affected user's
mailboxes (but will probably error out later in the process, when it
tries to upload the "missing" mailboxes).

The patch works as I expected under master's semantics, but fails
inappropriately under 2.5's semantics.  Investigation continues...

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: lmtp "Bad IPLOCALPORT value" and "Bad IPREMOTEPORT value" after upgrade 2.4.18 to 2.5.9

[ellie timoney via Info-cyrus]

Do you have "lmtp" defined in /etc/services?

On Thu, Aug 25, 2016, at 08:44 AM, Andy Dorman via Info-cyrus wrote:
> We have a cluster of servers that provide external email security 
> filtering and then postfix delivers the safe email via lmtp to mailboxes 
> on the various servers in the cluster (using SASLauth of course).
> 
> I just updated our dev server to 2.5.9 and while email delivery seems to 
> be working fine, now we are seeing these messages every time postfix 
> delivers email to the upgraded server:
> 
> cyrus/lmtp[8140]: Bad IPLOCALPORT value
> cyrus/lmtp[8140]: Bad IPREMOTEPORT value
> 
> Our cyrus.conf config for lmtp looked like this at the time of the
> change.
> 
> lmtpcmd="lmtpd" listen="*:lmtp" prefork=10 maxchild=20
> 
> I have also tried replacing the wildcard host with listen="lmtp" and 
> still receive "Bad IPPOST value".
> 
> So this almost seems like postfix is not communicating the lmtp 
> correctly and Cyrus is reporting it but then goes on to handle it?
> 
> FWIW, our postfix does an LDAP lookup to fetch the hostname to deliver 
> the email within the cluster.  The hostname is plugged into this
> 
> result_filter = lmtp:inet:%s:lmtp
> 
> So is Cyrus now looking for something different from postfix?  ie, 
> should I be talking to the Postfix folks instead of bothering everyone 
> here? ;-)
> 
> Sincere regards,
> 
> -- 
> Andy Dorman
> Ironic Design, Inc.
> AnteSpam.com
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: lmtp "Bad IPLOCALPORT value" and "Bad IPREMOTEPORT value" after upgrade 2.4.18 to 2.5.9

[ellie timoney via Info-cyrus]

The "Bad IP(LOCAL|REMOTE)PORT value" log entries come from sasl, not
from lmtpd itself -- https://git.io/v6jvs, lines 1171 and 1212.

The corresponding calls to sasl_setprop seem to come from lines 1088-9
of lmtpengine.c: https://git.io/v6jJf  The two IP address parameters in
this case are the local and remote IP of the socket connection -- so,
lmtpd's IP (local), and postfix's (remote).

It's a shame sasl doesn't log the bad values -- it might have saved some
time to know what they were.  On a hunch though, do you have reverse DNS
set up for the IPs postfix and cyrus are using?  Is DNS resolution
working correctly on the cyrus server?

I wonder if you also upgraded sasl when you upgraded cyrus?  If so, from
which version to which?  (Maybe this condition always existed but some
older version of sasl didn't bother reporting it.)

Cheers,

ellie

On Fri, Aug 26, 2016, at 11:02 PM, Andy Dorman via Info-cyrus wrote:
> Thanks for checking and good question, yes.
> 
> lmtp  2003/tcp
> 
> I doubt it would have been working fine for all these years with the 2.4 
> versions without that. ;-)
> 
> And let me be clear...it still works fine.  We just have these annoying 
> 2 log messages before auth happens (see below).
> 
> It looks like lmtp is reporting the IPLOCALPORT and IPREMOTEPORT value 
> problems before auth is done, but once auth finishes lmtp is happy.
> 
> 2016-08-25T10:27:40.130890-05:00 yorick cyrus/lmtp[5695]: Bad 
> IPLOCALPORT value
> 2016-08-25T10:27:40.130915-05:00 yorick cyrus/lmtp[5695]: Bad 
> IPREMOTEPORT value
> 2016-08-25T10:27:40.152889-05:00 yorick cyrus/lmtp[5695]: login: 
> ophelia.ironicdesign.com [192.168.0.9] cyrus PLAIN User logged in
> 
> I looked at the lmtpengine.c source on github and did not see an obvious 
> source for this particular log output...but I am not a C programmer and 
> could be completely missing what is going on here.
> 
> Andy Dorman
> 
> On 08/25/2016 06:19 PM, ellie timoney via Info-cyrus wrote:
> > Do you have "lmtp" defined in /etc/services?
> >
> > On Thu, Aug 25, 2016, at 08:44 AM, Andy Dorman via Info-cyrus wrote:
> >> We have a cluster of servers that provide external email security
> >> filtering and then postfix delivers the safe email via lmtp to mailboxes
> >> on the various servers in the cluster (using SASLauth of course).
> >>
> >> I just updated our dev server to 2.5.9 and while email delivery seems to
> >> be working fine, now we are seeing these messages every time postfix
> >> delivers email to the upgraded server:
> >>
> >> cyrus/lmtp[8140]: Bad IPLOCALPORT value
> >> cyrus/lmtp[8140]: Bad IPREMOTEPORT value
> >>
> >> Our cyrus.conf config for lmtp looked like this at the time of the
> >> change.
> >>
> >> lmtpcmd="lmtpd" listen="*:lmtp" prefork=10 maxchild=20
> >>
> >> I have also tried replacing the wildcard host with listen="lmtp" and
> >> still receive "Bad IPPOST value".
> >>
> >> So this almost seems like postfix is not communicating the lmtp
> >> correctly and Cyrus is reporting it but then goes on to handle it?
> >>
> >> FWIW, our postfix does an LDAP lookup to fetch the hostname to deliver
> >> the email within the cluster.  The hostname is plugged into this
> >>
> >> result_filter = lmtp:inet:%s:lmtp
> >>
> >> So is Cyrus now looking for something different from postfix?  ie,
> >> should I be talking to the Postfix folks instead of bothering everyone
> >> here? ;-)
> >>
> >> Sincere regards,
> >>
> >> --
> >> Andy Dorman
> >> Ironic Design, Inc.
> >> AnteSpam.com
> >>
> >> 
> >> Cyrus Home Page: http://www.cyrusimap.org/
> >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> >> To Unsubscribe:
> >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> > 
> > Cyrus Home Page: http://www.cyrusimap.org/
> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > To Unsubscribe:
> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> >
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus 2.5.9 imapd children count grows and exceeds limit

[ellie timoney via Info-cyrus]

> We now have 4 of the 14 servers where the number of imapd processes 
> slowly grows and usually within 12-14 hours reaches the imapd process 
> limit (currently set to 100).

How many user accounts are on these servers?  Each client connection
constitutes one imapd process on the server.

It's my understanding that Thunderbird (for example) in its default
configuration holds 5 connections open at once, and tries to keep them
open (either with IDLE, if enabled, or NOOP), and will reconnect them
after a while if they disconnect.  (I would expect most other IMAP
clients of a similar maturity to behave similarly.)  So it would only
take 20 users leaving their mail client open most of the time to hit
your limit of 100 concurrent imapd processes.  

So 100 seems like a very small number, unless you have a very small
number of user accounts.

> From what we can tell watching the 
> process list, the old processes are never shut down after reaching their 
> usage limit (which is the default).

Do you have more information about how you determined that their usage
limit had been reached?

> While we had this problem intermittently with 2.4.18 (once or twice a 
> month on a random server), it is much worse since the update to 2.5.9. 

This might simply be 2.5.9 doing a better job of helping clients keep
open connections open?

Cheers,

ellie

On Mon, Sep 12, 2016, at 01:01 PM, Andy Dorman via Info-cyrus wrote:
> Hi everyone.  We have 14 Debian servers running the current Debian 
> release of Cyrus imap 2.5.9.
> 
> We upgraded from 2.4.18 to 2.5.9 on Aug 28, upgraded the cyrus.conf & 
> imapd.conf to rename the deprecated options and switch from skiplist to 
> twoskip for all our dbs. Our cyrus.conf and imapd.con (minus comments) 
> is at the end of this email.
> 
> We now have 4 of the 14 servers where the number of imapd processes 
> slowly grows and usually within 12-14 hours reaches the imapd process 
> limit (currently set to 100).  From what we can tell watching the 
> process list, the old processes are never shut down after reaching their 
> usage limit (which is the default).
> 
> The difference between the 4 with the problem and the others have to be 
> the mail clients.  Each server supports a different group of mail 
> domains and users, so I suspect some of the users on the problem servers 
> are using one or more mail clients that are not well behaved, but I have 
> no indication of what they are doing wrong.
> 
> While we had this problem intermittently with 2.4.18 (once or twice a 
> month on a random server), it is much worse since the update to 2.5.9. 
> We now have to restart imapd on all the problem servers almost twice a
> day.
> 
> If it matters, we have tried both idled and polling and have not seen 
> any difference.
> 
> So has anyone else seen an issue like this?  Does anyone have any 
> suggestions on how to debug?
> 
> We have examined syslog and do not see any strange reports outside of 
> the normal berkeley DBERROR messages we always see due to bdb compiled 
> into the debian Cyrus release.
> 
> = cyrus.conf =
> START {
> recover cmd="/usr/sbin/cyrus ctl_cyrusdb -r"
> idlecmd="idled"
> delprunecmd="/usr/sbin/cyrus expire -E 3"
> tlsprunecmd="/usr/sbin/cyrus tls_prune"
> }
> 
> SERVICES {
> imapcmd="imapd" listen="*:imap" prefork=5 maxchild=100
> imaps   cmd="imapd -s" listen="*:imaps" prefork=5 maxchild=100
> lmtpcmd="lmtpd" listen="*:lmtp" prefork=5 maxchild=20
> sieve   cmd="timsieved" listen="*:sieve" prefork=0 maxchild=100
> }
> 
> EVENTS {
> checkpoint  cmd="/usr/sbin/cyrus ctl_cyrusdb -c" period=30
> delprunecmd="/usr/sbin/cyrus expire -E 3" period=120
> tlsprunecmd="/usr/sbin/cyrus tls_prune" period=60
> resquat cmd="/usr/sbin/cyrus squatter -s -r -i *" at=0437
> }
> 
> = imapd.conf =
> admins: cyrus
> 
> allowallsubscribe: on
> 
> allowanonymouslogin: no
> 
> allowplaintext: on
> 
> altnamespace: on
> 
> annotation_db: twoskip
> 
> anyoneuseracl: off
> 
> autocreate_quota: 512000
> 
> configdirectory: /var/lib/cyrus
> 
> defaultdomain: ironicdesign.com
> 
> defaultpartition: default
> 
> duplicate_db: twoskip
> 
> expunge_mode: delayed
> 
> fulldirhash: on
> 
> guid_mode: sha1
> 
> hashimapspool: on
> 
> idlesocket: /var/run/cyrus/socket/idle
> 
> improved_mboxlist_sort: on
> 
> lmtp_downcase_rcpt: on
> 
> lmtpsocket: /var/run/cyrus/socket/lmtp
> 
> mboxname_lockpath: /run/cyrus/lock
> 
> notifysocket: /var/run/cyrus/socket/notify
> 
> partition-default: /var/spool/cyrus/mail
> 
> popminpoll: 1
> 
> proc_path: /run/cyrus/proc
> 
> quota_db: twoskip
> 
> sasl_log_level: 7
> sasl_mech_list: PLAIN
> sasl_pwcheck_method: saslauthd
> sasl_minimum_layer: 0
> allowapop: no
> 
> sieveusehomedir: false
> sievedir: /var/spool/cyrus/sieve
> 
> statuscache_db: twoskip
> 
> syslog_prefix: cyrus
> 
> tls_sessions_db: twoskip
> 
> tls_client_ca_file: /etc/ssl/certs/mail.ironicdesig

Re: Cyrus 2.5.9 imapd children and tcp_timeout questions

[ellie timoney via Info-cyrus]

> We do not have a lot of active users on each server (probably under 20), 
> but enough that what you describe could be happening.

Keep in mind too, that your users might have multiple clients running at
the same time -- a desktop that's always on, a laptop that comes and
goes when the laptop is in use, their phone/tablet/etc -- each of which
might use multiple connections like Thunderbird does.  You might see as
many as 15 legitimate concurrent connections (and thus imapd processes)
for a single user account in normal use in this particular use case.

> As for how I am determining they are reaching their usage limit, all I 
> have are the process start times. When we hit the 100 limit I can see 
> that the preforked 5 are still running as are many that were started 
> many hours ago.

I would expect exactly this if you have long-lived client connections,
which is normal for IMAP.

If those first five clients (which might really be five connections from
a single client) don't disconnect themselves, and their network
connection doesn't drop out, and they behave well with respect to
IDLE/NOOP, and the server doesn't get restarted or lose its network
connection, then those connections, and therefore those particular
processes, will keep going.  This is how it's supposed to work. :)

> I could be crazy wrong (probably am ;-) but it seemed to 
> me that if processes were being killed and recycled I should not see a 
> lot of old processes sitting around.

Here's the section of imapd(8) describing the uses limit:
> -U uses
> The maximum number of times that the process should be used for new 
> connections before shutting down.  The default is 250.

Note that it talks about the process being used for new connections. 
Each imapd process serves one connection at a time (so if you have 50
client connections, you will have 50 imapd processes to serve them, plus
whatever you have preforked ready to serve new connections).  When one
of these connections disconnects, the imapd process that was serving
that connection goes back into the pool, ready to serve a new incoming
connection -- unless it has already served $uses connections, in which
case it exits, and master spawns a new one instead if necessary.

So you can see how, if you have mostly long-lived connections, each
imapd's count of how many connections it has served would grow very very
slowly -- many may not ever reach their limit, because you end up
needing to reboot the server (OS security updates, data centre works,
etc) before that occurs.

Cheers,

ellie

On Mon, Sep 12, 2016, at 10:40 PM, Andy Dorman via Info-cyrus wrote:
> Eliie, thank you for that information about imapd usage. I have only 
> been managing our imap for a few months and am still learning.
> 
> We do not have a lot of active users on each server (probably under 20), 
> but enough that what you describe could be happening.
> 
> As for how I am determining they are reaching their usage limit, all I 
> have are the process start times. When we hit the 100 limit I can see 
> that the preforked 5 are still running as are many that were started 
> many hours ago. I could be crazy wrong (probably am ;-) but it seemed to 
> me that if processes were being killed and recycled I should not see a 
> lot of old processes sitting around.
> 
> And I owe an apology to Konrad...I originally wrote this email a week 
> ago and then set it aside while I tried setting tcp_timeout=1.  Then 
> when that did not help, I sent the email and forgot to update my 
> imapd.conf.  This is what I have in my imapd.conf.
> 
> tcp_keepalive: 1
> tcp_keepalive_idle: 900
> tcp_keepalive_cnt: 5
> tcp_keepalive_intvl: 75
> 
> It would be great if anyone who is using tcp_keepalive could share their 
> settings in case I completely misunderstood how these work.
> 
> In the meantime I am going to bump my max processes up to 200 and my 
> monitoring warning level to 150 see how that works.
> 
> Thanks for all the help everyone.
> 
> Andy
> 
> On 09/11/2016 11:05 PM, ellie timoney via Info-cyrus wrote:
> >> We now have 4 of the 14 servers where the number of imapd processes
> >> slowly grows and usually within 12-14 hours reaches the imapd process
> >> limit (currently set to 100).
> >
> > How many user accounts are on these servers?  Each client connection
> > constitutes one imapd process on the server.
> >
> > It's my understanding that Thunderbird (for example) in its default
> > configuration holds 5 connections open at once, and tries to keep them
> > open (either with IDLE, if enabled, or NOOP), and will reconnect them
> > after a while if they disconnect.  (I would expect most other IMAP
> > clients of a similar maturity to behave similarly.)  So it would only

Re: Cyrus 2.5.9 imapd children and tcp_keepalive & idle socket questions

[ellie timoney via Info-cyrus]

record of "idlemethod" in commit history either, it's
not like it's leftover from something that's since changed.  Maybe it's
from a Debian-specific patch?

Anyway, it sounds like your setup is fine, and things are behaving as
expected. :)

> I know this user so I will ask her what email client she is using and 
> how often it is set to check email.

How often it is set to check email won't make much difference in this
case.  That is more for servers that don't support Idle, so the client
has to keep asking "have you got anything new" every so often.  The fact
these connections are Idle means it isn't doing that -- it just tells
the server it's Idle, and then waits for the server to tell it about any
new activity.

But understanding how the service is actually being used will do a lot
to calibrate your expectations of how the server should be behaving
(such as, how many concurrent connections/imapd processes you expect to
see) :)

Cheers,

ellie

On Thu, Sep 15, 2016, at 08:07 AM, Andy Dorman via Info-cyrus wrote:
> On 09/13/2016 02:15 AM, Bron Gondwana via Info-cyrus wrote:
> > On Tue, 13 Sep 2016, at 10:44, ellie timoney via Info-cyrus wrote:
> >> Note that it talks about the process being used for new connections.
> >> Each imapd process serves one connection at a time (so if you have 50
> >> client connections, you will have 50 imapd processes to serve them, plus
> >> whatever you have preforked ready to serve new connections).  When one
> >> of these connections disconnects, the imapd process that was serving
> >> that connection goes back into the pool, ready to serve a new incoming
> >> connection -- unless it has already served $uses connections, in which
> >> case it exits, and master spawns a new one instead if necessary.
> >>
> >> So you can see how, if you have mostly long-lived connections, each
> >> imapd's count of how many connections it has served would grow very very
> >> slowly -- many may not ever reach their limit, because you end up
> >> needing to reboot the server (OS security updates, data centre works,
> >> etc) before that occurs.
> >
> > If you look in /var/lib/imap/proc/* you should see one file for each 
> > process id,
> > which tells you who is connected to that process, and which folder they have
> > selected (if any).  This is quite useful for tracking down if there's a 
> > single user
> > causing issues.
> >
> > Bron.
> >
> >
> 
> Bron, that was incredibly useful (FWIW, the proc list in Debian is at 
> /run/cyrus/proc/)
> 
> I took a look and our worst case server appears to have two users 
> (actually one user with two separate addresses) that are causing the 
> trouble. I restarted cyrus-imapd about 10 hrs ago and we are up to 28 
> imapd processes with start times all through the 10-hr block and they 
> all appear to be owned by one of two addresses as you can see here:
> 
> [snip addresses]
> 
> So, given that all of these report as idle, should they not be shutting 
> down at some point?
> 
> I have the settings below in imapd.conf. My understanding of them is 
> after a connection has been idle for 15 min, try 5 tcp probes at 75 sec 
> intervals before marking the connection as broken.
> 
> tcp_keepalive: 1
> tcp_keepalive_idle: 900
> tcp_keepalive_cnt: 5
> tcp_keepalive_intvl: 75
> 
> So is it possible that this client is responding to the tcp keepalive 
> packets so the idle connection is never shut down, but the client keeps 
> opening new connections when it wants to check email (which appears to 
> happen about every 10 - 15 minutes)?
> 
> I know this user so I will ask her what email client she is using and 
> how often it is set to check email.
> 
> Also, I see this note in cyrus.conf about idled...
> 
> # this is only necessary if idlemethod is set to "idled" in imapd.conf
> idlecmd="idled"
> 
> But I can't find anything about idlemethod in any of the cyrus docs or 
> man pages.  All I can find to set in imapd.conf for idle is the socket:
> 
> idlesocket: /var/run/cyrus/socket/idle
> 
> Hm...wait a minute...in looking for the Debian proc directory I came 
> across another idle socket created when I restarted cyrus-imapd.
> 
> srwxrwxrwx 1 root root 0 Sep 14 06:10 /run/cyrus/socket/idle
> 
> However, our imapd.conf specified a different idle socket which was also 
> created when cyrus-imapd was restarted.
> 
> srwxrwxrwx 1 root root 0 Sep 14 06:10 /var/run/cyrus/socket/idle
> 
> So I am thinking I should modify my imapd.conf idlesocket to 
> "/run/cyrus/socket/idle" which cyrus seems to be setting up anyway.  Or 
> am I completely missing an obvious point here?
> 
> Sorry for being so clueless and thanks for all the help.
> 
> -- 
> Andy Dorman
> Ironic Design, Inc.
> AnteSpam.com
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus 2.5.9 imapd children and tcp_keepalive & idle socket questions

[ellie timoney via Info-cyrus]

Hi Andy,

Bringing this discussion back over here...

> FWIW, I have more empirical data to report on the above server with only
> 2 accounts that are being checked by an old BlackBerry (along with
> Thunderbird an hour or two each day...but the BB is the only device
> checking 24/7)
> 
> It has been ~24 hr since the last cyrus-imapd restart.
> 
> There are now 123 imapd processes, all idled, and all of course for
> those two accounts. Looking at the proc file times in the time-sorted
> list below they span from when the restart happened (Sep 20 0734) to
> just a few minutes ago.
> 
> root@dorcas:~# ls -alt /run/cyrus/proc/
> total 492
> drwx-- 2 cyrus mail 2500 Sep 21 07:19 .
> -rw--- 1 cyrus mail   84 Sep 21 07:10 20204
> [...]

I wonder if we can see what the old processes are actually doing...

Is there any chance you're able to obtain a backtrace of these?

You can use a command something like:

  gdb -p pid /path/to/imapd

where pid is the one of the imapd process ids (which is the last column
in that ls output from /run/cyrus/proc/) and where the correct path to
wherever your imapd executable actually lives (I think this might be
"/usr/lib/cyrus/bin/imapd" on Debian).  You should run this command as
the same user that your cyrus server runs as (probably "cyrus").

Once you're in gdb, look for output like "Reading symbols from
/path/to/imapd...".  If this line also contains "no debugging symbols
found", then we can't get anything useful like this, just "quit" -> "y"
to exit gdb.  But if it does find debugging symbols, great: run "bt" to
obtain a backtrace, then "quit" -> "y" to exit, and send through the
output from the session (probably best to paste the output into a text
file and then attach that to your email).

Another possibility is to trace the system calls.  You can use a command
something like this (also as the "cyrus" user):

strace -p pid

Do not blindly email the output from this command, it may contain
sensitive/private content!

Let this run for 10 minutes or so, then press ctrl-c to stop it, and
then I guess kind of try to describe the output, and we can go from
there.  If it's very noisy, you've caught it in the middle of doing
something. Wait a little while and try again, or try a different imapd
pid: we're trying to see what the inactive ones are waiting for.

Cheers,

ellie

On Fri, Sep 16, 2016, at 10:40 AM, ellie timoney via Info-cyrus wrote:
> > So, given that all of these report as idle, should they not be shutting
> > down at some point?
>
> "Idle" has special meaning in IMAP.  The last column in the proc/* files
> is the IMAP command currently being executed by the connected client;
> "Idle" here means the client is executing the IDLE command (RFC 2177 if
> you're curious).  It does not mean that the connection is unused and
> could be discarded.
>
> Well-behaved modern clients will spend most of their time in IDLE.
>
> Another interesting thing you can see from the proc/* files is how long
> ago the client's current IMAP command was invoked, by looking at the
> modification time of the proc file (e.g. with ls -l).  You'll probably
> find most of your "Idle" commands were invoked sometime within the last
> half-hour or so, even though the connection itself may have been up for
> many hours.
>
> If you see files in the Idle state that are more than half an hour old,
> then that might indicate a client that has entered the Idle state but
> then lost its connection (which is where the tcp_keepalive settings
> become useful).
>
> > I have the settings below in imapd.conf. My understanding of them is
> > after a connection has been idle for 15 min, try 5 tcp probes at 75 sec
> > intervals before marking the connection as broken.
>
> This is correct (except note that "idle" here takes the general meaning,
> not the IMAP meaning).  But if the connection is still there, then it's
> not broken, and tcp_keepalive won't mark it as broken.
>
> The fact that the connections aren't being disconnected by your
> tcp_keepalive settings indicates that the clients are still alive,
> idling politely until they have something to do (or until the server
> notifies them of activity).
>
> > Also, I see this note in cyrus.conf about idled...
>
> So, the client enters the Idle state when it has nothing useful to do.
> When it wants to execute a new command (maybe because the user has
> clicked something in the interface), it ends the idle state, sends the
> new command and deals with whatever response arrives, etc etc.  When it
> has nothing interesting to do anymore (maybe the user ha

Re: Cyrus 2.5.9 imapd children and tcp_keepalive & idle socket questions

[ellie timoney via Info-cyrus]

Hi Andy,

Thanks very much for the detailed information.  It may take me a while
to digest this, but it looks like it will be very helpful.

A very very initial (and, thus, possibly mistaken) assessment:

* pselect timing out every 60 seconds sounds like what I would expect
while the server is in the IDLE command
* the fact that it keeps timing out tells us that, at least as far as
the server is concerned, the client is still there
* if the connection had dropped out, tcp_keepalive would mark it as EOF,
and pselect would return a number greater than zero -- not "0 (Timeout)"
-- to indicate that one of the selected descriptors had activity on it
(the EOF is a kind of activity).

A couple of questions:

* in your cyrus.conf, do you have "idled" enabled?  (It might be in the
START or DAEMON sections.)  You might have already answered this, I'll
check through your older messages in a minute
* in your syslog logs, do you get lines like "IDLE: error sending
message INIT to idled for mailbox : . Falling back to
polling every 60 seconds."?  If so, what is the "error" section?

If you could continue monitoring the strace output for pid 16168 and
send an update if any new patterns emerge, that would be great.

Another thing that would be interesting would be if you could coordinate
with her to do a bunch of activity with the blackberry -- manually check
for new mail, manually open each folder, read some messages, change some
flags (such as toggling some messages as unread and then reading them
again), repeat for each folder, that sort of thing -- and continue
monitoring pid 16168 and see if the pattern changes at all.

If you can avoid restarting for as long as possible, so we can keep
watching 16168 specifically, that would help.

If you reach the point where it's going to be necessary to restart
cyrus, before you do so, try getting her to power cycle the blackberry,
and see what effect that has on 16168.  (But also don't do this until as
late as possible, so we can continue to look for interesting
patterns/deviations in the meantime.)

Cheers,

ellie

On Sat, Oct 1, 2016, at 12:00 AM, Andy Dorman via Info-cyrus wrote:
> On 09/28/2016 09:59 AM, Andy Dorman wrote:
> > On 09/27/2016 07:08 PM, ellie timoney via Info-cyrus wrote:
> >> I wonder if we can see what the old processes are actually doing...
> >>
> >> Is there any chance you're able to obtain a backtrace of these?
> >>
> >> You can use a command something like:
> >>
> >>   gdb -p pid /path/to/imapd
> >>
> >> where pid is the one of the imapd process ids (which is the last column
> >> in that ls output from /run/cyrus/proc/) and where the correct path to
> >> wherever your imapd executable actually lives (I think this might be
> >> "/usr/lib/cyrus/bin/imapd" on Debian).  You should run this command as
> >> the same user that your cyrus server runs as (probably "cyrus").
> >>
> >> Once you're in gdb, look for output like "Reading symbols from
> >> /path/to/imapd...".  If this line also contains "no debugging symbols
> >> found", then we can't get anything useful like this, just "quit" -> "y"
> >> to exit gdb.  But if it does find debugging symbols, great: run "bt" to
> >> obtain a backtrace, then "quit" -> "y" to exit, and send through the
> >> output from the session (probably best to paste the output into a text
> >> file and then attach that to your email).
> >>
> >> Another possibility is to trace the system calls.  You can use a command
> >> something like this (also as the "cyrus" user):
> >>
> >> strace -p pid
> >>
> >> Do not blindly email the output from this command, it may contain
> >> sensitive/private content!
> >>
> >> Let this run for 10 minutes or so, then press ctrl-c to stop it, and
> >> then I guess kind of try to describe the output, and we can go from
> >> there.  If it's very noisy, you've caught it in the middle of doing
> >> something. Wait a little while and try again, or try a different imapd
> >> pid: we're trying to see what the inactive ones are waiting for.
> >>
> >> Cheers,
> >>
> >> ellie
> >
> > Ellie, thanks for doing the above "extended debug" version of a reply.
> > While not a total "nube" to debugging compiled processes, I have not
> > done it for a long time and never with Linux, so I am unfamiliar with
> > the tools.  Your words are a great help.
> >
> > I installed and ran gdb on one of our "problem" 

Re: Cyrus 2.5.9 imapd children and tcp_keepalive & idle socket questions

[ellie timoney via Info-cyrus]

On Tue, Oct 4, 2016, at 12:46 AM, Andy Dorman via Info-cyrus wrote:
> On 10/02/2016 06:57 PM, ellie timoney via Info-cyrus wrote:
> > Hi Andy,
> >
> > Thanks very much for the detailed information.  It may take me a while
> > to digest this, but it looks like it will be very helpful.
> >
> > A very very initial (and, thus, possibly mistaken) assessment:
> >
> > * pselect timing out every 60 seconds sounds like what I would expect
> > while the server is in the IDLE command
> > * the fact that it keeps timing out tells us that, at least as far as
> > the server is concerned, the client is still there
> > * if the connection had dropped out, tcp_keepalive would mark it as EOF,
> > and pselect would return a number greater than zero -- not "0 (Timeout)"
> > -- to indicate that one of the selected descriptors had activity on it
> > (the EOF is a kind of activity).
> >
> > A couple of questions:
> >
> > * in your cyrus.conf, do you have "idled" enabled?  (It might be in the
> > START or DAEMON sections.)  You might have already answered this, I'll
> > check through your older messages in a minute
> > * in your syslog logs, do you get lines like "IDLE: error sending
> > message INIT to idled for mailbox : . Falling back to
> > polling every 60 seconds."?  If so, what is the "error" section?
> >
> > If you could continue monitoring the strace output for pid 16168 and
> > send an update if any new patterns emerge, that would be great.
> >
> > Another thing that would be interesting would be if you could coordinate
> > with her to do a bunch of activity with the blackberry -- manually check
> > for new mail, manually open each folder, read some messages, change some
> > flags (such as toggling some messages as unread and then reading them
> > again), repeat for each folder, that sort of thing -- and continue
> > monitoring pid 16168 and see if the pattern changes at all.
> >
> > If you can avoid restarting for as long as possible, so we can keep
> > watching 16168 specifically, that would help.
> >
> > If you reach the point where it's going to be necessary to restart
> > cyrus, before you do so, try getting her to power cycle the blackberry,
> > and see what effect that has on 16168.  (But also don't do this until as
> > late as possible, so we can continue to look for interesting
> > patterns/deviations in the meantime.)
> >
> > Cheers,
> >
> > ellie
> >
> > On Sat, Oct 1, 2016, at 12:00 AM, Andy Dorman via Info-cyrus wrote:
> >> On 09/28/2016 09:59 AM, Andy Dorman wrote:
> >>> On 09/27/2016 07:08 PM, ellie timoney via Info-cyrus wrote:
> >>>> I wonder if we can see what the old processes are actually doing...
> >>>>
> >>>> Is there any chance you're able to obtain a backtrace of these?
> >>>>
> >>>> You can use a command something like:
> >>>>
> >>>>   gdb -p pid /path/to/imapd
> >>>>
> >>>> where pid is the one of the imapd process ids (which is the last column
> >>>> in that ls output from /run/cyrus/proc/) and where the correct path to
> >>>> wherever your imapd executable actually lives (I think this might be
> >>>> "/usr/lib/cyrus/bin/imapd" on Debian).  You should run this command as
> >>>> the same user that your cyrus server runs as (probably "cyrus").
> >>>>
> >>>> Once you're in gdb, look for output like "Reading symbols from
> >>>> /path/to/imapd...".  If this line also contains "no debugging symbols
> >>>> found", then we can't get anything useful like this, just "quit" -> "y"
> >>>> to exit gdb.  But if it does find debugging symbols, great: run "bt" to
> >>>> obtain a backtrace, then "quit" -> "y" to exit, and send through the
> >>>> output from the session (probably best to paste the output into a text
> >>>> file and then attach that to your email).
> >>>>
> >>>> Another possibility is to trace the system calls.  You can use a command
> >>>> something like this (also as the "cyrus" user):
> >>>>
> >>>> strace -p pid
> >>>>
> >>>> Do not blindly email the output from this command, it may contain
> >>>> sensitive/private content!
> >>>>
> >>>> Let t

Cyrus IMAP 2.5.10 released

[ellie timoney via Info-cyrus]

The Cyrus team is proud to announce the immediate availability of a new
version of Cyrus IMAP: 2.5.10

This is a bug fix release for the 2.5 series.

Download URLs:

  http://www.cyrusimap.org/releases/cyrus-imapd-2.5.10.tar.gz
  http://www.cyrusimap.org/releases/cyrus-imapd-2.5.10.tar.gz.sig

  ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-2.5.10.tar.gz
  ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-2.5.10.tar.gz.sig

Please consult the release notes before upgrading to 2.5.10:

 http://cyrusimap.org/imap/download/release-notes/2.5/x/2.5.10.html

And join us at https://github.com/cyrusimap/cyrus-imapd to report
issues, join in the deliberations of new features for the next Cyrus
IMAP release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

Ellie Timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus mail spool entirely on SSD - tiered storage

[ellie timoney via Info-cyrus]

On Sat, Oct 22, 2016, at 10:25 AM, Rob N ★ via Info-cyrus wrote:
> The docs suggest this feature is in the currently-unreleased Cyrus 3.0
> series. That's probably true, but I can't be sure (FastMail runs the
> current dev/master branch, so we're ahead of 3.0). It'll be well worth
> the upgrade when the time comes though if you've got large mail
> stores. Its a rock-solid feature, I love it.

Confirming: yes, this feature is in the currently-unreleased 3.0 series (and 
has been since the first beta).

It's not available in the 2.5 series or earlier.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyrus master fails with status 71

[ellie timoney via Info-cyrus]

> accept failed: Software caused connection abort

Some sleuthing suggests that "Software caused connection abort"
corresponds with "ECONNABORTED".

The man page on my system for accept(2) unhelpfully defines this as:

>ECONNABORTED
>   A connection has been aborted.

But some digging around online suggests that this situation occurs when
a client connects, but subsequently disconnects  (RST) before the server
gets around to accept()ing the connection.  When the server does
eventually accept(), the accept() fails with this error.

Which sounds to me like we want to treat ECONNABORTED similarly to
EAGAIN, not as a fatal OS error.  I'll have a patch up for this shortly.

Cheers,

ellie

On Wed, Oct 26, 2016, at 09:27 AM, Eric Cunningham via Info-cyrus wrote:
> Having repeatedly experienced the "status 71" issue, I've been 
> incrementally bumping it's value up.  It's currently set to 32768 (!) 
> and that value was in place when it most recently failed.
> 
> 
> On 10/25/16 4:21 PM, Shawn Bakhtiar via Info-cyrus wrote:
> > H.. if that’s the case could you be hitting the the maximum number
> > of accepts??
> >
> > Check the 11.11.1.2. kern.ipc.soacceptqueue section of the FreeBSD handbook
> >
> > https://www.freebsd.org/doc/handbook/configtuning-kernel-limits.html
> >
> > Given the load you described perhaps 128 is just not enough?
> >
> >
> >
> >> On Oct 24, 2016, at 1:22 PM, Eric Cunningham via Info-cyrus
> >>  >> > wrote:
> >>
> >>
> >>
> >> =
> >>  Eric Cunningham
> >>  Information Services - http://whoi-it.whoi.edu
> >>  Woods Hole Oceanographic Institution - http://www.whoi.edu
> >>  Woods Hole, MA  02543-1541 phone: (508) 289-2224
> >>  fax: (508) 457-2174   e-mail: ecunning...@whoi.edu
> >> 
> >> =
> >>
> >> On 10/24/2016 03:45 PM, Bron Gondwana via Info-cyrus wrote:
> >>> On Tue, 25 Oct 2016, at 02:45, Eric Cunningham via Info-cyrus wrote:
>  Hi list, we're running cyrus imap 2.5.9 built from the FreeBSD 10-2
>  (release-p7) ports tree.
> 
>  The cyrus master process is failing periodically (every 1-2 weeks) as
>  follows:
> 
>  Oct 22 07:38:48 imap1 master[7767]: process type:SERVICE name:imaps
>  path:/usr/local/cyrus/bin/imapd age:305.215s pid:32760 exited, status 71
>  Oct 22 07:38:48 imap1 master[7767]: service imaps/ipv4 pid 32760 in
>  READY state: terminated abnormally
>  Oct 22 07:38:48 imap1 master[7767]: too many failures for service
>  imaps/ipv4, disabling until next SIGHUP
> 
>  This prevents new connections by clients until cyrus is restarted.  I've
>  looked around the web but have not seen this issue reported.
> 
>  A little background:
> 
>  Our initial thought on this was that we were running out of listen
>  queues so have upped that incrementally from the default of 32 to a
>  current setting of 32768 via /usr/local/etc/rc.d/imapd using the -l
>  option, with increased kern.ipc.soacceptqueue set to 32768, but that
>  hasn't helped.  Sometimes the "status 71" occurs during periods of light
>  use during off hours, like on Saturday mornings.
> 
>  We have ~1400 imap accounts, though the number of impad processes hovers
>  around 3,000-4,000.  There have been spikes observed as high as 12,000
>  imapd processes.  In that particular case, 1 user had 2 imap clients
>  accounting for near 6,000 of those connections.  We've attempted to
>  limit these high numbers using the following imapd.conf values:
> 
>  maxlogins_per_host: 50
>  maxlogins_per_user: 30
>  tcp_keepalive: 1
>  tcp_keepalive_cnt: 1
>  tcp_keepalive_idle: 30
>  tcp_keepalive_intvl: 900
> 
>  However, it seems that once these were reached, no new connections were
>  permitted and resulted in all manner of user complaints about not being
>  able to get at their email.
> 
>  Any ideas on this "status 71" issue?  Could an upgrade to 2.5.10
>  possibly address this?  Thanks!
> >>>
> >>> https://www.freebsd.org/cgi/man.cgi?query=sysexits
> >>>
> >>> EX_OSERR (71) An operating system error has been
> >>> detected.  This
> >>>   is intended to be used for such things as
> >>> ``cannot
> >>>   fork'', ``cannot create pipe'', or the
> >>> like.  It
> >>>   includes things like getuid returning a
> >>> user that
> >>>   does not exist in the passwd file.
> >>>
> >>> So the question is: what failed?  Is there anything earlier in the
> >>> log to suggest
> >>> what the imapd was doing when it died?
> >>>
> >>> Bron.
> >>>
> >>
> >> Using the example I posted, I traced back imaps process id 32760 and
> >> found only this:
> >>
> >> Oct

Re: cyrus master fails with status 71

[ellie timoney via Info-cyrus]

Hi Eric,

Patch attached.  I'd appreciate if you could advise whether this helps. 
Though I guess you won't be able to tell for a couple of weeks.

If it doesn't cause any new problems (I don't expect it to), then it
will be included in 2.5.11 (whenever that comes out).

Cheers,

ellie

On Wed, Oct 26, 2016, at 10:04 AM, ellie timoney via Info-cyrus wrote:
> > accept failed: Software caused connection abort
> 
> Some sleuthing suggests that "Software caused connection abort"
> corresponds with "ECONNABORTED".
> 
> The man page on my system for accept(2) unhelpfully defines this as:
> 
> >ECONNABORTED
> >   A connection has been aborted.
> 
> But some digging around online suggests that this situation occurs when
> a client connects, but subsequently disconnects  (RST) before the server
> gets around to accept()ing the connection.  When the server does
> eventually accept(), the accept() fails with this error.
> 
> Which sounds to me like we want to treat ECONNABORTED similarly to
> EAGAIN, not as a fatal OS error.  I'll have a patch up for this shortly.
> 
> Cheers,
> 
> ellie
> 
> On Wed, Oct 26, 2016, at 09:27 AM, Eric Cunningham via Info-cyrus wrote:
> > Having repeatedly experienced the "status 71" issue, I've been 
> > incrementally bumping it's value up.  It's currently set to 32768 (!) 
> > and that value was in place when it most recently failed.
> > 
> > 
> > On 10/25/16 4:21 PM, Shawn Bakhtiar via Info-cyrus wrote:
> > > H.. if that’s the case could you be hitting the the maximum number
> > > of accepts??
> > >
> > > Check the 11.11.1.2. kern.ipc.soacceptqueue section of the FreeBSD 
> > > handbook
> > >
> > > https://www.freebsd.org/doc/handbook/configtuning-kernel-limits.html
> > >
> > > Given the load you described perhaps 128 is just not enough?
> > >
> > >
> > >
> > >> On Oct 24, 2016, at 1:22 PM, Eric Cunningham via Info-cyrus
> > >>  > >> <mailto:info-cyrus@lists.andrew.cmu.edu>> wrote:
> > >>
> > >>
> > >>
> > >> =
> > >>  Eric Cunningham
> > >>  Information Services - http://whoi-it.whoi.edu
> > >>  Woods Hole Oceanographic Institution - http://www.whoi.edu
> > >>  Woods Hole, MA  02543-1541 phone: (508) 289-2224
> > >>  fax: (508) 457-2174   e-mail: ecunning...@whoi.edu
> > >> <mailto:ecunning...@whoi.edu>
> > >> =
> > >>
> > >> On 10/24/2016 03:45 PM, Bron Gondwana via Info-cyrus wrote:
> > >>> On Tue, 25 Oct 2016, at 02:45, Eric Cunningham via Info-cyrus wrote:
> > >>>> Hi list, we're running cyrus imap 2.5.9 built from the FreeBSD 10-2
> > >>>> (release-p7) ports tree.
> > >>>>
> > >>>> The cyrus master process is failing periodically (every 1-2 weeks) as
> > >>>> follows:
> > >>>>
> > >>>> Oct 22 07:38:48 imap1 master[7767]: process type:SERVICE name:imaps
> > >>>> path:/usr/local/cyrus/bin/imapd age:305.215s pid:32760 exited, status 
> > >>>> 71
> > >>>> Oct 22 07:38:48 imap1 master[7767]: service imaps/ipv4 pid 32760 in
> > >>>> READY state: terminated abnormally
> > >>>> Oct 22 07:38:48 imap1 master[7767]: too many failures for service
> > >>>> imaps/ipv4, disabling until next SIGHUP
> > >>>>
> > >>>> This prevents new connections by clients until cyrus is restarted.  
> > >>>> I've
> > >>>> looked around the web but have not seen this issue reported.
> > >>>>
> > >>>> A little background:
> > >>>>
> > >>>> Our initial thought on this was that we were running out of listen
> > >>>> queues so have upped that incrementally from the default of 32 to a
> > >>>> current setting of 32768 via /usr/local/etc/rc.d/imapd using the -l
> > >>>> option, with increased kern.ipc.soacceptqueue set to 32768, but that
> > >>>> hasn't helped.  Sometimes the "status 71" occurs during periods of 
> > >>>> light
> > >>>> use during off hours, like on Saturday mornings.
> > >>>>
> > >>>> We have ~1400 imap a

Cyrus IMAP 3.0.0-beta4 released

[ellie timoney via Info-cyrus]

The Cyrus team is proud to announce the immediate availability of the
fourth beta from the Cyrus IMAP 3.0 series: 3.0.0-beta4.

As this is a beta release it is not considered stable for production
usage.  Interfaces, APIs, features, etc are likely to change.

Download URLs:

http://www.cyrusimap.org/releases/cyrus-imapd-3.0.0-beta4.tar.gz
http://www.cyrusimap.org/releases/cyrus-imapd-3.0.0-beta4.tar.gz.sig

ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-3.0.0-beta4.tar.gz
ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-3.0.0-beta4.tar.gz.sig

Please consult the release notes before upgrading to 3.0.0-beta4:


http://www.cyrusimap.org/dev/imap/download/release-notes/3.0/x/3.0.0-beta4.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to
report issues, join in the deliberations of new features for the next
Cyrus IMAP release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

Ellie Timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyrus master fails with status 71

[ellie timoney via Info-cyrus]

On Tue, Nov 8, 2016, at 05:01 AM, Eric Cunningham wrote:
> Nov  7 10:18:19 imap1 lmtpunix[58768]: unable to setsocketopt(TCP_KEEPCNT): 
> Invalid argument
> Nov  7 10:18:19 imap1 lmtpunix[58768]: unable to setsocketopt(TCP_KEEPIDLE): 
> Invalid argument
> Nov  7 10:18:19 imap1 lmtpunix[58768]: unable to setsocketopt(TCP_KEEPINTVL): 
> Invalid argument
> 
> Why is lmptunix complaining about options passed to imapd?

There's a bug here:

Based on a comment ("/* tcp only */"), master/service.c is trying to
limit the application of the tcp_keepalive config to TCP sockets only. 
But it's doing this by checking for the type being SOCK_STREAM -- which
is not exclusive to TCP sockets.  Notably, AF_UNIX domain sockets may
also have type SOCK_STREAM, and this looks like the default
configuration for "lmtpunix".

So, those errors are because it's trying to set TCP_* socket options on
a socket that is not a TCP socket.  (Notice too that it doesn't error
when trying to set SO_KEEPALIVE, because that's a valid SOCK_STREAM
option.  It only fails to set the TCP_* options.)

Looks like this has been fixed on master already.  I'll see if the fix
backports to 2.5 cleanly, but in the meantime you can ignore these
errors.

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.0.0-beta5 released

[ellie timoney via Info-cyrus]

The Cyrus team is proud to announce the immediate availability of the
fifth beta from the Cyrus IMAP 3.0 series: 3.0.0-beta5.

As this is a beta release it is not considered stable for production
usage.  Interfaces, APIs, features, etc are likely to change.

Download URLs:

http://www.cyrusimap.org/releases/cyrus-imapd-3.0.0-beta5.tar.gz
http://www.cyrusimap.org/releases/cyrus-imapd-3.0.0-beta5.tar.gz.sig

ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-3.0.0-beta5.tar.gz
ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-3.0.0-beta5.tar.gz.sig

Please consult the release notes before upgrading to 3.0.0-beta5:


http://www.cyrusimap.org/dev/imap/download/release-notes/3.0/x/3.0.0-beta5.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to
report issues, join in the deliberations of new features for the next
Cyrus IMAP release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

Ellie Timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: 2.5.10 autocreate on login not working?

[ellie timoney via Info-cyrus]

Hi Nels,

The autocreate code in 2.5 is full of LOG_DEBUG syslogs -- I guess you
don't have this turned on, or you're logging that log level to a
different file.  It would be helpful to see this output.

I think you can turn this logging on in Cyrus by adding "debug: yes" to
your imapd.conf.  You might need to do something with your syslog
configuration too -- it's been a while since I set this up myself and I
don't remember the details offhand.

Cheers,

ellie

On Wed, Dec 7, 2016, at 09:49 AM, Nels Lindquist via Info-cyrus wrote:
> Hi, Bron.
> 
> On 2016/12/05 3:21 PM, Bron Gondwana via Info-cyrus wrote:
> 
> > Is there anything in syslog?
> 
> Just the record of the login:
> 
> Dec  6 17:44:10 rapier imap[13351]: login: rapier [198.50.220.242] nels 
> DIGEST-MD5 User logged in 
> SESSIONID=
> 
> Or an encrypted session:
> 
> Dec  6 17:47:57 rapier imap[13176]: login: rapier [198.50.220.242] nels 
> DIGEST-MD5+TLS User logged in 
> SESSIONID=
> 
> Nels Lindquist
> 
> 
> 
> > On Tue, 6 Dec 2016, at 05:40, Nels Lindquist via Info-cyrus wrote:
> >> I'm experimenting with a build of 2.5.10 on CentOS 7 in preparation for
> >> upgrading from our installed 2.3.16 (built from Simon Mattar's RPMs).
> >>
> >> We rely on mailbox auto-creation functionality tied to LDAP
> >> authentication with virtual domain support; as long as the LDAP account
> >> exists the mailbox may be autocreated.
> >>
> >> As the autocreate patch has now been incorporated in 2.5, I was hoping
> >> it would work fairly seamlessly, but even after updating all the
> >> deprecated imapd.conf directives, I'm having trouble.
> >>
> >> I'm able to log in to IMAP successfully with an existing LDAP account,
> >> but a LIST command produces no output, and if I log in with cyradm the
> >> expected mailboxes are not present.  I'm able to create mailboxes
> >> manually with cyradm and everything works as expected, but I really need
> >> autocreate to work.
> >>
> >> Here's the relevant section of my imapd.conf:
> >>
> >> virtdomains: yes
> >> defaultdomain: example.com
> >> username_tolower: yes
> >> lmtp_downcase_rcpt: yes
> >> autocreate_quota: 0
> >> autocreate_quota_messages: 0
> >> autocreate_inbox_folders: Drafts|Sent|Trash
> >> autocreate_subscribe_folders: Drafts|Sent|Trash
> >> autocreate_post: yes
> >>
> >> Anything I've done obviously wrong?
> >>
> >> 
> >> Nels Lindquist
> >> 
> >> 
> >> Cyrus Home Page: http://www.cyrusimap.org/
> >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> >> To Unsubscribe:
> >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> >
> >
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: 2.5.10 autocreate on login not working?

[ellie timoney via Info-cyrus]

On Tue, Dec 13, 2016, at 07:04 AM, Nels Lindquist via Info-cyrus wrote:
> On 2016/12/12 4:06 AM, ellie timoney wrote:
> 
> >> Any ideas?
> >
> > No, unfortunately.
> >
> > We have some basic syslog instructions (aimed at developers setting up a
> > Cyrus development environment), but it sounds like you've probably got
> > this stuff covered already.  This was enough to get my dev setup running
> > (Debian):
> > http://cyrusimap.org/dev/imap/developer/installguide.html#setting-up-syslog
> >
> > Hopefully someone who's familiar with syslog as an admin rather than as
> > a dev can chime in with some help here?
> 
> That syslog setup for devs is pretty much identical to what I have set 
> up, so I'm not sure why I'm not seeing anything related to autocreate in 
> the logs.
> 
> Are there any options for debug in imapd.conf, eg. a numeric parameter 
> for different levels or sections of logging, or is it an all-or-nothing 
> option?

I don't believe so.  All levels of logging are programmatically enabled
by default, and then the following lines turn the LOG_DEBUG level -off-
if debug=no in imapd.conf (which is also its default value if not set):
https://github.com/cyrusimap/cyrus-imapd/blob/cyrus-imapd-2.5/imap/global.c#L208-L210

Has this been altered in the source package you're building from?

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: SOLVED (was: Re: 2.5.10 autocreate on login not working?)

[ellie timoney via Info-cyrus]

Oh awesome, thanks for the update :)

On Tue, Dec 13, 2016, at 07:23 AM, Nels Lindquist via Info-cyrus wrote:
> Okay I figured it out.
> 
> I was working with the Kolab packages, and went back and checked their 
> distributed .spec file again, and the --enable-autocreate option was not 
> present in the configuration section.  I added it, rebuilt the packages 
> and now it's working as expected.
> 
> My apologies for the wild goose chase!
> 
> Nels Lindquist
> 
> 
> 
> On 2016/12/12 1:04 PM, Nels Lindquist via Info-cyrus wrote:
> > On 2016/12/12 4:06 AM, ellie timoney wrote:
> >
> >>> Any ideas?
> >>
> >> No, unfortunately.
> >>
> >> We have some basic syslog instructions (aimed at developers setting up a
> >> Cyrus development environment), but it sounds like you've probably got
> >> this stuff covered already.  This was enough to get my dev setup running
> >> (Debian):
> >> http://cyrusimap.org/dev/imap/developer/installguide.html#setting-up-syslog
> >>
> >>
> >> Hopefully someone who's familiar with syslog as an admin rather than as
> >> a dev can chime in with some help here?
> >
> > That syslog setup for devs is pretty much identical to what I have set
> > up, so I'm not sure why I'm not seeing anything related to autocreate in
> > the logs.
> >
> > Are there any options for debug in imapd.conf, eg. a numeric parameter
> > for different levels or sections of logging, or is it an all-or-nothing
> > option?
> >
> > Nels Lindquist
> > 
> > 
> > 
> > Cyrus Home Page: http://www.cyrusimap.org/
> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > To Unsubscribe:
> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.0.0-beta6 released

[ellie timoney via Info-cyrus]

The Cyrus team is proud to announce the immediate availability of the
sixth beta from the Cyrus IMAP 3.0 series: 3.0.0-beta6.

As this is a beta release it is not considered stable for production
usage.  Interfaces, APIs, features, etc are likely to change.

Download URLs:

http://www.cyrusimap.org/releases/cyrus-imapd-3.0.0-beta6.tar.gz
http://www.cyrusimap.org/releases/cyrus-imapd-3.0.0-beta6.tar.gz.sig

ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-3.0.0-beta6.tar.gz
ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-3.0.0-beta6.tar.gz.sig

Please consult the release notes before upgrading to 3.0.0-beta6:


http://www.cyrusimap.org/dev/imap/download/release-notes/3.0/x/3.0.0-beta6.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to
report issues, join in the deliberations of new features for the next
Cyrus IMAP release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

Ellie Timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.0.0-rc1 released

[ellie timoney via Info-cyrus]

The Cyrus team is proud to announce the immediate availability of the
first release candidate from the Cyrus IMAP 3.0 series: 3.0.0-rc1.

As a release candidate, it is considered near-stable for production
usage.   Interfaces, APIs, features, etc are not likely to change
between now and the full release.

If you plan to upgrade to 3.0, we recommend starting to test your
upgrade process now.  Feedback and bug reports will be greatly
appreciated!

Download URLs:

http://www.cyrusimap.org/releases/cyrus-imapd-3.0.0-rc1.tar.gz
http://www.cyrusimap.org/releases/cyrus-imapd-3.0.0-rc1.tar.gz.sig

ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-3.0.0-rc1.tar.gz
ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-3.0.0-rc1.tar.gz.sig

Please consult the release notes and upgrade documentation before
upgrading to 3.0.0-rc1:


http://www.cyrusimap.org/dev/imap/download/release-notes/3.0/x/3.0.0-rc1.html
http://www.cyrusimap.org/dev/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to
report issues, join in the deliberations of new features for the next
Cyrus IMAP release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Is guid_mode: sha1 really gone from imapd.conf?

[ellie timoney via Info-cyrus]

For what it's worth, looking at commit history, it looks like it went
away sometime before 2.4.0

On Tue, Jan 17, 2017, at 09:41 AM, Bron Gondwana via Info-cyrus wrote:
> Yep, it's gone away - sha1 is the only mode right now.  We're going to
> want to look at blake2 soon though I reckon.
> 
> Bron.
> 
> On Tue, 17 Jan 2017, at 03:48, Andy Dorman via Info-cyrus wrote:
> > So the subject is the question...
> > 
> > Running cyr_info conf-lint against our imapd.conf indicates "guid_mode" 
> > is gone and indeed, the imapd.conf man page doesn't mention it now.
> > 
> > But many older versions of the man page do mention it and I can not find 
> > any mention in past release notes at
> > 
> > http://www.cyrusimap.org/dev/imap/download/release-notes/index.html
> > 
> > of it being deprecated or removed.
> > 
> > So is it really gone or could it have been accidentally left out of the 
> > man page and conf-lint check code? I agree that is very unlikely, but 
> > with all the massive amount of excellent work that has been done by the 
> > Cyrus team in the last few years, sometimes a small thing like this can 
> > fall through a crack.
> > 
> > Hopefully it will be OK just to remove the "guid_mode" sha1" line and 
> > restart and nothing will explode?
> > 
> > Thanks and my apologies if the above is a really dumb question.
> > 
> > -- 
> > Andy Dorman
> > Ironic Design, Inc.
> > AnteSpam.com
> > 
> > 
> > Cyrus Home Page: http://www.cyrusimap.org/
> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > To Unsubscribe:
> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> 
> -- 
>   Bron Gondwana
>   br...@fastmail.fm
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus IMAP 3.0.0-rc1 Config Lint Surprise

[ellie timoney via Info-cyrus]

This is now fixed, thanks Andy for the report and Nic for the pull
request :)

On Tue, Jan 17, 2017, at 09:42 AM, Bron Gondwana via Info-cyrus wrote:
> Thanks Nic, yeah - that page was back to front.  Good find Andy.
> 
> Bron.
> 
> On Tue, 17 Jan 2017, at 04:01, Nic Bernstein via Info-cyrus wrote:
> > Thanks for the heads-up.  We'll get this fixed, it is the man page which 
> > is correct.
> >  -nic
> > 
> > On 01/16/2017 10:19 AM, Andy Dorman via Info-cyrus wrote:
> > > On 01/13/2017 12:07 AM, ellie timoney via Info-cyrus wrote:
> > >> The Cyrus team is proud to announce the immediate availability of the
> > >> first release candidate from the Cyrus IMAP 3.0 series: 3.0.0-rc1.
> > >>
> > >> As a release candidate, it is considered near-stable for production
> > >> usage.   Interfaces, APIs, features, etc are not likely to change
> > >> between now and the full release.
> > >>
> > >> If you plan to upgrade to 3.0, we recommend starting to test your
> > >> upgrade process now.  Feedback and bug reports will be greatly
> > >> appreciated!
> > >>
> > >
> > > Hi everyone.  We started the process of checking our Cyrus configs in 
> > > preparation for testing our dev/beta Debian Cyrus install (2.5.10-3) 
> > > and ran into an issue right away.
> > >
> > > I believe the instruction line at this URL:
> > >
> > > http://www.cyrusimap.org/dev/imap/download/upgrade.html#copy-config-files-and-update
> > >  
> > >
> > >
> > > for lint checking the cyrus & imapd configs have the file types 
> > > backwards.  The instruction shows:
> > >
> > > cyr_info conf-lint -C  -M 
> > >
> > > But according to the man page the command should look like this:
> > >
> > > cyr_info [ -C alt imapd.conf ] [ -M alt cyrus.conf ] command
> > >
> > > And indeed, if you run the command as shown on the web page you get an 
> > > error caused by the first line of the START section.
> > >
> > > /usr/lib/cyrus/bin/cyr_info conf-lint -C /etc/cyrus.conf -M 
> > > /etc/imapd.conf
> > > fatal error: invalid option name on line 6 of configuration file 
> > > /etc/cyrus.conf
> > >
> > > If I run it as the man page suggests, the output looks a little more 
> > > reasonable. (and I have a little bit of work to do :-)
> > >
> > > Hope this helps.
> > >
> > 
> > -- 
> > Nic Bernstein n...@onlight.com
> > Onlight Inc.  www.onlight.com
> > 6525 W Bluemound Rd., Ste 24  v. 414.272.4477
> > Milwaukee, Wisconsin  53213-4073  f. 414.290.0335
> > 
> > 
> > Cyrus Home Page: http://www.cyrusimap.org/
> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > To Unsubscribe:
> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> 
> -- 
>   Bron Gondwana
>   br...@fastmail.fm
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.0.0-rc2 released

[ellie timoney via Info-cyrus]

The Cyrus team is proud to announce the immediate availability of the
second release candidate from the Cyrus IMAP 3.0 series: 3.0.0-rc2.

As a release candidate, it is considered near-stable for production
usage.   Interfaces, APIs, features, etc are not likely to change
between now and the full release.

If you plan to upgrade to 3.0, we recommend starting to test your
upgrade process now.  Feedback and bug reports will be greatly
appreciated!

Download URLs:

http://www.cyrusimap.org/releases/cyrus-imapd-3.0.0-rc2.tar.gz
http://www.cyrusimap.org/releases/cyrus-imapd-3.0.0-rc2.tar.gz.sig

ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-3.0.0-rc2.tar.gz
ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-3.0.0-rc2.tar.gz.sig

Please consult the release notes and upgrade documentation before
upgrading to 3.0.0-rc2:


http://www.cyrusimap.org/dev/imap/download/release-notes/3.0/x/3.0.0-rc2.html
http://www.cyrusimap.org/dev/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to
report issues, join in the deliberations of new features for the next
Cyrus IMAP release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus