Re: Test version of the cyrus-imapd-2.4.6 Invoca RPM package

[Greg Banks]

On 19/01/11 22:48, Adam Tauno Williams wrote:
> On Wed, 2011-01-19 at 07:17 +0100, Simon Matter wrote:
>>> On Tue, Jan 18, 2011 at 02:59:53PM +0100, Simon Matter wrote:
 I've attached my version of a patch to the bugzilla entry. It is also in
 the just released 2.4.6-1 RPM.
>>> Woot :)
>>> Have you considered learning a bit of git and making patches directly?
>> Yes I have considered it - like so many other things. I guess I'll do it
>> just after digging into IPv6 a bit deeper :)
> Do git first. :)
>
> IPv6 is actually really nice; certainly more elegant than than IPv4.  It
> makes sense.

Yeah, sure it does.

http://arstechnica.com/business/news/2010/09/there-is-no-plan-b-why-the-ipv4-to-ipv6-transition-will-be-ugly.ars

> git makes people cry.

Oh I dunno.  Git used to make me scream, but I got used to it.  Memories 
of the IPV6_V6ONLY sockopt on Irix still make me want to cry.

-- 
Greg.


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: mailbox referrals in 2.4.12

[Greg Banks]

On Fri, Dec 16, 2011, at 02:05 PM, Andrew Morgan wrote:
> 
> I have never looked at Cassandane before (neat stuff!).
> 
> There were two tests I ran to verify referrals were disabled:
> 
> 1. Check that MAILBOX-REFERRALS is not present in the CAPABILITIES string 
> when connecting.

Easy enough to do, you can start by copying code from any of the several tests 
in Cassandane::Cyrus::Conversations which do something similar:

sub test_append
{
my ($self) = @_;
my %exp;

# check IMAP server has the XCONVERSATIONS capability
$self->assert($self->{store}->get_client()->capability()->{xconversations});



> 
> 2. Verify that a mailbox referral is not given:
> 
>a. Connect to Cyrus as a regular user
>b. RLIST "" "*"
>c. CREATE "test"
>d. DELETE "test"
> 
> When referrals are allowed, step 2d (DELETE "test") will cause a referral 
> to be generated.

Also reasonably straightforward, there are several tests which create and 
delete mailboxes, e.g. in Cassandane::Cyrus::Quota

sub test_using_storage
{
my ($self) = @_;

...
my $talk = $self->{store}->get_client();

$talk->create("INBOX.sub") || die "Failed to create subfolder";
...
# delete subfolder
$talk->delete("INBOX.sub") || die "Failed to delete subfolder";
}

You may also find $talk->get_last_completion_response(), and 
$talk->get_response_code('referral') useful.


> 
> Of course, these tests only make sense in a Murder.
> 

Ah, now here you're heading off the beaten path a little bit.  We haven't 
currently got any Murder tests in Cassandane.  Now Cassandane is designed with 
multi-instance setups in mind, and we do test replication with a pair of 
instances, so it's entirely possible, but you'll be pioneering.

You need to be basically familiar with the Cassandane::Config and 
Cassandane::Instance classes; there's some documentation for them in 
doc/adding_tests.txt.  What you'll be doing is creating two or more Instance 
objects with carefully tuned Configs which set them all up in a Murder.  The 
code which does this for all the existing cases is in 
Cassandane::Cyrus::TestCase::_create_instances().  You'll want to copy that 
code and tweak it, and also disable the original by passing an argument to the 
TestCase constructor thusly

sub new
{
my $class = shift;
return $class->SUPER::new({
instance => 0,
}, @_);
}

I would also suggest writing a test for the non-Murder case first just to get 
the hang of the Cassandane environment.

Do let me know if you have any difficulty or need any more help.

-- 
Greg.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: [SOLVED] Re: Set quota "none" using Cyrus::IMAP::Admin

[Greg Banks]

G'day,

On Tue, Jan 17, 2012, at 01:18 PM, Wolfgang Breyha wrote:
> Ram wrote, on 17.01.2012 12:17:
> > I am using cyrus-imapd-2.4.12-2
> > As Wolfgang suggested using
> > $imap->setquota("user/$user") removed the quota.
> 
> Looking at the code of Cyrus::IMAP::Admin there is no code checking for
> "none"
> or something like that. There are two ways to "craft" a proper IMAP
> command.
> First the way I described setting no STORAGE argument at all resulting
> in:
> SETQUOTA id ()
> to remove quota.

That's the correct way according to RFC2087.

> And second setting
> ->setquota(<-id->, "STORAGE", "-1")
> resulting in
> SETQUOTA id (STORAGE -1)
> what Bron suggested. That should also work IMO. At least on 2.4.12+.

And that isn't correct according to RFC2087.  It works only because the
server is deliberately not enforcing syntax correctly, to workaround a bug in a
different part of the code.

> The man page of Cyrus::IMAP::Admin does not describe how to remove quota
> at
> all. Neither "none" nor anything else. Nothing wrong in that part so far.

You're right, that documentation is quite unclear.

This reminds me, we need to update the documentation on the master branch
to mention the other resources we support there.

-- 
Greg.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: total space = (used quota) + (to be expunged)?

[Greg Banks]

On Mon, Feb 6, 2012, at 09:58 PM, Andre Felipe Machado wrote:
> Hello
> After reading some archived list msg [1], and database formats doc
> (quotas 
> topic)[2], I realized that there is not a "high level" cyrus command to
> display 
> total disk/partition space actually used at a Cyrus imap backend.

That's correct, we don't track that information, because doing it
properly would be hard.  We could track the size of messages in
expunged-but-not-unlinked state and accumulate them for display,
but accounting for the space used by database backends would be
quite hard.

> In order to get exact (approximated when single instance store is
> enabled) to be 
> expunged messages space should a du / df be used ?
> 
> df_used_blocks - used_quota = space_to_be_expunged

Sure.

> Is there a better approach (more exact, faster, lower system load)?

Sadly, no.

> What is the correct way to account for the single instance store effect?
> df or du -L ?
> (du -L)-(du)?

I'm confused why you would want to use du -L.  The -L follows symlinks; Cyrus 
doesn't use symlinks internally at all (except for sieve scripts).  The 
"singleinstancestore" option is implemented using hardlinks, and du already 
accounts for those properly assuming you include all the links in the file 
trees seen by du.

> At big partitions (with millions of messages) du may cause a high disk
> load...
> Suggestions?

Only the obvious: you could use df, and not have anything else
but Cyrus data on the partition.  This works for us :)

-- 
Greg.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: total space = (used quota) + (to be expunged)?

[Greg Banks]

Sent from my iPhone

On 08/02/2012, at 0:51, Adam Tauno Williams   
wrote:

> Quoting Andre Felipe Machado :
>
>> Hello
>> After reading some archived list msg [1], and database formats doc  
>> (quotas
>> topic)[2], I realized that there is not a "high level" cyrus command
>> to display
>> total disk/partition space actually used at a Cyrus imap backend.
>> In order to get exact (approximated when single instance store is
>> enabled) to be
>> expunged messages space should a du / df be used ?
>> df_used_blocks - used_quota = space_to_be_expunged
>
>  is possibly  
> related

Indeed, and we certainly could do that, although it involves a  
cyrus.index header format change, which is unnecessarily painful with  
the current data format. The question is how much value would you get  
from it, given that the user's contribution to the cyrus.cache,  
cyrus.header, mboxlist, annotations (and on the fastmail branch,  
conversations) dbs are not accounted for.

Greg.
>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: deleting top-level INBOX

[Greg Banks]

Sent from my iPhone

On 22/02/2012, at 7:23, Vilius Šumskas  wrote:

> Hi,
>
>  I have accidently created top level "INBOX.Spam" mailbox in Cyrus. I
>  have managed to delete the Spam subfolder, but when deleting INBOX I
>  get:
>
>  localhost> dm INBOX
>  deletemailbox: Operation is not supported on mailbox

Cyrus is obeying the IMAP RFC which says that it is an error to try to  
delete your INBOX.

http://tools.ietf.org/html/rfc3501#section-6.3.4

Why not? I'm sure there was a perfectly good reason at the time.

>
>  All needed permissions is there:
>
>  localhost> lam INBOX
>  cyrus lrswipkxtecda
>
>  Is there a way to delete this top-level INBOX?
>

Be an admin user that doesn't own the mailbox, and use "dm user.foo".

Good luck,

Greg.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: Cyrus 2.4 IMAP SEARCH

[Greg Banks]

G'day,

As I'm looking at the Cyrus search code for other reasons, I thought I'd
field this one.

On Fri, Jun 29, 2012, at 12:57 PM, David Carter wrote:
> 
> RFC 3501, section 6.4.4 "SEARCH Command" says:
> 
> In all search keys that use strings, a message matches the key if
> the string is a substring of the field.  The matching is
> case-insensitive.
> 
> and I guess "" is strictly speaking a substring of all other strings.
>

Ah, the good old days when RFCs could be written assuming one language,
four timezones and one hemisphere and everybody Just Knew What You
Meant.  As far as I can see there is absolutely no definition of either
"substring" or "case-insensitive" in RFC3501 or any of the documents it
references.

These days however we have Unicode and in particular RFC 5051 which
tells us how to handle text when sorting and searching.  That document
references RFC 4790, section 4.2.3 of which has a useful working
definition of what a comparator's substring operation can do, including
this

>   A string is a substring of itself.  The empty string is a substring of all 
> strings.

which is quite clear.  So your guess is correct, and Cyrus 2.4 is
behaving incorrectly.  Please raise a Bugzilla.

> Was this a deliberate change

It's difficult to tell without figuring out who changed it and asking
them, but I'm guessing "no" :)

>  or just a consequence of the search engine 
> being reworked in 2.4? I don't imagine that it is a very common problem.

It does seem like a corner case.

-- 
Greg.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: RFC 5464 IMAP METADATA Extension Errata

[Greg Banks]

G'day,

On 30/07/2012, at 7:20, kael  wrote:

> On 07/26/2012 07:28 AM, Bron Gondwana wrote:
>> https://bugzilla.cyrusimap.org/show_bug.cgi?id=3723
> 
> 
> Implementing unsolicited METADATA responses at the same time would be 
> awesome, though.

Sheesh, give 'em an inch...

When I was doing the last round of annotation code rewrites, I deliberately 
avoided implementing unsolicited responses, because it's hard to do properly 
with the existing database structure. We'd need to store something like a 
modseq per annotation. The old format stored a timestamp because one of the 
early annotatemore drafts said that clients could access a modified time per 
annotation. That draft is obsolete now and we don't store modtimes anymore. In 
any case using timestamps to report changes is problematical, which is why 
modseqs are used to do that job for messages. Unfortunately, there's no single 
modseq-like counter in Cyrus which works for per-server, per-mailbox and 
per-message annotations.

But if there's a good use case for it, we could implement such a thing.

Greg.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: RFC 5464 IMAP METADATA Extension Errata

[Greg Banks]

On Mon, Jul 30, 2012, at 09:36 AM, Greg Banks wrote:
> G'day,
> 
> On 30/07/2012, at 7:20, kael  wrote:
> 
> > On 07/26/2012 07:28 AM, Bron Gondwana wrote:
> >> https://bugzilla.cyrusimap.org/show_bug.cgi?id=3723
> > 
> > 
> > Implementing unsolicited METADATA responses at the same time would be 
> > awesome, though.
> [...]
> 
> But if there's a good use case for it, we could implement such a thing.

I should mention also that Bron has been talking for some time now about
storing modseqs on per-message annotations as a way to make replication
more efficient.  I think this is an excellent idea and is easy enough to
do, because we have a good modseq counter with sensible semantics for
the mailbox and could store it in the db in a semi backwards compatible
manner, and the CONDSTORE rfc allows for servers to keep separate
modseqs on any kind of per-message data.  Once we do that, it would be
relatively easy to emit unsolicited ANNOTATION responses.

It might be feasible to do similar things for per-mailbox annotations. 
But per-server annotations are...less obvious.

-- 
Greg.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Question about 2.4.x mailbox rename/locking behaviour

[Greg Banks]

On Fri, Aug 10, 2012, at 06:04 AM, Bill Ryder wrote:
> Hi all,


Hi Bill,

Sorry for the slow reply, I was away Friday.

> We have two problems with cyrus in our environment which is constantly
> used as ammunition against it.
> 
> We are running 2.3.16 right now.
> 
> The problems are
> 
> 1. Can not unreserve a mailbox without restarting cyrus - these can be
> caused by number 2 below
> 2. Mailbox renames - or deletes - can take a LONG time - and delivery
> stops eventually.
> [...]
> Has anything changed in 2.4 that will stop this kind of thing from
> happening?

So I joined the project at the point when 2.3 was already in the past,
but on examining the source I see that both 2.4 and the development
branch that will be 2.5 both have the same behaviour you describe.  Each
individual file in a folder in linked to its new location and unlinked
again from it's old location, while keeping the mailbox lock.  This is a
long way south of optimal for the common case where an entire user is on
the same filesystem, but that code path is designed to handle all the
cases and handle them safely.

> Depending on fileserver load etc it can take 30 minutes to do one of
> these moves (artists often have wacom email folder drag and 
> drop events which can start huge unexpected mailbox renames)

Cyrus' design generally assumes that folder management is something you
do rarely and deliberately, because this is true for the majority of
people.   Perhaps your email client can be convinced to disable folder
renames via dragging?

OTOH, there's no really good reason for renames to be slow in the common
case, and we've fixed worse problems. We have spoken about improving
folder deletes and renames several times, but there's been no real
impetus.  Raise a bug and we can have a discussion there.

> if the mailbox is large I have to kill the imapd doing the move and clean it
> up. This leads to a reserved mailbox that I can't clear without a reboot.

In 2.4 mailbox reserving is no longer used (unless you're running a
Cyrus murder).

-- 
Greg.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus 2.5

[Greg Banks]

On Wed, Jan 30, 2013, at 11:04 PM, Bron Gondwana wrote:
> On Wed, Jan 30, 2013, at 07:06 PM, Pepe Charli wrote:
> > Hi
> > 
> >  Is there any estimate about when we can expect cyrus 2.5?
> 
> Soon, honestly.  We keep getting caught up with other things.
> 
> >  Will it include the search engine based on Xapian?
> 
> Probably not.

Definitely not.  The Xapian support is based on a large lump of work
which adds a completely new message_t object API.  This is far too large
a change to be pushing into 2.5 at this stage.  This is how far ahead
the fastmail branch is relative to cmu/master:

 150 files changed, 41392 insertions(+), 4627 deletions(-)

Also, we want to get some more operational experience with Xapian before
inflicting it on other folks.  If you're adventurous the code is out
there at https://github.com/gnb/cyrus-imapd/tree/fastmail

>   It's time to draw a line in the sand and throw 2.5 over
> the fence,

Absolutely.

> Hopefully I can talk to Greg about those next week when he's back in
> the office, and we can get the couple of other fixes he's had sitting
> around into the tree.
> 
> Then it's just documentation time!

Bron and I are spending this week actively scavenging changes which can
be pushed into 2.5 from our various branches.

-- 
Greg.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus