ctl_mboxlist -u not working properly

[Pascal Gienger]

I just did a hard test by just dumping the mailboxlist to a file via

ctl_mboxlist -d > mailboxlistdump

The file "mailboxlistdump" does carry the right information then, one
mailbox per line with

namepartitionuseracluser2acl2 and so on

As test, i deleted the db/ directory and the "mailboxes.db" file (Sleepycat
DB3 format).
After running

ctl_mboxlist -u < mailboxlistdump

I got a new "mailboxes.db" and a "ctl_mboxlist -d" gives THE SAME 
dump as before.
So I thought things are going well.

But nothing worked. The server always replyed "no such mailbox"... (i started
the server _after_ having reconstructed the mailboxes-file via 
ctl_mboxlist -u).

Then i did a telnet to localhost 143 (the cyrus server) and tried
a

. login ... ...
. OK 

. SETACL INBOX pascal lrwspicda

and "hop" the user.pascal (INBOX) mailbox worked again!

Does "ctl_mboxlist -u" does forget something about ACLs or what is this
behaviour all about?

Pascal
-- 

Re: peculiar lmtpd behavior

[Pascal Gienger]

>We are experiencing a behavior in lmtpd such that it inserts
>an "@" sign and hostname in the Return-Path if the Return-Path
>does not contain an "@" sign. So bounces on our system have a
>Return-Path with <@hostname> and no username.
>
>(As an aside, we are running qmail as opposed to sendmail.)
>
>We are investigating this further and would appreciate
>any assistance in understanding this behavior and
>identifying a resolution.

As a first hint, I think you must search at qmail's configuration because
Cyrus cannot produce a bounce, can it?

Cyrus only understands LMTP and IMAP and both of them do not generate bounces
(Sieve Scripts are a different story, because they are using
/usr/lib/sendmail when forwarding mail).

It would be nice if you could tell us your configuration (how did you
couple QMAIL with Cyrus? via /var/users/assign?) and then we can look if we
can help you (sorry for the "we" as I am very new to this mailing list but
very "old" when it goes to Cyrus IMAPD things, I am using this beast from Ver-
sions 1.5 on and now we have 2.0.16 working like a charm).

Pascal Gienger
Finesse Networks, Germany
-- 

Re:

[Pascal Gienger]

>imap:/var/imap #  /usr/cyrus/bin/ctl_mboxlist -u < mailboxes
>fatal error: can't initialize mboxlist environment
>imap:/var/imap #  /usr/cyrus/bin/ctl_mboxlist -c
>fatal error: can't initialize mboxlist environment
>imap:/var/imap #
>
>this errors i got ... mboxlist environment, this looks like that the imap
>file structure are damaged or ?

Hoping you are talking about Cyrus 2.0.x:

There can be a simple reason: Do you run "ctl_mboxlist" as
the cyrus imap user? These programs are checking the effective user id of
the process and are denying it if it is 0 (root). But normally (looking at
the source code), it should give "fatal error: must run as the Cyrus user"
then...

Second explanation try:
It means that there is an inconsistency with your mailboxes.db file (the
name(s) depend on your Database library lying under it). If you compiled
Cyrus with DB3, then you can do a

db_dump /var/imap/mailboxes.db
(assuming your cyrus config is in /var/imap)

You should get something like

VERSION=3
format=bytevalue
type=btree
HEADER=END
[... many other info...]

For other db, dbm or gdbm database types (if you have mailboxes.pag and
mailboxes.dir, you have the dbm library) there should be appropriate
commands.
Try also

ctl_mboxlist -d

to see if it works.

As posted before, I also noticed some problems with ctl_mboxlist -u.
Backup your /var/imap/mailboxes.db (or.pag and .dir) and /var/imap/db/*.
Delete all items in /var/imap/db afterwards and your mailboxes.db (or .pag
and .dir). Do a "touch /var/imap/mailboxes.db" (or
touch /var/imap/mailboxes.pag /var/imap/mailboxes.dir) and run

ctl_mboxlist -u < mailboxes

afterwards. Face yourself to the fact that you have to redo all ACLs 
afterwards (see my former posting to this list).

Finally.

Pascal
-- 

Re: peculiar lmtpd behavior

[Pascal Gienger]

>In our configuration the set of users that can receive mail locally is
>the contents of the assign file.
>Messages are passed to deliver via the users' .qmail files.
>
>The transformation of the Return-Path happens after qmail has
>handed off the message to cyrus for final delivery.
>
>We believe it's this bit of code from savemsg() in lmtpengine.c:
>
> if (!strchr(rpath, '@')) {
> hostname = config_servername;
> }
> fprintf(f, "Return-Path: <%s%s%s>\r\n",
> rpath, hostname ? "@" : "", hostname ? hostname : ""
>
>If anyone can help us understand the reasons for this, we'd
>greatly appreciate it.


Ah NOW I understand. You mean the final delivery of a bounce generated by
any other MTA to the sender of the erroneous message (the sender has 
a cyrus imap mailbox). Yes then you are right, I think this is a bug.

There should be some matching for "<>" as Return-Path in lmtpengine.c, because
this is the only correct Return-Path for an automatically generated bounce
message... :(

Pascal
-- 

Re: your DB3/Cyrus problems

[Pascal Gienger]

>I got the following messages if i want start the /usr/cyrus/bin/master &
>process

[...]

>Sep 22 18:33:49 imap imapd[3665]: DBERROR db3: region error detected; run
>recovery.
>Sep 22 18:33:49 imap imapd[3665]: DBERROR: dbenv->open '/var/imap/db'
>failed: DB_RUNRECOVERY: Fatal error, run database recovery

That is clear, isn't it? ,-)
The underlying DB3 library is complaining about a corrupted .db-File (.db
is not in sync with the db/* journal and log files (*) ).

In your DB3 distribution there should be a program called "db_recover".

Backup /var/imap/db (the directory) and /var/imap/mailboxes.db.

Run:
db_recover -h /var/imap/db

That's it.

Start the master then and it will work.

Pascal

(*) DB3 does not only use the .db file but also a "db" subdirectory in your
 database path. So Cyrus with DB3 compiled in will use:
 /var/imap/mailboxes.db
 /var/imap/db/db...-files and
 /var/imap/db/log..-files.


-- 

Re: AW: your DB3/Cyrus problems

[Pascal Gienger]

>hello pascal,
>
>thank you for your fast answer!
>i've done this but at this time i got this message:

[...]


>Sep 22 20:15:25 imap ctl_mboxlist[3893]: running mboxlist recovery
>Sep 22 20:15:25 imap ctl_mboxlist[3893]: DBERROR db3: Invalid log file:
>log.01: No such file or direc
>tory

It complains about not finding the transaction log of your DB3 database.
What did you do with your database?

You may still try the hard way via a "catastrophic recovery":

db_recover -c -h /var/imap/db

If you are new to DB3, I suggest reading the manual pages for these
db_-utilities, available here:

http://www.sleepycat.com/docs/utility/

Pascal Gienger
-- 

Re: AW: your DB3/Cyrus problems

[Pascal Gienger]

>In my opinion the "tools/mkimap" will create the directorys under /var/imap/
>... or ???
>
>But if i start this tool, i get the following message:

That is _NOT_ your mailboxes file. It will "only" create your mailstore
folders.

Pascal
-- 

Re: AW: your DB3/Cyrus problems

[Pascal Gienger]

>Pascal Reinheimer wrote"
>>  After i've used "db_recover -c -h /var/imap/db" the master process starts,
>>  but then a client connect to the imap server the following messages
>arrives:
>>
><...>
>>  Sep 22 20:47:46 imap master[4029]: process 4035 exited, signaled to death
>by
>>  11
>>
>Have you checked the FAQ?
>   http://cyrus-utils.sourceforge.net/faq

I don't think that is his problem because his cyrus daemons _DO_ find the
db3-Library, you can see that clearly with the logfile posted from him.

This Sig11 is caused by a NULL pointer returned by a DB3 function not checked
by the cyrus daemon (consequence of an unreadable, corrupted database).

Pascal
-- 

Re: Can't deliver to INBOX.folder

[Pascal Gienger]

>  > sam  anyone p
>I tried looking through the imapd.conf manpage, but I didn't find a 
>option that
>all subfolders of a user got this flag set. AFAIK the flag is set 
>when the user
>creaes a new subfolder.


AFAIK, the correct syntax would be:

deliver -m user.username.folder -u user

If using "deliver" locally (delivery via the local lmtp socket), all
connections will pe preauthenticated as "postman" and accepted by lmtpd.
So it is sufficient for the ordinary user to have "p" right.

Pascal
-- 

Re: sasldb2 and other issues...

[Pascal Gienger]

>Several problems i'm occuring here,
>
>1) with SSL enabled in Cyrus Imapd cyradm fails to run, with a 
>missing library something
>from openssl i believe.  Would i need to recompile Perl to support SSL?

What is the exact word of the error message?
On Solaris you can try to point LD_LIBRARY_PATH (environment variable) to
the OpenSSL Library directory. If it works then you did not link with
-rpath/-R enabled.

On Solaris 8, you can setup hints. They are stored in /var/ld/ld.config.
For 64 bit objects, it is /var/ld/64/ld.config
You can set library search paths in this file (never edit it by hand!) via
the crle(1) command.

If you have dynamic SSL in /usr/local/ssl/lib, you may try:

crle -l /usr/lib -l /usr/openwin/lib -l /usr/local/lib -l /usr/local/ssl/lib \
  -i /usr/lib -i /usr/openwin/lib -i /usr/local/lib -i /usr/local/ssl/lib

This change is permanent.
Now your cyradm should work.

>2) Authentication via saslauthd, i'm having problems getting the 
>authentication to work on
>authentication type of shadow.  It authenticates, yet cyradm wont 
>authenticate without me
>adding a user to sasldb2.


I don't know this.

>
>3) Do i need to add all my users and their passwords to sasldb2?


As I prefer running Cyrus IMAPD as a sealed server I don't have any 
user mail password in /etc/shadow.
The saslpasswd2 command has a command line option for getting the password
via stdin. Nifty for creating hundreds of users via a tiny little script
using random passwords.

>
>4) How about importing the previous mbox's, i have a few users with 
>several hundred megs in
>their mailbox that i would like to import into the cyrus partition.


There are several utilities to split mbox-files into multiple e-mails,
perfectly pipe-able into "deliver".
I am using "splitmail" (forgot where I found that) and I ran (using the user-
name as script argument and having had the cyrus lmtpd locally on the same
machine including the message store):

--- cut here ---
#!/bin/sh

user="$1"
pp=$$
mkdir -p /tmp/$pp

chmod 700 /tmp/$pp
cd /tmp/$pp

splitmail /var/spool/mail/$user
for i in /tmp/$pp/*
do
   /opt/cyrus/bin/deliver $user < $i   # insert your cyrus deliver here
done

rm /tmp/$pp/*
rmdir /tmp/$pp
--- cut here ---




Pascal Gienger
-- 

Re: CyrusIMAP for AMD64 Opteron ??

[Pascal Gienger]

BipinDas <[EMAIL PROTECTED]> wrote:

> Dear List,
>
> Is there any specific source of CyrusIMAP for AMD 64 Opteron series ?. I
> am getting strange error while compiling CyrusIMAP2.3.1 on the above
> said server. The error is as follows.
> -
> -
> --
> LD_RUN_PATH="/usr/lib64:/lib64" gcc  -shared -O2 -g -pipe -Wall
> -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
> --param=ssp-buffer-size=4 -m64 -mtune=generic IMAP.o  -o
> blib/arch/auto/Cyrus/IMAP/IMAP.so ../../lib/libcyrus.a
> ../../lib/libcyrus_min.a   \
>-ldb -lsasl2 -lssl -lcrypto  \
> /usr/bin/ld: cannot find -lssl
> collect2: ld devolvió el estado de salida 1
> make[2]: *** [blib/arch/auto/Cyrus/IMAP/IMAP.so] Error 1
> make[2]: se sale del directorio `/opt/src/cyrus-imapd-2.3.1/perl/imap'

It can't find your 64bit-SSL-libraries to build the perl modules. Is your 
perl up to date? Is the compiler used to build perl the same as the one 
used to compile Cyrus IMAP?

If you can't find the solution, just skip the "perl" target in the 
Makefile, the perl modules are not needed to run cyrus imap.

Which operating system are you using?

Pascal

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[Pascal Gienger]

Vincent Fox <[EMAIL PROTECTED]> wrote:

> Our working hypothesis is that CYRUS is what is choking up at a certain
> activity level due to bottlenecks with simultaneous access to some shared
> resource for each instance.

Did you do a

lockstat -Pk sleep 30

(with "-x destructive" when it complains about the system being 
unresponsive)?


We had that result, among others:

Adaptive mutex block: 2339 events in 30.052 seconds (78 events/sec)

Count indv cuml rcnt nsec Lock   Caller
---
  778  79%  79% 0.00 456354473 0xa4867730 zfs_zget
   61   6%  85% 0.00 466021696 0xa4867130 zfs_zget
8   1%  87% 0.00 748812180 0xa4867780 zfs_zget
   26   1%  88% 0.00 200187703 0x9cf97598 dmu_object_alloc
2   1%  89% 0.00 1453472066 0xa4867de0 zfs_zget
   12   1%  89% 0.00 204437906 0xa4863ad8 dmu_object_alloc
4   1%  90% 0.00 575866919 0xa4867838 zfs_zinactive
5   1%  90% 0.00 458982547 0xa48677b8 zfs_zget
4   1%  91% 0.00 563367350 0xa4867868 zfs_zinactive
3   0%  91% 0.00 629688255 0xa48677b0 zfs_zinactive

Nearly all locks caused by zfs. The Disk SAN system is NOT the bottleneck 
though, having average service times from 5-8 ms, and no wait queue.

456354473 nsecs are 0,456 secs, that is *LONG*.


What's also interestring is tracing open()-calls via dtrace.
Just use this:

#!/usr/sbin/dtrace -s
#pragma D option destructive
#pragma D option quiet

syscall::open:entry
{
self->ts=timestamp;
self->filename=arg0;
}

syscall::open:return
/self->ts > 0/
{
zeit=timestamp - self->ts;
printf("%10d %s\n",zeit,copyinstr(self->filename));
@["open duration"] = quantize(zeit);
self->ts=0;
}

It will show you all files opened and the time needed (in nanosecs) to 
accomplish that. After hitting CTRL-C, it will summarize:

  open duration
   value  - Distribution - count
1024 | 0
2048 |@80
4096 |@1837
8192 |@@   521
   16384 |@@@  602
   32768 |@@@  229
   65536 |@92
  131072 | 2
  262144 | 0
  524288 | 1
 1048576 | 1
 2097152 | 1
 4194304 | 3
 8388608 | 12
16777216 |@51
33554432 | 38
67108864 | 25
   134217728 | 9
   268435456 | 2
   536870912 | 3
  1073741824 | 0

You see the arc memory activity from 4-65 mikroseconds and disk activity 
from 8-33ms. And you see some "big hits" from 0,13 - 0,5 secs (!). This is 
far too much and I did not figure out why this is happening. As more users 
are connecting this "really long opens" become more and more.

We have a Postfix spool running on the same machine and we got some relief 
in deactivating its directory hashing scheme. ZFS is very "angry" about 
having a deep directory structure it seems. But still, these "long opens" 
do occur.


Pascal

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Just in case it is of general interest: ZFS mirroring was the culprit in our case

[Pascal Gienger]

Our latency problems went away like a miracle when we detached one half of 
the mirror (so it is no more a mirror).

Read-Rates are doubled (not per device, the total read rate!), latency is 
cut off. No more latency problems.

When attaching the volume again, resilvering puts the system to a halt - 
reads and writes do block for seconds (!). We will go on directly with Sun 
to solve the problem. Their "lowest I/O-priority to resilver disks" does 
not seem to be effective. It really blocks the kernel and you end up with 
thousand locks in "zfs_zget".

We have two SAN volumes in different buildings which are NOT the 
bottleneck, tests show it.

Pascal

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[Pascal Gienger]

Michael Bacon <[EMAIL PROTECTED]> wrote:

> I have heard tell of funny behavior that ZFS does if you've got
> battery-backed write caches on your arrays.


/etc/system:

set zfs:zfs_nocacheflush=1


is your friend. Without that, ZFS' performance on hardware arrays with 
large RAM caches is abysmal.

Some arrays do have the possibility to ignore these flush requests 
although, but still following them when internal battery storage is faulted 
or in phase of regeneration (alltogether with setting write-through-mode 
on).

Pascal

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[Pascal Gienger]

Rob Mueller <[EMAIL PROTECTED]> wrote:

>
>> About 30% of all I/O is to mailboxes.db, most of which is read.  I
>> haven't personally deployed a split-meta configuration, but I
>> understand the meta files are similarly heavy I/O concentrators.
>
> That sounds odd.
>
> Given the size and "hotness" of mailboxes.db, and in most cases the size
> of  mailboxes.db compared to the memory your machine has, basically the
> OS  should end up caching the entire thing in memory.

Solaris 10 does this in my case. Via dtrace you'll see that open() on the 
mailboxes.db and read-calls do not exceed microsecond ranges. mailboxes.db 
is not the problem here. It is entirely cached and rarely written 
(creating, deleting and moving a mailbox).

Pascal




Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[Pascal Gienger]

Rob Banz <[EMAIL PROTECTED]> wrote:

>
> We went through a similar discussion last year in OpenAFS land, and
> came the same conclusion -- basically, if your filesystem is
> reasonably reliable (such as ZFS is), and you can trust your
> underlying storage not to lose transactions that are in-cache during a
> 'bad event', the added benefit of fsync() may be less than its
> performance cost.

Would'nt it be nice to have a configuration option to completely turn off 
fsync() in Cyrus? If you want, with a BIG WARNING in the doc stating NOT TO 
USE IT unless you know what you doing. :)



Pascal (in train of reconfiguring our SAN to make more cyrus checks)

PS: Putting deliver.db on tempfs seems to be a nice idea, but in current 
cyrus code you may not give extra paths to single cyrus databases. Our 
actual deliver.db on one machine is ca 600 MB tall, so it won't be of any 
problem to store it completely on tmpfs.


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[Pascal Gienger]

Vincent Fox <[EMAIL PROTECTED]> wrote:

> This thought has occurred to me:
>
> ZFS prefers reads over writes in it's scheduling.
>
> I think you can see where I'm going with this.  My WAG is something
> related to Pascal's, namely latency.  What if my write requests to
> mailboxes.db
> or deliver.db start getting stacked up, due to the favoritism shown to
> reads?

I got substantial benefits from setting compression=on and recordsize=32K 
on the filesystem where deliver.db resides. After talking with our SAN 
staff it showed up that storage was our problem - it has problems with 
concurrent write and read calls, the system won't answer read requests if 
the write channel is "full". I don't know whether it is a firmwire issue or 
a non-capability of the storage system.

Lowering ZFS' recordsize and activating compression on that partition cut 
down i/o rate and things are going normal here again.

Thanks to all who helped!

Pascal

PS: The mirror resilvering problem was a misconfiguration of a brocade 
switch... Sometimes you don't see the forest due to so many trees (german 
proverb, "Man sieht den Wald vor lauter Bäumen nicht")...

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus on Solaris at universities?

[Pascal Gienger]

Vincent Fox <[EMAIL PROTECTED]> wrote:

> Just wondering what other universities are runing Cyrus on Solaris?
>
> We know of:
> CMU
> UCSB

University of Konstanz, Germany

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Solaris ZFS & Cyrus : BugID 6535160

[Pascal Gienger]

Vincent Fox <[EMAIL PROTECTED]> wrote:

> With ZFS there are a number of hits with fdsync bugids.  This connects
> back to I believe Pascal Giengers' thread, which I think resulted in him
> turning off ZIL to bump up performance.  This is speculation at this
> point, anyhow here's the BugID information for those what's interested:

I did not turn off ZIL because I did not want to bump my head out... 
Turning off ZIL is not a very good idea. You see, every file system can be 
tuned to be fast turning off all what does guarantee its consistency. 
Nevertheless I did not want to have a fast mail storage which is completely 
unreliable ;-)

The zil lock problem appeared once but disappeared completely like a 
miracle.
I have reasonable figures now on our servers doing reconfiguration on the 
storage side, it appeared to be a random i/o problem on storage, and Cyrus 
is _HEAVY_ random i/o. Not huge data rates but huge seek rates.

You can give your storage system some hints, writes do appear in ZFS as 
sequential ones. Blocks are written in a sequential way  (due to COW), but 
are read in a random way (for sure).



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: fosdem

[Pascal Gienger]

Rudy Gevaert <[EMAIL PROTECTED]> wrote:

> Hi,
>
> Any anybody who is using cyrus heading to Fosdem (that's in Brussels,
> Belgium)?  http://www.fosdem.org
>

I would like very much to come but I did not find a registration link on 
the webpage. And  love Brussels *g*

Pascal

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Endgame: Cyrus big install at UC Davis

[Pascal Gienger]

Vincent Fox <[EMAIL PROTECTED]> wrote:

> Several people have asked for the IDR number from Sun that gave
> us the performance optimizations we needed.
>
> Management says the agreement for the IDR we got prohibits this.

I just opened a case at our Sun Representative so we hope to get this zfs 
patch as well.
filebench varmail leaves us with > 1,5 seconds (!) for an fsync(), which is 
really a big performance hit.





As a side note:
I had some fun with dtrace visualizing read and write access to our zpool 
mirror (ZFS mirror pool) on our mail server, consisting of two fiber 
channel RAID devices. You see that writes are nearly sequential (due to 
ZFS) and reads are random i/o with some hot spots. Sure, you don't have the 
effect of memory caching (ARC) included.
This zpool is used for IMAP storage, queue spool and log files. All 
filesystems are compressed.

You may find the animated GIF (be warned, it is 5 MB in size) in my little 
blog:

http://southbrain.com/south/




Pascal Gienger
-- 
Rechenzentrum Universität Konstanz
Computing Center University of Constance

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Endgame: Cyrus big install at UC Davis

[Pascal Gienger]

Pascal Gienger <[EMAIL PROTECTED]> wrote:

> I just opened a case at our Sun Representative so we hope to get this zfs
> patch as well.
> filebench varmail leaves us with > 1,5 seconds (!) for an fsync(), which
> is  really a big performance hit.

FYI:
I just got a reply from Sun Support in Germany stating that the patch for 
bug 6535160 (the bug vincent fox submitted) will go public on sunsolve 
tomorrow. For x86 the patch will have id 127729-07. For SPARC it will have 
the major number 127728.
So hopefully no need for special customer service patches for this issue.

Pascal

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Endgame: Cyrus big install at UC Davis

[Pascal Gienger]

Vincent Fox <[EMAIL PROTECTED]> wrote:

> If someone else comes up with quicker confirmation please post.

We applied that patch (for x86) yesterday on our machines during 
maintenance. Delivery times are much better and even on a busy system the 
filebench varmail pattern gives 10ms for fsync().

It would be good if the Cyrus folks could include a warning in their README 
that when using Cyrus IMAP on Solaris 10 and ZFS, be sure to apply 
127728-06 resp. 127729-07, depending on your architecture.




Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

[Pascal Gienger]

Jeff Fookson <[EMAIL PROTECTED]> wrote:

> Databases are all skiplist.

As a rule of thumb, do not use skiplist for the duplicate delivery 
suppression database (deliver.db). Even if everybody hates it, use 
BerkeleyDB, Version 4.4.52 or higher. Give it a quite fair amount of shared 
memory. And run cyr_expunge often to prune that database so that no entry 
is older than - say - 3 days.

We have approx 10-15 messages/sec incoming on one node.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus infrastructure performance less than expected

[Pascal Gienger]

Eric Déchaux <[EMAIL PROTECTED]> wrote:

> The older infrastructure can stand the 42 000 concurrent sessions, the
> new one can't : I was expecting each frontend to be able to handle 5 500
> concurrent sessions but they are not. Around 3 000 / 3 500 concurrent
> sessions the frontends begin to SWAP and are not more able keep up the
> load.

Did I undertand correctly: You are virtualizing each component and your 
frontends begin to swap in their virtual environment?
Is there any reason why you don't assign more RAM to them? Does your 
frontend virtual machines run a 64 bit kernel?

We abandoned all Linux for our Cyrus Servers and switched to Solaris 10 
with Zoning and ZFS. We have less concurrent users than you but more 
storage (10 T at the moment).


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus infrastructure performance less than expected

[Pascal Gienger]

Eric Déchaux <[EMAIL PROTECTED]> wrote:

> I would have loved to put Solaris, Zones and Massaging Server here but it
> was not a possibility. Custorme chose was VMware + Linux + Cyrus.

Just as a sidenote:
As closed source is not an option here, we use cyrus imap 2.3.12 on Solaris 
and not messaging server. 10-12 GB RAM is used as ZFS ARC cache, and things 
are going fast enough now.


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Any command to get rapidly ALL annotations?

[Pascal Gienger]

Is there a way to get all annotations with one imap or cyrus command?

We are using annotations here to be able to set an expiration time for spam 
mailboxes (messages older then x days are deleted automatically at night 
with cyr_expire).

To get a tiny statistic we are going through all mailboxes and use 
GETANNOTATION to retrieve possible annotations, which is a time consuming 
progress. GETANNOTATION does not like wildcards like LIST.

Berkeley DB db_dump is not a good idea either, because even with "-p" it 
gives database corruption in certain circumstances and it won't work any 
more when we move to skiplist for the annotation database.

Pascal
-- 
[EMAIL PROTECTED]
http://southbrain.com/


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus vs Dovecot

[Pascal Gienger]

Mathieu Kretchner <[EMAIL PROTECTED]> wrote:

> Ok thanks ! A good point on the cyrus side !
>
> What about the performance ?

Depends on which size your scenario will be.

> Does my next configuration will run correctly ? Which hardware should I
> buy for this activity ?

That's like asking a crystal sphere whether the woman you love is the right 
one to marry... How many users did use your old server? How many incoming 
messages per second? How much storage was used? How many clients did 
connect simulteanously to your old Cyrus server?

> At present, we have a lot of I/O, we wonder if the last version of cyrus
> is improved for this point ?

We stored (before using graylisting) 30 messages/sec via Cyrus on two 
mirrored SAN volumes. 1200 imapd are running on peak times. No murder. 8 TB 
storage; 2TB used. 58,000 mailboxes; 12,000 users.

20 GB RAM, 14 GB ARC (zfs cache). SAN via Sun MPxIO (scsi_vhci).

You see, without knowing the size of your old Cyrus solution it is not 
possible to say whether it is "enough" or not.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus vs Dovecot

[Pascal Gienger]

Adam Tauno Williams <[EMAIL PROTECTED]> wrote:

> If you have allot of users and allot of mail you are going to have allot
> of I/O.  No way around that, regardless of the server.  Cyrus' indexes
> headers better now, so that might help.  But you still need adequate
> through-put.  I've found that /var/lib/imap (meta-data) needs fast I/O
> but /var/spool/imap (message store) doesn't due too bad tossed onto a
> SAN or slower disks.

That depends how many mails per second you will have to deliver.
Normally you are right, the meta partition is heavy random i/o.
With high mail receiving rates you need a fast write storage for your imap 
spool as well. Filesystems like ZFS which write their contents nearly 
sequentially are ideal for this kind of work.

And: Use a 64bit system and add much RAM. Your OS will (hopefully) cache 
all metadata which is heavily used in RAM. We are having 98% hits on our 
meta partition.



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus vs Dovecot

[Pascal Gienger]

Mathieu Kretchner <[EMAIL PROTECTED]> wrote:

> I thanks you for your participation but not for the way you do it because
> if you had read my second mail on this topic you wouldn't have to ask
> your crystal sphere to wonder what is my configuration !!

I did miss your 2nd post, I am sorry.
J'ai manqué votre 2ème message, prière de m'excuser.


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: [Dovecot] Cyrus vs Dovecot

[Pascal Gienger]

Mathieu Kretchner <[EMAIL PROTECTED]> wrote:

> kbajwa a écrit :

>> Cyrus  =  0
>> Dovecot=  100

> I guess you've right but I can't post this answer at Cyrus mailing list.
> I'm just trying to have my own opinion of imap server and I already have
> sarcastic answer on the cyrus mailing list !

Stop. What's this?

a) crossposing content to the dovecot mailing list
b) talking about "sarcastic" answers when users try to help you saying that 
migrating from an old cyrus release to a new one is easier then migrating 
to a new system?
c) many users here have described their running configuration to help you.
d) starting an advocacy war?

What are you trying to do?


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Conversion Debian Cyrus 2.1 to 2.2, experiences

[Pascal Gienger]

Paul van der Vlis <[EMAIL PROTECTED]> wrote:

> then convert the databases (on one line):
> find /var/lib/cyrus/ -name \*.db -print -exec /usr/bin/db4.2_upgrade {} \;

> db_upgrade: /var/lib/cyrus/mailboxes.db: unrecognized file type

> So "mailboxes.db" did not work, but the other databases did.

Just a side note: I am pretty sure your mailboxes.db is a skiplist database 
which is AFAIK the default for mailboxes.db in Cyrus IMAP 2.1 and 2.2. No 
conversion is necessary.
Do you have any database type declarations in your imapd.conf?

Pascal


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus vs Dovecot

[Pascal Gienger]

Mathieu Kretchner <[EMAIL PROTECTED]> wrote:

> Ian G Batten a écrit :
>> We have mailboxes.db and the metapartitions on ZFS, along with the zone
>> iteself.  The pool is drawn from space on four 1rpm SAS drives
>> internal to the machine:

To give (hopefully) comparable comparison:

We have our meta files and spool files also on ZFS, with mirrored pools:

# zpool status
   pool: cyrus
 state: ONLINE
 scrub: resilver completed with 0 errors on Sun May 25 12:17:46 2008
config:

NAME   STATE READ WRITE 
CKSUM
cyrus  ONLINE   0 0 
0
  mirror   ONLINE   0 0 
0
c6t600D0230006B66680C50AB4F92F61000d0  ONLINE   0 0 
0
c6t600D0230006C1C4C0C50BE4DFE511B00d0  ONLINE   0 0 
0

errors: No known data errors

  pool: mail
 state: ONLINE
 scrub: resilver completed with 0 errors on Sun May 25 01:05:02 2008
config:

NAME   STATE READ WRITE 
CKSUM
mail   ONLINE   0 0 
0
  mirror   ONLINE   0 0 
0
c6t600D0230006B66680C50AB0F36ADF100d0  ONLINE   0 0 
0
c6t600D0230006C1C4C0C50BE57396E9F00d0  ONLINE   0 0 
0
  mirror   ONLINE   0 0 
0
c6t600D0230006B66680C50AB5675F91300d0  ONLINE   0 0 
0
c6t600D0230006C1C4C0C50BE16FF1FE200d0  ONLINE   0 0 
0

errors: No known data errors


"cyrus" is our log pool, "mail" our imap spool pool.


IO ist mostly write:

# zpool iostat mail 2
   capacity operationsbandwidth
pool used  avail   read  write   read  write
--  -  -  -  -  -  -
mail2.08T  6.02T226163  1.36M  1.67M
mail2.08T  6.02T358 10  1.35M  94.4K
mail2.08T  6.02T234599  1.08M  10.0M
mail2.08T  6.02T 77  0   425K  3.98K
mail2.08T  6.02T 85306   484K  3.39M
mail2.08T  6.02T 95  8   405K  75.6K
mail2.08T  6.02T107  6   798K  47.8K
mail2.08T  6.02T 73232   281K  2.30M
mail2.08T  6.02T 77  2   304K  9.95K
mail2.08T  6.02T 66469   254K  5.84M
mail2.08T  6.02T 83  4   409K  17.9K


As with Ian's setup, most read requests are serviced from ARC.
We have BOTH data (meta and spool) on this ZFS pool, however we defined an 
extra ZFS filesystem for metadata to make distinct snapshots.
cyrus.header remains on the imap spool partition.


Raw Disk I/O is different as ZFS pulls out up to "recordsize" from disk per 
request (128k by default).

Load is 0.47 at the moment, 1355 imapd processes, 10 lmtpd processes 
(limited by delivering gateway), 34 pop3d processes.
The machine is a two-processor Opteron (dualcore) machine, so 4 cores are 
available. It has 20 GB ram  and ARC (zfs) uses:

# kstat zfs:0:arcstats:size
module: zfs instance: 0
name:   arcstatsclass:misc
size9308832256

9 GB zfs file cache.

Hope this helps you a little bit.

Pascal


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: IMAPS terminating abnormally

[Pascal Gienger]

David Korpiewski <[EMAIL PROTECTED]> wrote:

> I have two cyrus machines running and on both systems I'm getting a TLS
> error and then the error "in BUSY state: terminated abnormally".

Which cyrus imapd version?
Can you set the loglevel to debug in your syslog.conf?

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: IMAPS terminating abnormally

[Pascal Gienger]

David Korpiewski <[EMAIL PROTECTED]> wrote:

> Thank you for asking questions, I'm very interested in getting this
> problem solved ASAP.
>
> I have turned on debugging level up by editing my /etc/syslog.conf file
> and adding this line: "local6.debug
> /var/log/mailaccess.log"
> However, I don't see any additional debug information (as shown below).
> I originally had "local6.*" which should have gotten the debug
> information anyways.
>
> The version of the OS is OSX 10.5.5
> The version of cyrus is: (not sure if this is it, but)
> mail2:bin root# ./deliver
> 421-4.3.0 usage: deliver [-C  ] [-m mailbox] [-a auth] [-r
> return_path] [-l] [-D]
> 421 4.3.0 v2.3.8-OS X Server 10.5:  9C31


Ok I have to pass.
This is the Apple Version of their Mail Server, they included many 
extensions to the original cyrus code. They added netinfo support in SASL2, 
and Rendezvous/Zeroconf in IMAP. The only thing I know is that SSL handling 
has been improved since Version 2.3.8 (which is supposedly the version 
Apple used as its base). We are at 2.3.12, and 2.3.13 as Release Candidate.
Did you open a service request Apple with this issue? If it is OS X 10.5.5 
server you'll have support.

If you can live without rendezvous, you can compile an actual release of 
Cyrus IMAP, using Apple's SASL2 library. So you won't give up netinfo 
capabilities. You will lose Apple support though.

Cyrus IMAP 2.3.12 compiles fine under OS X 10.5 when the Apple SDK is 
installed (gcc et.al.).


Pascal

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Mapping a username to a Mailbox via LDAP?

[Pascal Gienger]

Daniel Dewald <[EMAIL PROTECTED]> wrote:

> I want the user to login with his AD credentials and
> still be routed to his correct mailbox. Is there a mapping feature in
> Cyrus for mailbox names I’m not aware of? It would be perfect if

There are two SASL plugins dealing with login and user names:

1. canonical
   translates the given username to an internal username, which cyrus imapd 
uses as mailbox name (with user prefix "user.").
2. auxprop
   takes the given username, retrieves the stored secret, and returns it to 
the SASL library. The auxprop can also do the mapping of the given username 
to another user name schema used in the authentication/secret database. It 
passes the given username unchanged to the imap daemon which will be the 
mailbox name then.

In our setup, users do log in with their e-mail-address, but the mailboxes 
have our internal uid as name. A canonical plugin does the translation. In 
your case, a canonical plugin should convert the username into the sid, 
cyrus imap will use that as mail box name.

Pascal Gienger
Universität Konstanz

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: choosing a file system

[Pascal Gienger]

LALOT Dominique  wrote:

> zfs (but we should switch to solaris or freebsd and throw away our costly
> SAN)

Why that? SAN volumes are running very fine with Solaris 10 hosts (SPARC 
and x86). You have extended multipathing (symmetric and asymmetric) onboard.
Solaris accepts nearly all Q-Logic FC cards  (according to my experience).

Pascal


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: choosing a file system

[Pascal Gienger]

Robert Banz  wrote:

> At my last job, we had explored a Dell/EMC SAN at one point. Those
> folks don't seem to understand the idea that Fibre Channel is a well
> established standard -- they only expect you to connect their
> supported stack of hardware and software, otherwise they don't wanna
> talk.

Regarding to support as described by the support contract you are right - 
but I had many EMC big iron SAN devices running without a problem with 
Solaris 10. You have to adapt scsi_vhci.conf if you want symmetric 
multipathing as Sun does not recognize many of the FC devices which can 
handle symmetric links out there.

ZFS with SAN devices is perfectly OK. We have 33 million files on our 
(single!) ZFS mail pool, running gzip compression (Solaris 10 Patch 
137137-09 resp. 137138-09). Our Tivoli Storage Manager backup (tsm) runs 
every night for three hours approximately. Within this 3 hours it does scan 
all files. We do a zfs snapshot every day and we are holding 14 days 
snapshots to restore mailboxes. We are not conservatice enough to run scrub 
regularly, the last time I did was last week, without any error.

A happy and successful 2009 for all of you!

Pascal
-- 
Pascal Gienger
pas...@southbrain.com
http://southbrain.com/


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: choosing a file system

[Pascal Gienger]

Henrique de Moraes Holschuh  wrote:

> Ext4, I never tried.  Nor reiser3.  I may have to, we will build a brand
> new Cyrus spool (small, just 5K users) next month, and the XFS unlink
> [lack of] performance worries me.

Nobody likes deletes. Even databases used to mark deleted space only as 
"deleted" until a vacuum (Postgres) or other periodical maintenance command 
was run. Cyrus offers a similiar construct named "delayed expunge". Before 
we migrated our mail system to Solaris 10 it ran on Linux 2.4 with XFS on a 
FC SAN device. Deletes were extremely slow so we had to delay the expunges 
until the weekend, even on night they were too slow and too IO congesting.

On the other hand, XFS was the only Linux filesystems capable to handle our 
5 million files (at that time, we're now at 33 million) we had in these 
days with an acceptable performance. Ext3 was way too slow with directories 
with > 1000 files (but many things have changed from kernel 2.4.x to 
nowadays kernels), IBM jfs was not stable (it crashed during a high load 
test, which was an immediate k.o.). We were reluctant to use Reiser then as 
it was "too new" in 2001.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Doing Cyrus *right* for a large installation

[Pascal Gienger]

Andrew Morgan wrote:

> But then I started thinking about how I was going to backup all this new 
> data...  Our backup administrator isn't too excited about trying to backup 
> 12TB of email data.

Just a side note:

For most backup systems it is not the size which matters much (because
only a small portion of data gets changed each day on a typical IMAP
storage) it is the *NUMBER* of files, because - for an incremental
backup - each file stat data has to be read and compared.

With ZFS (gzip compression activated), we get 31 million files in 3-4
hours (example with tivoli storage manager):

01/03/09   23:38:48 --- SCHEDULEREC STATUS BEGIN
01/03/09   23:38:48 Total number of objects inspected: 31,305,555
01/03/09   23:38:48 Total number of objects backed up:   25,347
01/03/09   23:38:48 Total number of objects updated:  0
01/03/09   23:38:48 Total number of objects rebound:  0
01/03/09   23:38:48 Total number of objects deleted:  0
01/03/09   23:38:48 Total number of objects expired:308,991
01/03/09   23:38:48 Total number of objects failed:   0
01/03/09   23:38:48 Total number of bytes transferred:2.56 GB
01/03/09   23:38:48 Data transfer time:  280.87 sec
01/03/09   23:38:48 Network data transfer rate:9,594.01 KB/sec
01/03/09   23:38:48 Aggregate data transfer rate:209.05 KB/sec
01/03/09   23:38:48 Objects compressed by:   15%
01/03/09   23:38:48 Elapsed processing time:   03:34:49
01/03/09   23:38:48 --- SCHEDULEREC STATUS END
01/03/09   23:38:48 --- SCHEDULEREC OBJECT END SA_SO_MAIL 01/03/09
20:00:00
01/03/09   23:38:48
Executing Operating System command or script:
   /mail/bin/pr_backupsnapshot_off

Just to give you a "real life" number...

Pascal

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Solaris10/ZFS

[Pascal Gienger]

James M McNutt schrieb:
> We are currently running Solaris 9 with VxVM/VxFS
> and looking to move to Solaris 10 with ZFS.
> 
> I was looking for some feedback from those using ZFS.
> 
>  what type of system?

2x X4200 with 32 GB RAM each

>  what type of storage?

2x FiberChannel Storage Systems in two different locations, mirrored via 
zfs.

>  how large?

2x8 TB mirror

>  compression?

gzip

>  replication?

zfs mirror

>  problems?

at the beginning (2006) many, nowadays X patches and kernels after, none 
left.


Pascal

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: unexpunge segfaults with -l on some mailboxes

[Pascal Gienger]

Patrick Boutilier schrieb:
> Plus you lose all the messages that are in "delayed expunge" state after 
> running a reconstruct. :-(

Just delete cyrus.expunge in the appropriate mailboxes meta directory 
before running reconstruct.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Automatically moving marked mails?

[Pascal Gienger]

Ian Eiloart wrote:

> I was speaking to a friend who provides Exchange servers for small 
> businesses locally. He says that the most important thing is to have a 
> really good (fast, available and accurate) disaster recovery procedure, 
> because you need it a lot.
> 


Here in Germany we have a bigger pressure. Microsoft offers university 
to "get Exchange for free for the whole campus at Microsoft's cloud", so 
they want to offer a complete outsourcing.
Sure, they don't have any procedure how to get all data out of Exchange 
after this "for free" period but they get very aggressive, writing 
directory to the board of directors of the university.
Whilst it is complete nonsense that an internet cut results in 
non-mail-connectivity between one office to the other (how dumb is that, 
to write to your room neighbour, you have to go to via a remote exchange 
cloud...).

Things are getting hard.
We believe in open standards, we want to have our mails and appointments 
in a system which is at every time perfectly changeable. We don't want a 
"data dead end" resulting in a complete dependency on one manufacturer.

Zimbra is another show stopper here. Many want "Zimbra" because it is 
soo cool and blah blah blah. But with 14,000 accounts, our central LDAP 
infrastructure and the Solaris 10 servers with ZFS, running Cyrus IMAP, 
there is no really good reason to migrate all to Zimbra just to have 
CalDAV calendaring.  Zimbra means endless redo logs, bad performance 
with many accounts, ... ...

I don't like these "all in one solutions", but the people here LIKE 
THEIR OUTLOOK! Everybody wants to use Outlook and our students want 
Google, they like Gogle! Safe harbour for personal data? not 
interesting to this youth which even posts pictures of their drunk 
parties on facebook :-\

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Security impact of lmtpd with pre-auth

[Pascal Gienger]

Nikolaus Rath schrieb:
> Hello,
> 
> Apparently (http://wiki.exim.org/CyrusImap) I need to let lmtpd accept
> connections from localhost as pre-authenticated to make cyrus and exim
> work nicely together.
> 
> Can someone explain what this actually means security wise? I.e. what
> could a malicious user on localhost do with a pre-authed connection?

He can put/deliver mail in whatever mailbox.

The other side: If you have a "malicious unix user" on your Cyrus Box, 
you'll have a bunch of another problems, far aside from delivering mails 
to every mailbox...

Delivering mails from localhost to localhost via lmtp with 
authentication has the problem that the sending side does need to now 
the credential. If the sending side knows that credential, a "malicious 
user" does have access to it because the sending side is on the same 
box, the same container, ...

Just my $0.02,

Pascal
-- 
Pascal Gienger
University of Konstanz, IT Services Department ("Rechenzentrum")
Electronic Communications and Web Services
Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Security impact of lmtpd with pre-auth

[Pascal Gienger]

Nikolaus Rath schrieb:

> But unless I have some exotic filtering and/or rate limiting configured,
> he can do exactly the same thing by connecting to localhost:smtp, or
> invoking sendmail directy, can't he? So why the additional protection
> for lmtp?

Imagine a Cyrus Box only accepting LMTP connections, no sendmail, no 
Postfix, no other SMTP MTA running on it.
Then imagine a frontend smtp relay delivering directly via LMTP over TCP 
to your Cyrus box. You can use lmtp auth then to prevent other machines 
from directly delivering mails via lmtp.

Pascal
-- 
Pascal Gienger
University of Konstanz, IT Services Department ("Rechenzentrum")
Electronic Communications and Web Services
Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: DBERROR with Cyrus 2.2.12

[Pascal Gienger]

Christophe Boyanique schrieb:
> In fact I managaged to solve the problem by removing the quota file in
> /var/lib/imap/quota/x// and using "quota -f" on the mailboxes.

So the quota file was corrupt. Ok. Do you really use DB4 quota files?

> But I still get repeating error messages like this:
> 
> Jul  7 14:58:51 mail7702 lmtpunix[8988]: DBERROR db4: 9 lockers
> Jul  7 15:04:24 mail7702 lmtpunix[9249]: DBERROR db4: 11 lockers
> 
> Should I be worried by these messages ?

No. In fact they are not errors but informational messages, so in newer 
versions of cyrus imapd you will see

Jul  8 15:39:28 atlanta lmtpunix[29159]: [ID 366844 local6.info] DBMSG: 
2279 lockers
Jul  8 15:51:34 atlanta lmtpunix[29077]: [ID 366844 local6.info] DBMSG: 
2249 lockers
Jul  8 18:12:14 atlanta lmtpunix[4289]: [ID 366844 local6.info] DBMSG: 
1583 lockers

lmtpunix wants to open delivery.db which normally is a DB4 backed 
database unless you have changed the default in imapd.conf.

Some old DB4 version had the problem of not releasing these locks so 
there was an overflow after some time. If the number of lockers keep 
growing without every becoming smaller from time to time you are running 
in this bug.


-- 
Pascal Gienger
University of Konstanz, IT Services Department ("Rechenzentrum")
Electronic Communications and Web Services
Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: tls_sessions.db will not created

[Pascal Gienger]

Martin Schweizer schrieb:
> Hello
> 
> I have the following system
> FreeBSD acsvfbsd06.acutronic.ch 7.2-RELEASE FreeBSD 7.2-RELEASE #1:
> Thu Jun 11 16:16:57 CEST 2009
> mar...@acsvfbsd06.acutronic.ch:/usr/obj/usr/src/sys/GENERIC  amd64
> 
> and I fresh installed  Cyrus IMAPD v2.3.14.
> In my /var/imap directory the file annotations.db will create
> automaticly at each restart from Cyrus (if it's not there) but not
> tls_sessions.db. My compile options are:
> 
>   $ ./configure --sysconfdir=/usr/local/etc
> --with-cyrus-prefix=/usr/local/cyrus --with-cyrus-user=cyrus
> --with-cyrus-group=cyrus --with-tls-db=skiplist --wit
> h-sasl=/usr/local --with-bdb=db41 --with-com_err --with-openssl=/usr
> --with-perl=/usr/local/bin/perl5.8.9
> --with-bdb-incdir=/usr/local/include/db41 --with-bdb
> -libdir=/usr/local/lib --with-snmp=no --prefix=/usr/local
> --mandir=/usr/local/man --infodir=/usr/local/info/
> --build=amd64-portbld-freebsd7.2

a) Is SSL enabled? Did you try a connect via imaps or imap/starttls?
b) what's in the log  after this connect?
c) is the tls_session.db there after your tls connect?

The --with-tls-db-Switch should just define the default database backend 
for that database. It can be overriden at any time via imapd.conf.

Pascal

-- 
Pascal Gienger
University of Konstanz, IT Services Department ("Rechenzentrum")
Electronic Communications and Web Services
Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter exits with "fatal error: Virtual memory exhausted" on huge mailbox

[Pascal Gienger]

Sebastian Hagedorn schrieb:
> Processing index character 101, 681642 total words, temp file size is 
> 2107147
> fatal error: Virtual memory exhausted

4 GB limit of 32 bit binaries?
How much RAM does squatter allocate before it dies?

-- 
Pascal Gienger
University of Konstanz, IT Services Department ("Rechenzentrum")
Electronic Communications and Web Services
Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: squatter exits with "fatal error: Virtual memory exhausted" on huge mailbox

[Pascal Gienger]

Sebastian Hagedorn schrieb:
>>> fatal error: Virtual memory exhausted

> Of course it's possible that it then tried to allocate one huge chunk, 
> but I can't see that. Are there better tools to monitor the memory 
> allocation of a process?

Swap file/partition full?
Background:
I think the message "Virtual memory exhausted" is coming from your 
operating system and not from the squatter process.


Squatter would have been said

   switch (err) {
   case SQUAT_ERR_OUT_OF_MEMORY:
 fprintf(stderr, "SQUAT: Out of memory (%s)\n", s);
 break;



So I think it is a Virtual Memory/Swap problem in your OS.

-- 
Pascal Gienger
University of Konstanz, IT Services Department ("Rechenzentrum")
Electronic Communications and Web Services
Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Migrating 32bit to 64bit Debian Lenny

[Pascal Gienger]

Simon Matter wrote:

> I'm wondering how much of all this was really needed for the migration
> from 32bit to 64bit? Are the BerkeleyDB ondisk files different on
> 32/64bit?

Yes they are. It's not the OS that matters but the architecture of the 
libdb4.so file.
It is still a good idea not to use Berkeley DB for real important data. 
Here at our university's cyrus we are using Berkeley for the duplicate 
delivery and the tls databases - both of them are easily set to zero in 
case of problems without deep impact on the functionality (in case the 
delivery db crashes users can get some mails two times (doubling), in 
the latter case (tls db crash) a returning client has to re-initiate a 
TLS handshake including key exchange).

Pascal Gienger
-- 
Pascal Gienger
University of Konstanz, IT Services Department ("Rechenzentrum")
Electronic Communications and Web Services
Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Logging region out of memory

[Pascal Gienger]

Marc Patermann wrote:
> Hi,
> 
> I have IMAPd 2.2.12 and BDB 4.2.52:
> 
> When I got:
> 
> Sep  2 11:28:39 rzhs199 local6:warn|warning lmtpunix[1171642]: DBERROR
> db4: Logging region out of memory; you may need to increase its size

Increase logging region size.

> I found DB_CONFIG in /mail/imap/
> 
> :/mail/imap # cat DB_CONFIG
> set_cachesize 0 8388608 8
> set_lg_regionmax 524288
> set_lg_bsize 2097152
> 
> and these files

The file DB_CONFIG has to be in the "db" subdirectory (/mail/imap/db in 
your case).
Be warned: Some parameters of DB_CONFIG also change the on-disk-format, 
so backup your db files before (after shutting down cyrus) and restart 
after your changes.

Don't delete "skipstamp" in this db directory as it is used by your 
skiplist databases.

Just a personal biased hint:
You should not use Berkeley DB for important data of your cyrus system. 
Berkeley has a very rapid random read performance which is important (in 
our case) with the duplicate delivery database (now 1,3 GB in size). But 
even that should be feasible with skiplist.

Pascal
-- 
Pascal Gienger
University of Konstanz, IT Services Department ("Rechenzentrum")
Electronic Communications and Web Services
Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: No e-mail notification with sieve, Thunderbird and Cyrus-imap

[Pascal Gienger]

Ludovic Gasc schrieb:
> Hi everybody,
> 
> We're using Cyrus-imap during some time, it's a good tool for us.
> 
> We've a strange behaviour (bug)? with sieve, Thunderbird and Cyrus-imap.
> I want to listen your opinions, because I'm not sure to understand
> correctly the problem.
> 
> We use some sieve scripts to filter the e-mails in the sub-folders of INBOX.

I never had this problem. Be sure to mark every subfolder you need with 
"Check for new messages" (right click on the folder you want to be 
checked, then click on "Properties").

Thunderbird opens a new IMAP connection for each folder. For each folder 
marked with "Check for new message" ("Auf neue Nachrichten überprüfen" 
in my case, I have a german localized Thunderbird) it will issue an 
"IDLE" command (easily traceable).

Pascal


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: VMware for Cyrus?

[Pascal Gienger]

John Madden schrieb:
> FWIW, I won't run anything on hardware anymore unless I absolutely have 
> to.  To me, the benefits of running virtualized outweigh the pitfalls -- 
> dealing with real OS installs on real hardware, dealing with 
> multipathing and SAN (virtual disks are easy), etc.

Our Cyrus runs on Solaris with proper ZFS storage. This kind of storage 
is fast, reliable and supports many nodes per directory without a 
problem. Files check for backup is done in two hours for 50 million 
files (Tivoli Storage Manager Backup).

We just can't virtualize this because in whatever "solution" the 
underlying block devices get virtualized again. The only solution we 
would have is to bind these storage devices (fc) exclusively on the 
virtualized guest systems. Problem remains: Solaris 10 is not well 
supported in VMware (no client tools and without them access remains 
_SLOW_) nor in Xen/Sun xVM. In the latter OpenSolaris (and Solaris 11) 
is the way to go (architecture i86xpv).

Just to give a reason why sometimes it _IS_ necessary or better to have 
a real iron.

Pascal
-- 
Pascal Gienger
University of Konstanz, IT Services Department ("Rechenzentrum")
Electronic Communications and Web Services
Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: ANNOTATEMORE => METADATA and rfc 5464

[Pascal Gienger]

Bron Gondwana schrieb:
> Does anybody out there use annotations much?  Does anybody know any code
> that would be broken by changing the way annotations are done?

We are using annotations to define expire times for spam folders and to 
define mailboxes to be indexed by squatter. Approx. 4 annotations 
are always set.
-- 
Pascal Gienger
University of Konstanz, IT Services Department ("Rechenzentrum")
Electronic Communications and Web Services
Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Condstore & Squat through IMAP

[Pascal Gienger]

Nybbles2Byte schrieb:
> Hello Info-cyrus,
> 
>   When you use cyradm you can do these:
> 
>mboxcfg mailbox_name condstore true
>mboxcfg mailbox_name squat true
> 
> to set Constore and Squat respectively on a mailbox.
> 
> However, I am writing a configuration program of my own to work through 
> a web interface and I can't see how to do this through the normal IMAP 
> command and response strings.


For squat it is


. SETANNOTATION "user..mybox" "/vendor/cmu/cyrus-imapd/squat" 
"value.shared" "true"

Side note: Expiration time is set via


. SETANNOTATION "user..Spam" "/vendor/cmu/cyrus-imapd/expire" 
"value.shared" "7"

(for 7 days expire timeout on Folder Spam from user ).


Correct me if I am wrong.


As far as condstore: I never used that so I don't know the syntax. It 
should be pretty the same but I don't know the appropriate keyword after 
"cyrus/imapd/".

Pascal
-- 
Pascal Gienger
University of Konstanz, IT Services Department ("Rechenzentrum")
Electronic Communications and Web Services
Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Any interest to implement RFC4978 (IMAP COMPRESS)?

[Pascal Gienger]

RFC 4978 [1] defines an IMAP COMPRESS command to compress IMAP data 
communication.

Is there any interest to implement this extension in the cyrus imap server?

For low bandwith connections this could be useful but I don't know if 
that's a typical case nowadays. Together with the IMAP IDLE command it 
should be fine for mobile devices...

[1] http://tools.ietf.org/html/rfc4978

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Anyone using SunFire X4140's (AMD) or X4170's (Intel Xeon) as Cyrus servers ?

[Pascal Gienger]

Le 15/03/10 17:08, Eric Luyten a écrit :
>

X4200 here (also AMD, 2x dualcore).
Operating system?
Solaris x86 or Linux?

Pascal


-- 
Pascal Gienger
University of Konstanz, IT Services Department ("Rechenzentrum")
Electronic Communications and Web Services
Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Is Reiserfs better than ext3

[Pascal Gienger]

mb  wrote:

PS I know some people say XFS, JFS etc are brilliant but my local
experience is that although they may be faster in Linux 2.4, they are more
prone to corruption.. (haven't tried either in the last 6 months tho')
As for XFS, I never had any problems with it (Version 1.2.0). We are 
running a 600 GB Mailspool with it. I would call it reliable.

Pascal

Re: Mailbox list in txt file

[Pascal Gienger]

Zitat von Bartosz Jozwiak <[EMAIL PROTECTED]>:

> This is what i get:
> 
> bartek:/# su - cyrus
> bartek:/# /usr/cyrus/bin/ctl_mboxlist -d
> fatal error: must run as the Cyrus user

Your "cyrus" user account does not have a valid shell or home directory.
Check this.

Pascal Gienger
---
Home Page: http://asg.web.cmu.edu/cyrus
Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: complett relocation from old to new server and cyrus version

[Pascal Gienger]

"Hans M. Schleidt" <[EMAIL PROTECTED]> wrote:

Hi.

I hat to rebuild a new server with fully new cyrus. Now i must bring the
old  messages to the new cyrus. Only copy get wrong.
What are the goals to do that? Wich files (mailboxes.db, cyrus.cache,
cyrus.header, and so on) shout i bring it to the new system?
1. Dump your mailboxes-Database on the old system:
  ctl_mboxlist -d > mailboxes.dump
  Copy the resulting "mailboxes.dump" to the new server.
2. Copy the imap mail store 1:1 to the new server:
  example (here on both machines, the store is in /var/spool/imap,
  the target directory must exist and be writable by the cyrus user):
  cd /var/spool/imap
  tar cvpf - . | ssh [EMAIL PROTECTED] "cd /var/spool/imap; tar xpf -"
3. Copy the following directories to the new system
  (do it like you have done with your mail store):
  - Sieve(e.g. /var/imap/sieve)
  Databases:
  - Quotas   (e.g. /var/imap/quota)
  - Seen/Subscribed  (e.g. /var/imap/user)
  Warning! This will work only if your new cyrus system uses the
  same database backends as the old one!
  Losing the databases results in the following:

  - Quotas: The user will not have any quota restriction anymore.
  - Seen/Sub: The user will not see which messages are marked
"read" nor it will now which mailboxes he was subscribed to.
4. Reconstruct a new mailboxes-Database on the new system:
  ctl_mboxlist -u < mailboxes.dump
5. Run reconstruct on the new system.
  reconstruct -f
6. When used quotas on the old system, run on the new one:
  quota -f
7. Tricky part: SASL user database
  When sasldb was used, then:
  If your new sasl lib uses the same database backend as the old one, you 
may simply copy
  your old sasldb to the new server when these conditions are met:

  - Your realm is the same as on the old server.
  - The database backend is the same
  You may still set the same realm/imap server name as the old one in your 
imapd.conf.

  If your IMAP realm is your hostname or your sasl database backend is 
another one, you
  must use a program to dump the contents of the old sasldb. As passwords 
are stored in clear-
  text it is not very difficult to accomplish that.

Pascal
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: reading quotas

[Pascal Gienger]

Colin Bruce <[EMAIL PROTECTED]> wrote:

Dear All,

This may be a stupid question but is there any reason why someone should
not be able to obtain their own e-mail quota. If I do something like
telnet imaphost 143
. login ccx004 password
. getquota user.ccx004
I get "NO Permission denied"
GETQUOTA is reserverd to a user with admin privilege ("admins:" in 
imapd.conf). Use GETQUOTAROOT instead to get your quota usage:

telnet server 143
. login myusername mypasswort
. getquotaroot INBOX
* QUOTAROOT INBOX INBOX
* QUOTA INBOX (STORAGE 519605 3145728)
. OK Completed
Pascal
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Client authentication via client certificate on ssl/tls

[Pascal Gienger]

Hi,
does anybody on the list already had the idea to use an information of a 
client certificate for authentication in IMAPD?

There could be 3 solutions for it:
1. the TLS part can pass information of the presented client certificate to 
imapd, so a normal anonymous login would be sufficient - the imapd process 
would use an attribute of the client certificate as user-id.

2. Using an external X509 SASL mechanism - but this requires special 
software on client side and you would present your client certificate 2 
times: First in SSL handshake and second via "AUTHENTICATE".

3. you could use Kerberos 5 and a special signon program to get your ticket 
and use GSSAPI as SASL mechanism.

1. has to be done in imapd and pop3d code.
2. has to be done via an x509 auxprop or external x509 authenticator.
3. means to build a kerberos5 infrastructure around failover kdc's. This 
may work well for Windows boxes but how about other operating systems?

Would be nice to use client-cert-ssl on the whole campus to login for 
services.

Pascal
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: help on failed upgrade

[Pascal Gienger]

[EMAIL PROTECTED] wrote:


Hi there!

Last week we upgraded our server from cyrus 2.1.15
(SuSE 9.0) to 2.2.21 (SuSE 10.1). we had some
/var/lib/imap problems that prevented us from even
starting cyrus. We managed to boot it up after
deleting some databases in db/ and backup/, after
following some thread in a forum. The result, all
email messages stored lost their state (read, unread,


The standard for the seen DB was changed from flat to skiplist as my memory 
recalls on that suse package. You should see database errors in your syslog 
coming from lmtpd and imapd.
You need to convert all your seen-databases to skiplist if you want to use 
the default configuration.
New states will not be added because the seen_db-engine can not even read 
your old flat files.


Workaround (ONLY if you see such errors in your log stating that seen_db is 
not readable): put


seenstate_db: flat

in your imapd.conf and do a "rcimap restart", and things should just 
continue as they did.


Pascal

Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: lmtp over tcp sockets, access denied and lmtp error: Message contains invalid header

[Pascal Gienger]

Rudy Gevaert <[EMAIL PROTECTED]> wrote:



Now postfix complains:
Jul 21 15:25:42 oliebol postfix/qmgr[7484]: 2A9BA7458:
from=<[EMAIL PROTECTED]>, size=348, nrcpt=1 (queue active)
Jul 21 15:25:42 oliebol postfix/lmtp[7490]: 2A9BA7458:
to=<[EMAIL PROTECTED]>, relay=none, delay=4236, status=deferred (connect
to mail2.ugent.be[157.193.71.18]: Connection refused)


Postfix seems to connect to a wrong port number. What's in your master.cf 
and main.cf regarding that lmtp transport?



I want to test my lmtp setup, but this fails too:


No it did not :)


oliebol:/etc/postfix# telnet mail2.ugent.be 2003
Trying 157.193.71.18...
Connected to mail2.ugent.be.
Escape character is '^]'.
220 mail2.ugent.be LMTP Cyrus v2.3.7 ready
LHLO foo.edu


[...]


mail from:<[EMAIL PROTECTED]>
250 2.1.0 ok
rcpt to:<[EMAIL PROTECTED]>
250 2.1.5 ok
DATA
354 go ahead
daf

[...]

.
554 5.6.0 Message contains invalid header



That is normal. "daf" is not an allowed mail header.

You seem to have configured 2003 as your lmtpd port in your /etc/services 
of your cyrus host. Does lmtp use the same port?


Pascal
Pascal

Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: missing plain authentication?

[Pascal Gienger]

Ross Boylan <[EMAIL PROTECTED]> wrote:


No; that was a transcription error.  Sorry about that.
So the original file has
allowplaintext: yes


This is the traditional imap plaintext login without sasl. IMAP4 has 
plaintext authentication as a builtin. The syntax is


A001 LOGIN username password

You may replace the prefix A001 with a dot but this will only work for 
cyrus not for many other imap servers out there.


Pascal


Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Replication woes with a specific mailbox...

[Pascal Gienger]

We had a strange exception here, a user's inbox could be replicated without 
problems, but doing it for a second time does not do it.


sync_client was invoked like this:

sync_client -v -u X

Jul 28 10:52:17 priscilla sync_client[21326]: RENAME received NO response: 
Rename failed user.X -> user.X.Uni: Operation is not 
supported on mailbox
Jul 28 10:52:17 priscilla sync_client[21326]: do_folders(): failed to 
rename: user.X -> user.pX.Uni
Jul 28 10:52:17 priscilla sync_client[21326]: Error in do_user(X): 
bailing out!


sync_client seems to try to RENAME the Inbox to the subfolder "Uni" which 
is complete nonsense.


It is the only mailbox where this appears.

Replicating in mailbox mode does work however.

I did a mailboxes.db-Dump (ctl_mboxlist -d) and could not find any 
inconsistencies with that user's mailboxes, including the one named "Uni".


Anyone had the same problem? :) Difficult to submit a bug report because I 
can't tell the exact condition when this appears.

The username was replaced by "X" for privacy reasons.


Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Replication woes with a specific mailbox...

[Pascal Gienger]

David Carter <[EMAIL PROTECTED]> wrote:


Do the mailboxes have the same UniqueID (see cyrus.header files)? The
replication engine expects UniqueID to be unique. Cyrus makes a bit of a
hash of renaming user inboxes (user.XXX -> user.XXX.Uni). Removing the
cyrus.header file and running reconstruct should fix the problem.



That fixed the problem. Thank you! I wonder why these IDs were unique...

Pascal

Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: High availability email server...

[Pascal Gienger]

David Korpiewski <[EMAIL PROTECTED]> wrote:


I spent about 6 months fighting with Apple XSAN and Apple OSX mail to try
to create a redundant cyrus mail cluster.  First of all, don't try it, it
is a waste of time.  Apple states that mail on an XSAN is not supported.
The reason is that it simply won't run.   The Xsan can't handle the large
amount of small files and will do things like disconnect or corrupting
the file system.


STOP!
The capability to handle small files efficiently is related to the 
filesystem carrying the files and NOT to the physical and logical storage 
media (block device) under it.


A SAN is a network where physical and logical block devices are shared 
between nodes and which makes it possible to mount a harddisk or raid 
partition as a block device even if the disk is some miles away - from this 
point of view there is no difference between iSCSI (IP) and FiberChannel 
(FCP) besides different hardware.


For your host/mail server there is no difference between having the 
filesystem on - say - a 300 GB local hard drive partition or having it on a 
SAN volume routed between SAN switches. The OS "sees" it like a normal 
block device, and the filesystem just uses that.


So if Apple says that Xsan does not handle many files they admit that their 
HFS+ file system is crap for many small files.


Most file systems are not built to carry many small files in a directory 
and all are prone to performance losses at a certain number of files in a 
single directory.


There are techniques to handle these situations - for xfs (as an example) 
consider having *MUCH* RAM in your machine and always mount it with 
logbufs=8.



I would NEVER suggest to mount the cyrus mail spool via NFS, locking is 
important and for these crucial things I like to have a real block device 
with a real filesystem, so SANs are ok to me.


We are having here a RAID device with 1,5 TB wich is shared between 2 mail 
nodes and 2 test nodes. The switch can be done manually (10 seconds 
downtime) and - if you wish - via Heartbeat HA software. The only dangerous 
thing is to ensure that NEVER, really NEVER a second node mounts your SAN 
partition while another has mounted it already. Immediately kernel halts 
and data losses are the result.


Pascal Gienger

Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

RE: High availability email server...

[Pascal Gienger]

"David S. Madole" <[EMAIL PROTECTED]> wrote:


That's just not true as a general statement. SAN is a broad term that
applies to much more than just farming out block devices. Some of the
more sophisticated SANs are filesystem-based, not block-based. This
allows them to implement more advanced functionality like cross-platform
sharing of volumes, simultaneous mounts of volumes from different hosts,
backups (and single-file restores) performed by the SAN system, pooling
of free space, transparent migration to offline storage, etc., etc., etc.


In my "classical" view a SAN is a network used for storage applications to 
give a view on shareable block devices. There are hardware applications 
giving access to the same filesystem in a shareable manner (as GFS or ocfs) 
but this is software logic in the filesystem and firmware level and not in 
the classical SAN components like JBOD arrays, RAID controllers and FC or 
IP switches.


In the Apple case we need to distinguish Apple XSAN Harddisk chassis and 
the XSAN software. The XSAN software seem to give you a special filesystem 
for SAN issues (at least I read this on their webpage). So if Apple says 
that this is not suited well for many small files I would not use it for 
that.



Another instance of a SAN filesystem that I do happen to be familiar with
is IBM's:

http://www-03.ibm.com/servers/storage/software/virtualization/sfs/index.h
tml


Also this filesystem lives above the FCP (Fiberchannel) protocol forming a 
filesystem including multipathing elements and concurrent access 
strategies. Still you have to distinguish the block-level access to SAN 
devices and the filesystems build above them. It is true that "SAN" is 
marketing speech for all kind of things.


Pascal

Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Does the quota include deleted but not yet expunged mails in v2.3 with delayed expunge?

[Pascal Gienger]

David Carter <[EMAIL PROTECTED]> wrote:


On Thu, 9 Nov 2006, Farzad FARID wrote:


I'm running Cyrus Imapd 2.3.7 with the delayed expunge mode. Do the
messages deleted by the user, but not yet expunged by the system, count
in the user's quota? I'd say yes but I'd like a confirmation.


Yes. \Deleted is just another flag on messages.


He does not talk about deleted and not yet expunged mails, he talks about 
the delayed expunge mode.
That means, the user expunges the mailbox, so all messages marked with 
\Deleted are expunged and removed from the user's quota. But the messages 
still remain physically in the file system until they are really expunged 
by the expire process. This is helpful if you want to be able to restore 
accidently deleted files very fast or if your file system has a very bad 
unlink performance, like xfs.


So Mr Farid, messages deleted and expunged by the user do not count in the 
user's quota even if the delayed expunge mode is turned on.


But keep in mind that the delayed messages still gobble up space in your 
filesystem.


Pascal Gienger

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Mailbox is locked by POP server

[Pascal Gienger]

former03 | Baltasar Cevc <[EMAIL PROTECTED]> wrote:


Hi Martin,

Personally I do not use POP3 any more - however I remember that there
was a limitaition of 1 connection per mailbox in other servers.


Yes.
The POP3 RFC states that operations such as DELEte a message will become 
"final" after issuing a "QUIT" command. If connection breaks, no change is 
made to the mailbox.


For having multiple concurrent POP3 access to a mailbox a complete 
transaction based model including rollbacks would be needed to implement 
proper POP3. I don't think anybody wants to improve pop3d to accomplish 
that ;-)


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Weird cyrus user

[Pascal Gienger]

Bob Marcan <[EMAIL PROTECTED]> wrote:


May 17 10:41:06 populus2 lmtpunix[15310]: DBERROR: error fetching
user.avucajnk: cyrusdb error


Does a

reconstruct -f -r user.avucajnk

help?
If so, the cyrus.*-Database-Files were corrupted in the users' 
maildirectory.


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Penalty timeout bug in pop3d

[Pascal Gienger]

Hi,

was there a special reason to "forget" the sleep(3) penalty timeout after 
an "invalid user" auth nak message?

I did that because we had a POP3 client running wild while trying out 
unknown sasl users...

# diff pop3d.c.orig pop3d.c
1220d1219
<   prot_printf(popd_out, "-ERR [AUTH] Invalid user\r\n");
1223a1223,1224
>   sleep(3);
>   prot_printf(popd_out, "-ERR [AUTH] Invalid user\r\n");

Pascal

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Penalty timeout bug in pop3d

[Pascal Gienger]

Pascal Gienger <[EMAIL PROTECTED]> wrote:

> Hi,
>
> was there a special reason to "forget" the sleep(3) penalty timeout after
> an "invalid user" auth nak message?
>
> I did that because we had a POP3 client running wild while trying out
> unknown sasl users...

Imapd version is 2.3.9, diff -c is as follows:


# diff -c pop3d.c.orig pop3d.c
*** pop3d.c.origMon Sep 17 13:57:19 2007
--- pop3d.c Mon Sep 17 13:58:01 2007
***
*** 1217,1226 
  !(config_virtdomains &&  /* allow '.' in dom.ain */
(domain = strchr(userbuf, '@')) && (dot > domain))) ||
 strlen(userbuf) + 6 > MAX_MAILBOX_NAME) {
-   prot_printf(popd_out, "-ERR [AUTH] Invalid user\r\n");
syslog(LOG_NOTICE,
   "badlogin: %s plaintext %s invalid user",
   popd_clienthost, beautify_string(user));
  }
  else {
popd_userid = xstrdup(userbuf);
--- 1217,1227 
  !(config_virtdomains &&  /* allow '.' in dom.ain */
(domain = strchr(userbuf, '@')) && (dot > domain))) ||
 strlen(userbuf) + 6 > MAX_MAILBOX_NAME) {
syslog(LOG_NOTICE,
   "badlogin: %s plaintext %s invalid user",
   popd_clienthost, beautify_string(user));
+   sleep(3);
+   prot_printf(popd_out, "-ERR [AUTH] Invalid user\r\n");
  }
  else {
popd_userid = xstrdup(userbuf);




-- 
Pascal Gienger
Rechenzentrum Univ. Konstanz

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Cyrus IMAP 2.3.9 on Solaris 10 with ZFS and SAN

[Pascal Gienger]

Just a little note, for those who have perhaps the same problem.

We saw performance problems after we switched from a Linux installation to 
a Solaris 10 cluster connected to our SAN (using scsi_vhci and 2 Qlogic 
Controllers).


Problems arose when real load came to the machine, despite having tested it 
with some load simulation scenarios...


Looking at the attached chart[0], you will notice up to 10 seconds time to 
select a mailbox, far too long.


The problem was ZFS - ZFS is perfect for rubbish, cheap disks with ugly 
firmwares. So if your storage is "too good" it makes things go bad.


First: The file prefetch algorithm does not seem to be very good for 20 
million mail files and 300,000 cyrus meta files... ;-)


So first step was (at 11am in the chart) to disable this prefetching 
routine. [1]


This cut off 4 seconds from requests, but still 10 second-timeouts were 
seen.


Next step was to disable the zfs cache flush. As said, ZFS is good for 
rubbish disks, so every 5 seconds it instructs the SATA or SCSI drives to 
flush their internal ram to disk. This is good for cheap disks, but a no-go 
if you have a SAN RAID storage having 2 Gigabytes and more of RAM storage, 
buffered with a battery. In fact, our storage system really flushed all 5 
second its complete RAM cache, you even saw it watching the "blinkenlights".

Plus, every fsync() call did the same... :(

There is a trick: You can disable this in ZFS [2]. You see the result at 
4pm (16:00 european notation in the chart). CHILDREN DO NOT TRY THIS AT 
HOME. Only do this if you don't have any "real" physical disk storages 
attached to your system with zfs pools on them - otherwise you will lose 
data on power outages as the ram cache of your hard disk is not buffered by 
a battery.


Now the machine runs and handles all mail without noticeable delays.

[0] http://priscilla.rz.uni-konstanz.de/mailserver/

[1] in /etc/system:   set zfs:zfs_array_rd_sz=0
   on a live system using mdb -kw: zfs_array_rd_sz/Z0x0

[2] in /etc/system:   set zfs:zfs_nocacheflush=1
   on a live system using mdb -kw: zfs_nocacheflush/W0t1

<>
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

RE: Cyrus IMAP 2.3.9 on Solaris 10 with ZFS and SAN

[Pascal Gienger]

Vincent Fox <[EMAIL PROTECTED]> wrote:

> Pascal,
>
> How many accounts did you have per mail-store?

We have 14k(*) users and a failover mailstore. In our SAN we have 3,5T 
storage reserved, which can be expanded to our needs. We use 2 SAN 
storages, location-separated, so when one location gets fire, the system 
still works. The mirroring between them is done via ZFS, the redundancy 
between the two FiberChannel-Links to each storage is done via scsi_vhci.

Pascal


(*) 100k mailboxes and a webmail application which often LOGINs and 
SELECTs, multiple times per webpage reload.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus IMAP 2.3.9 on Solaris 10 with ZFS and SAN

[Pascal Gienger]

Wesley Craig <[EMAIL PROTECTED]> wrote:

> I might suggest running a connection caching daemon, up-imapproxy
> springs to mind.  LOGIN & SELECT are not exactly lightweight.

Actually, login is lightweight and never was source of performance misses. 
As the webmail applications are "talking" via a private subnet to our cyrus 
servers there is no need to use some highly secure algorithms, and cram-md5 
is fast.

For the SELECTs, I will try to use Bron's patch to install a status 
database. Seems to be a smart idea, on my test system here it runs already.

A proxy would mean another possibility of failure. How stable is that proxy 
daemon? Another two boxes for redundancy?

In our installation it is not possible to log on as cyrus or other admin 
users from outside, I patched the imapd to check the caller's ip address - 
and the firewall does the rest so that no forged packet (with an inside 
sender ip coming from outside...) will trigger it.

Pascal

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus IMAP 2.3.9 on Solaris 10 with ZFS and SAN

[Pascal Gienger]

Bron Gondwana <[EMAIL PROTECTED]> wrote:

>
> But yeah, connection caching is nice.  Even just the fork overhead
> on the backend servers is something we can do without if it's
> avoidable.

My own Perl IMAP mod_perl routines use this:



Works flawlessly - but you still have the problem that you have X 
connections per user for X httpd processes running. I/We wrote a web 
frontend for those who don't want to use Webmail (the majority do use their 
own MUA), but still want to control their anti spam settings, quotas, 
automatic vacation responses, automatic expiration of messages in 
user-selectable folders (via annotations.db). We also use annotaton 
heaviliy to preempt spam folders to be indexed by squatter.

Pascal

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus IMAP 2.3.9 on Solaris 10 with ZFS and SAN

[Pascal Gienger]

You wrote:

> By the way I tried this on a fully patched
> Solaris 10u3 system
> and get this notice during boot:

> sorry, variable 'zfs_nocacheflush' is not 
> defined in the 'zfs' module

We have Solaris 10 08/07. Also fully patched.
Kernel is 

SunOS 5.10 Generic_120012-14 i86pc 

This is 10u4 for x86. The SPARC version U4 has the same ZFS vars.
They changed some zfs variables and are doing the same again in their Nevada 
release.

As I am on vacation at the moment I don't have the u3 info here. I'll return 
Sep 29th.

Pascal

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LIST is slow for 35K mailboxes

[Pascal Gienger]

Ian G Batten <[EMAIL PROTECTED]> wrote:

>
> On 09 Oct 07, at 1522, Blake Hudson wrote:
>>>
>> Could database type differences (or contention) be an issue here? What
>> database format are each of you using?
>
> Yes.  With skiplist (took me several stabs at it to get the
> conversion to work) it takes 0.19s.  Versus ~250s with BDB.  A slight
> difference.  Thanks for pointing me at the idea.

Yes, this is the expected result. Skiplist is good in enumerating as there 
is always a "lane" pointing from one mailbox to the next one without 
skipping.

Berkeley DB is very fast in doing _real_ random accesses and to insert new 
or changed values, but it does a lousy lob in enumerating "entries" 
alphabetically or numerically. So it is perfect to use Berkeley for the tls 
session database, for annotations, for duplicate delivery suppression and - 
as Bron Gondwana does - to cache status data.

Pascal

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: UC Davis Cyrus Incident September 2007

[Pascal Gienger]

Vincent Fox <[EMAIL PROTECTED]> wrote:

> The root problem seems to be an interaction between Solaris' concept of
> global memory consistency and the fact that Cyrus spawns many processes
> that all memory map (mmap) the same file.  Whenever any process updates
> any part of a memory mapped file, Solaris freezes all of the processes
> that have that file mmaped, updates their memory tables, and then
> re-schedules the processes to run.

Now we come closer, I experience the same behaviour when loads are going up.
Which file is mmap'd by _all_ Cyrus processes?

I understand that the local index files in every mailboxes are mmap'd after 
a customer logged in to an imap process or when a delivery via lmtpd is 
being made in this mailbox, but which global file is mmap'd by all 
processes? mailboxes.db using skiplist?

Does the problem also arise with 2000 processes using the same db4 library 
to access the same berkeley database?

Pascal

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: UC Davis Cyrus Incident September 2007

[Pascal Gienger]

Scott Adkins <[EMAIL PROTECTED]> wrote:

> Meanwhile, we hacked around this in a very cool way.  We copied the imapd
> process 60 times (assuming average of 12,000 processes, shooting for 200
> processes per executable, that is 60 individual executables).  These were
> named /usr/cyrus/bin/imapd_001 through /usr/cyrus/bin/imapd_060.  We then
> symlinked the "imapd" binary to imapd_001.  We then wrote a cron job that
> ran once a minute and relinked the imapd symlink to the next numbered
> executable,

And how did you solve the problem of the "deadlock" resulting in deleteing 
the symbolic link, setting the new one? Between these events an exec of 
imapd would break resulting in an error to the customer.

Funny hack though.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: lmtpd delivery problem: User unknown

[Pascal Gienger]

Kai Lanz <[EMAIL PROTECTED]> wrote:

To me it looks like lmtpd can't write to our system inboxes under
/var/spool/mail.  Probably a permissions problem; so we reset the files
in this directory to be owned by group "mail" with group RW permissions.
That didn't help; we still get the User unknown complaint.
lmtpd writes in a Cyrus Mail Store. It is not compatible with standard unix 
mbox files in /var/spool/mail, the maildir format comes closer, but cyrus 
still uses cache databases within these directories.
What is in your imapd.conf? Where are your imap partitions?

If you want to access concurrently with multiple clients, your only chance 
is to use ONLY pop3 and imap4 and do not access the cyrus mail store 
directly (file-based). This won't work in a useable manner and may break up 
your whole store. If you already did this, reconstruct(8) should be your 
friend.

Pascal
--
Pascal Gienger, Kanzleistrasse 14, D-78462 Konstanz
Postfix+SASL-AUTH+LDAP Solaris 8:   http://pgienger.de/postfix/

Re: Cyrus 2.1.13 and cygwin?

[Pascal Gienger]

'Etienne Goyer' <[EMAIL PROTECTED]> wrote:

when you compiled SASL 2 ?  Did you changed the prefix ?  By default,
SASL 2 will be installed in /usr/local; you need to make a soft link
from /usr/local/lib/sasl2 to /usr/lib/sasl2.
You can still configure sasl2 with "--with-plugindir=/usr/local/lib/sasl2" 
to get rid of that soft link.

Pascal
--
Pascal Gienger, Kanzleistrasse 14, D-78462 Konstanz
Postfix+SASL-AUTH+LDAP Solaris 8:   http://pgienger.de/postfix/

4.3.0 lmtpd: Virtual memory exhausted - problem :(

[Pascal Gienger]

I am using Cyrus IMAP 2.1.12 (Suse Linux package, no source install allowed 
here), and we see some errors from lmtpd saying "Virtual memory exhausted".

ulimits are unlimited for every resource (no limit), and there are 4 
Gigabytes of swap used by 5% and 2 Gigabytes of RAM.
System is GNU/Linux 2.4.20-64GB.

Our postfix mailer tries to deliver a message to a local lmtpd (Cyrus 
system) and here is the result:

Jun 30 07:38:43 rauch postfix/lmtp[3129]: EC7FD4400088: 
to=<[EMAIL PROTECTED]>, 
orig_to=<[EMAIL PROTECTED]>, 
relay=/mail/cyrus/socket/lmtp[/mail/cyrus/socket/lmtp], delay=142730, 
status=deferred (host /mail/cyrus/socket/lmtp[/mail/cyrus/socket/lmtp] 
said: 421 4.3.0 lmtpd: Virtual memory exhausted (in reply to end of DATA 
command))

Has anybody seen this already? Which memory lmtpd claims to be exhausted?
With no process limits at all?
Pascal
--
Pascal Gienger, Kanzleistrasse 14, D-78462 Konstanz
Postfix+SASL-AUTH+LDAP Solaris 8:   http://pgienger.de/postfix/

Re: 4.3.0 lmtpd: Virtual memory exhausted - problem :(

[Pascal Gienger]

I investigated and found a curiosity:

That user has installed a sieve script to sort mail.
In his script he used the keyword
 keep;

to keep the mail in his INBOX.

This "keep" is going havoc ("virtual memory exhausted").
After replacing "keep" with 'fileinto "INBOX"' all things went
well.
Is this a bug in the sieve function of lmtpd?

here the NON WORKING part:

if header :comparator "i;ascii-casemap" :contains "From" "Muchsel"  {
   addflag "\\Flagged";
   keep;
}
This sieve code results in "4.3.0 lmtpd: Virtual memory exhausted".
here the WORKING variant:

if header :comparator "i;ascii-casemap" :contains "From" "Muchsel"  {
   addflag "\\Flagged";
   fileinto "INBOX";
}
Any comments?

Pascal

Re: 4.3.0 lmtpd: Virtual memory exhausted - problem :(

[Pascal Gienger]

John Alton Tamplin <[EMAIL PROTECTED]> wrote:

Pascal Gienger wrote:

What version of DB are you running?  Does this happen immediately or only
after it has been running a while?  On Solaris 9 with DB 4.1.24-25, we
had a problem where DB functions would eventually return ENOSPC, which
No it is not a DB problem, it IS a sieve problem:

Look at this piece of sieve code:

if header  :contains "From" "Muchsel"  {
   addflag "\\Flagged";
   fileinto "INBOX";
   removeflag "\\Flagged";
}
It is generated by Horde Ingo. lmtpd breaks with "4.3.0 virtual memory 
exhausted". Ok the "removeflag" is worthless because the message is already 
in the INBOX - but it should be silently ignored.

When removing the "removeflag"-Statement, lmtpd will accept the mail 
normally.

Pascal
--
Pascal Gienger, Kanzleistrasse 14, D-78462 Konstanz
Postfix+SASL-AUTH+LDAP Solaris 8:   http://pgienger.de/postfix/

pop3d vs. Microsoft Outlook [Express]

[Pascal Gienger]

The Microsoft Mail Products seem to have a problem with the generated UIDLs 
of the Cyrus pop3d.

Cyrus pop3d uses UIDLs with a pattern of "UnixTimestamp.InternalNumber",
so the "UIDL" command results in something like this:
[...]
966 1049815830.12154
967 1049815830.12155
968 1049815830.12156
Apparently Outlook and OE have problems with this as they begin getting 
mail as "new" even if they already have downloaded it ("Leave mail on 
server").

I activated the telemetry log for a user using Outlook but this shows that 
Cyrus does things 100% correct. But Outlook and Outlook Express seem to 
randomly download mail.

It seems that Outlook only uses a fixed number of chars to hold the UIDL, 
the rest is dicarded. The problem arose when when message-numbers reached 
1000. So it seems that Outlook uses 14 or 15 chars. So then for Outlook all 
messages have the same UIDL, resulting in this behaviour.

Our previous mail system ("Sun iPlanet") used UIDLS as 18328-100938292. So 
the problem did not arise.

Had anybody the same problem?
We are encouring users to use IMAP4 if they wish to leave mail on the 
server. Besides this, we did not find a solution. And Microsoft does not 
reply to an error report.

Pascal

Order of SASL2 methods announced? (Cyrus IMAPD2)

[Pascal Gienger]

We have some kind of - problem.

We try to understand HOW the different SASL plugins are ordered when doing 
an announcement (. CAPABILITY).

The problem arises (again) with Microsoft Outlook and Outlook Express.

Outlook breaks when "AUTH=NTLM" is not the FIRST method announced! It gives 
me an error saying "DIGEST-MD5: authentication failed" in Outlook (sure, 
Microsoft products only handle GSSAPI, NTLM and plaintext).

It is nearly impossible to describe how much crap these apps are :(

Does anybody have an idea on how to change the order of the announcement?

Thanks!

Pascal

Re: Order of SASL2 methods announced? (Cyrus IMAPD2)

[Pascal Gienger]

Rob Siemborski <[EMAIL PROTECTED]> wrote:

Mostly Randomly.  Somewhat based on the order the plugin is loaded.
Security requirements of SASL basicly dictate that the client ignore the
order they are advertised.
The problem arises (again) with Microsoft Outlook and Outlook Express.

Outlook breaks when "AUTH=NTLM" is not the FIRST method announced! It
gives me an error saying "DIGEST-MD5: authentication failed" in Outlook
(sure, Microsoft products only handle GSSAPI, NTLM and plaintext).
So, if you don't want to use DIGEST (or whatever), restrict what is
advertised with sasl_mech_list.
So I would have to disable all but NTLM to be sure AUTH=NTLM is the first 
or only "AUTH" visible. No I won't do this for Microsoft users only because 
of their broken clients.
Users noticed the behaviour because sending mail with SPA/NTLM did work 
(our mail relays use sasl2 with postfix and there "AUTH NTLM"/"AUTH=NTLM" 
is surprisingly the first auth announced):

250-AUTH NTLM PLAIN LOGIN DIGEST-MD5 CRAM-MD5
250-AUTH=NTLM PLAIN LOGIN DIGEST-MD5 CRAM-MD5
So this worked. My imapd however gives this:

* CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ MAILBOX-REFERRALS NAMESPACE 
UIDPLUS ID NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND SORT 
THREAD=ORDEREDSUBJECT THREAD=REFERENCES IDLE STARTTLS AUTH=DIGEST-MD5 
AUTH=NTLM AUTH=CRAM-MD5 ANNOTATEMORE X-NETSCAPE

And Outlook ALWAYS tries to use "DIGEST-MD5" saying it can't do so. What a 
perfectly dumb and broken client.
I set up a fake imapd (using echo and read) to see how Outlook behaves when 
parsing "AUTH". When putting "AUTH=NTLM" before DIGEST-MD5, Outlook works. 
Quite funny. It's just for the record in case anybody experiences the same 
strange behaviour.

I won't change anything in my installation.
Outlook users can still use SSL if they don't want their password exposed.
Thank you for your clarification!

Pascal

Re: cyrus-imap and fetchmail's "keep"

[Pascal Gienger]

Michi Müller <[EMAIL PROTECTED]> wrote:


unix mboxes... now, using imap, it works as before (i can
see this in procmail's mail.log) but cyrus seems to drop
all the mails which it has already delivered before.
Yes, you just met that "duplicate delivery suppression" feature of your 
Cyrus lmtpd. The duplicate  delivery database is purged every 3 days (see 
the appropriate process in your cyrus.conf). You can turn this suppression 
off if you want:

Put in your /etc/imapd.conf (or whereever you have your config fule):

duplicatesuppression: no

HTH,
Pascal Gienger

Re: Order of SASL2 methods announced? (Cyrus IMAPD2)

[Pascal Gienger]

Nikola Milutinovic <[EMAIL PROTECTED]> wrote:

250-AUTH=NTLM PLAIN LOGIN DIGEST-MD5 CRAM-MD5
I don't have this second line on Sendmail 8.12.9 - am I missing something?
No, it is for (again) broken clients which are using "AUTH=". Some kind of 
legacy auth.

Is it for sure that O and OE cannot use DIGEST-MD5? Why do they try at
all? Is DIGEST-MD5 actually working on your IMAP? Try "imtest".
DIGEST-MD5 works. I am using it every day with Mulberry.

With the latest installments, you should be able to use Kerberos5 as well
(GSS-API), since Active Directory uses it as an authentication engine.
Maybe *that* will help you sort out M$ sillyness...
Our mail servers don't have any GSS-Modules installed. And we don't want to 
support it as well.

Thanks for your reply! I'll try my luck with the plugin files ordering :-)

Pascal

Re: How to hand over mails from postfix to cyrus

[Pascal Gienger]

"Oliver Demetz - Hardware-XPress.de" <[EMAIL PROTECTED]> wrote:

The commandline, which is executed by Postfix (defined in master.cf)
is:
pipe user=cyrus argv=/usr/cyrus/bin/deliver -e -r ${sender} -m
${extension} ${user} so, deliver is told to store the mail into a
Don't use that. Use a virtual alias table (see Postfix docs) and use lmtp 
to send mail to your cyrus store. See "lmtp" transport and "lmtpunix/lmtpd" 
in your cyrus.conf. Use a unix socket if you have both services on the same 
machine.

Pascal

Re: How to hand over mails from postfix to cyrus

[Pascal Gienger]

Like I already told you in the postfix-users-list, the best way to do local 
mailbox delivery with postfix and cyrus is using lmtp over a local socket 
(if both services are on the same machine).

Cyrus lmtpd expects a valid user/mailbox name as local part. So create 
valid address to sasl usernames using a postfix virtual table. I described 
it for you on the postfix-users list because this is not a cyrus imap 
problem. ;-)

Pascal

Re: Postfix can't connect to lmtp

[Pascal Gienger]

[EMAIL PROTECTED] wrote:

(connect  to /var/imap/socket/lmtp[/var/imap/socket/lmtp]: Permission
denied)
The file system permissions of your socket file /var/imap/socket/lmtp(*) are
set in a way so that the postfix process (running under user "postfix")
just can't write to it.
Pascal

(*) The name and location of the socket is defined in your cyrus.conf,
   lmtpd(8) via unix socket transport.

Re: Hight Aviability and Cyrus

[Pascal Gienger]

Earl R Shannon <[EMAIL PROTECTED]> wrote:

Bonjour,

While there have been a couple of mentions that high availability is
being considered by CMU, it has not been done natively to the
IMAP server. In other words, the IMAP server does not do high
availability. While it does have the cluster implementation,
( the murder ) this allows scalability, not high availablity.
The only possible thing which can be setup in a timely doable manner is a 
failsafe cluster configuration. Use e.g. a little SAN Network with dual 
FC-Switches so that both computers forming the IMAP Cluster see the same 
partitions. Use some kind of cluster software to do it. Writing an agent 
for Veritas Cluster is not difficult, nor it is for the Linux HA project.

Doing this for parallel clusters is quite of lot of work, because you would 
have to keep much data in sync.

Pascal

Re: Mailbox

[Pascal Gienger]

Norman Zhang <[EMAIL PROTECTED]> wrote:

Hi,

I created a mailbox nzhang by

1. cyradm localhost
2. cm user.nzhang
3. quit
4. saslpasswd nzhang
Err it should be "saslpasswd2" - otherwise you are using a SASLv1 linked 
imap server, e.g. Cyrus 1.6.x...

But I don't see nzhang created under /var/spool/mail. I did set
mail_spool_directory = /var/spool/mail in /etc/postfix/main.cf. Is there
something trivial that I'm missing? Do I need smtpd.conf (I can't find
this file)? Won't postfix take care of mail transfer?
mail_transport = lmtp:$myhostname
No...
Use
fallback_transport = lmtp:unix:/path/to/your/lmtp/socket/lmtp

if you want to get mail delivered in your cyrus imap spool and local unix 
users will still have their /var/spool/mail-Files.

The Cyrus mail system does NOT use normal mailfiles, it has its own spool.

Pascal

Re: Changelog, LDAP features

[Pascal Gienger]

Thomas Luzat <[EMAIL PROTECTED]> wrote:

a) Fetching Sieve-scripts from LDAP (would guess not)
b) Fetching Quota settings from LDAP (same)
It's probably best to write some LDAP->Cyrus gateway for that, right?


For the university of Constance I wrote a little Daemon program which 
synchronizes OpenLDAP with Cyrus databases (and mailboxes) - because they 
did not want to compile the postfix and cyrus things themselves (lack of 
support).

It uses the rather simple openldap-replication-mechanism to accomplish this.

For the staff there, the postfix/cyrus server is completely in the LDAP 
tree including passwords, quotas, forwards  and autoreplies (via a special 
autoreply program, also written by myself because most of them lying around 
send out too many autoreplies (to lists, errors, ...) and could not take 
the autoreply message via LDAP. Here we use the standard 
autoreply-LDAP-attributes.).

Mailboxes get created automatically when an LDAP entry comes in and it gets 
deactivated when it is removed.

So the user support personnel can just create an LDAP entry to make a valid 
postfix-alias and cyrus mailbox available immediately.

Works like a charm but it is not very elegant (I must admit it). I did not 
find any other solutions than to write it on my own.

If there are other solutions, let me know.
I packaged the whole system to a package named "priscilla" -
Pascal

Re: Changelog, LDAP features

[Pascal Gienger]

Simon Matter <[EMAIL PROTECTED]> wrote:


The University of Athens is doing some cool work here
http://email.uoa.gr/projects/cyrusmaster/
I'll take a look at it!


What is the license of your package, can it be downloaded somewhere?
Yes it will be next week. Had enough time to test it, it works in 
production for approx 15000 users since 6 months now - the daemon logs 
activity via syslog.

It is not a "click here to install"-Package though, you will have to read 
readme-Files and edit some configurations with a text editor... See it as a 
glue between cyrus and OpenLDAP.

Pascal

Re: Marking Messages going to Cyrus as read procmail.

[Pascal Gienger]

Hexren <[EMAIL PROTECTED]> wrote:


Now what I think I need is the following:
Based on  I want to deliver the mail and have
cyrus mark it as "read" immediatly.


Why don't you consider using sieve for that task?

Pascal

Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html