>
> I had an eye on Su's postings the last days because of own interest in
> getting performance messurement data examples. I then started to do my
> own tests because I just wanted to see how my installation performs and
> was surprised that I got a much better performance without knowing that
> much about filesystems and BerkleyDB than he actually does. Maybe
> someone can handle me a script or something that shows me how you
> messure things so that I can be shure that I do the same messurements
> than you all do! Would be great to see how other people are testing
> theire email performance...
>
> ---Christian---
Perhaps the best thing to do for folks setting up cyrus high end mail systems
would be to describe our own setups and let folks (like Su) look them over to
see what they can pick off for his own setup.
The stuff I'm currently working on is based on Linux, so most of my setup is
linux centric, but the generic basics would apply to most systems. My system
would look roughtly as follows (pardon if my ascii art sucks):
+------------+ +------------+ +------------+ +------------+
| mta server | | mta server | | imap proxy | | imap proxy |
| postfix | | postfix | | perdition | | perdition |
+------------+ +------------+ +------------+ +------------+
| | \ | \ | \
| | \ | \ | \
\ / \ \ \ / \
\ / \ \ \ / \
\ / \ \ \ / \
----------- \ ----------- \
/ \ \ / \ \ +----------+
/ \ \----/--------------------------| ldap dir |
+------------+ +------------+ / | openldap |
| imap server| | imap server| / +----------+
| cyrus |--| cyrus |-------
+------------+ +------------+
Optimization for the mta servers is as follows:
Two sets of disk mirrors at RAID 0
First set is the system disks
Second set is the postfix mail queues
Use a nice hardware RAID controller with
a fat slice of cache (128MB cache on an
Mylex ExtremeRaid here). Set the cache to
run in writeback mode.
Setup the mail queus on ext3 fs with full
journaling data=journal (note there are
some details Stephen Tweedie is working on
with regards to data=journal, they've been
fixed but in patches, not mainline yet)
Create the ext3 journal on a separate set
of disks, specifically allocate a partition
to hold the journal (I put the journal on a
dedication partition on the system disks as
their load normally isn't all that
high). Make the journal HUGE, ie 250 + MB.
Mount the mail queues with the noatime mount
option. Also mount /var/log with noatime.
chattr -S /var/log and the mail queues.
Disable syncronous logging in syslogd or
use network syslogging to another host.
Consider using tmpfs for lmtpd temp directory
(I haven't done this yet, but I really like
the idea).
Bump the number of open file descriptors
available to the system to something like
16k - 32k.
Use direct TCP lmtp delivery to the cyrus
backends.
Optimization for the cyrus servers is as follows:
All the same optimizations for the mta's.
Don't run a specific mta on the backend
(use TCP lmtp). Note: I do run an mta
but it's only used for local system
mail and notifications, not for cyrus
delivery.
Allocate multiple cyrus partitions. Keep
each partition on it's own RAID 0 disk set.
Keep the partitions on a single disk pair
(don't stripe 6 drives together, break them
up into 3 two disk stripe sets, one set for each
partition). Maximize the number of independent
spindles. Allocate the journals on separate
disks.
chattr -S the mail partitions (I do keep the
mailboxes.db, etc, in the conf dir syncronous)
Disable duplicate delivery suppression.
Theory of operation:
Operation of the system hinges on LDAP. LDAP is used by postfix
for all the mail routing (to get the message to the
correct cyrus backend). It's also used for authentication
(saslauthd). It's also used by the perdition proxies to
determine the correct cyrus backend to route connections
to.
An LDAP entry looks as follows:
dn: cn=john.doe,ou=real,ou=addresses,ou=mail,dc=somecompany,dc=com
sn: Doe
givenName: John
userPassword: {MD5}+/xxxxxxxxxxxxDZgg==
description: mail user account
uid: [EMAIL PROTECTED]
cn: john.doe
mail: [EMAIL PROTECTED]
mail: [EMAIL PROTECTED]
mail: [EMAIL PROTECTED]
mailLocalAddress: [EMAIL PROTECTED]
smtpauthAccess: enabled
popimapAccess: enabled
smtprouteAccess: enabled
I have postfix transport maps for all the cyrus backends in
postfix (mail-store1, mail-store2, .....) to route the messages
via ltmp to the correct backend. There's an LDAP virtual map on
the postfix mtas that looks like this:
virtual_maps = ldap:ldapvirtual, hash:/etc/postfix/virtual
ldapvirtual_server_host = ldap1.somecompany.com
ldapvirtual_server_port = 389
ldapvirtual_search_base = ou=addresses,ou=mail,dc=somecompany,dc=com
ldapvirtual_timeout = 20
ldapvirtual_query_filter = (&(mail=%s)(smtprouteAccess=enabled))
ldapvirtual_result_attribute = mailLocalAddress, mailRoutingAddress
ldapvirtual_scope = sub
ldapvirtual_bind = yes
ldapvirtual_bind_dn = cn=mailro,dc=somecompany,dc=com
ldapvirtual_bind_pw = xxxxxxx
Then, when a message arrives for [EMAIL PROTECTED] or
[EMAIL PROTECTED] the virtual map rewrites the
_envelope_ address (the original To: address is preserved)
to [EMAIL PROTECTED] From there lmtp
delivery kicks it to mail-store1 which has a mailbox for
user/john.doe (unix separator).
If the user connects to perdition to check his mail. He logs in
as the user [EMAIL PROTECTED] (aka the uid). Perdition does
an LDAP query to resolve the true mail backend. Specifically it
uses the mailLocalAddress with the mailbox being the username
and the domain being the host. Something like
<username>@<mail backend>. It then connects to the correct mail
backend (mail-store1.somecompany.com in this case) as the
user john.doe. Note: There _is_ a mailbox john.doe on
mail-store1, which is legal.
Cyrus then attemps to authenticate the user john.doe against
LDAP via saslauthd. Saslauthd is setup as:
ldap_servers: ldap://ldap1.somecompany.com/
ldap_bind_dn: cn=mailro,dc=somecompany,dc=com
ldap_bind_pw: xxxxxxx
ldap_timeout: 20
ldap_scope: one
ldap_search_base: ou=real,ou=addresses,ou=mail,dc=somecompany,dc=com
ldap_auth_method: bind
ldap_filter: (&(cn=%u)(popimapAccess=enabled))
ldap_debug: 2
ldap_verbose: on
ldap_ssl: no
ldap_start_tls: no
Saslauthd resolves the user via the cn attribute (hence why it is
john.doe) and uses the userPassword attribute for the password.
Postfix is also tweaked to allow SMTP AUTH access as well, using
a running instance of saslauthd on the mtas for password
resolution. The SMTP AUTH username is the same [EMAIL PROTECTED]
(the uid attribute). Access to SMTP AUTH is also controlled via
the smtpauthAccess attribute.
Of course, since saslauthd is in use (thus requiring plain text
passwords everywhere), IMAP connections to the perdition
proxies as well as the SMTP AUTH connections to the mtas are
protected via SSL/TLS.
Management of the system is performed via a custom MySQL database
backend fronted by some perl scripts. The perl scripts can put
entries into LDAP as well as connect to cyrus (via the cyradm module)
to create the required mailboxes.
From this setup, the cyrus backends are horizontally scaleable
(just add more backends) and the postfix mtas are horizontally
scaleable (add more if load demands it). Mtas, proxies, etc,
are either load balanced via round robin DNS or some hardware
based solution.
Whew, okay, I'm done typing for now. Hope this helps :)
Cheers,
Jeremy
