> On Thu, Nov 15, 2001 at 09:58:09AM -0600, John Wade wrote:
>
> > If anyone is interested in the full set of steps, I could post it to the
> > list.
Hi All,
Since a number of people requested it, I am sending my complete list of steps (with
added comments) for moving from one Cyrus server to another. Sorry about the size of
this posting
I used this procedure to migrate two servers from older Pentium Pro hardware running
Cyrus 1.5.19 to newer P3 Xeon servers running Cyrus 2.0.16. One server had ~20,000
accounts and 10GB of mail, the other had ~1300 accounts and 35GB of mail. These
instructions assume that you have both servers set up and fully tested before moving
the users' data. Our new environment uses:
RedHat 7.0 (2.2.19 kernel)
Cyrus 2.0.16 for IMAP
Cyrus SASL 1.5.24 with LDAP authentication (via hacked pwcheck)
qmail 1.0.3 as the MTA
ezmlm-idx for mailing lists
WebSieve and Easysieve for sieve
The mail on the old server was all stored in a single 35GB ext2fs file system mounted
as /var. Within this file system, we had hashed the mail spool via cyrus partitions
using the first letter of the user's login id to choose the partition (i.e.
/var/spool/imap/a/ for abelincoln ) This is similar to the new hashmailspool option.
On the new server, we had a 190GB array and I was afraid to make this a single file
system. In order to end up with a managable number of file systems (of manageable
size) , we decided to store mail in 5 - 33GB file systems ( mounted
as/var/spool/imap/0, /var/spool/imap/1, /var/spool/imap/2, /var/spool/imap/3 and
/var/spool/imap/4) This corresponded to 5 matching Cyrus partitions (0,1,2,3,4)
IMPORTANT DISCLAIMER: While I sometimes feel like I know what I am doing, there is no
guarranty that I did this in the most efficient way or even that I did it correctly,
all I know is that it seems to have worked for me
Warning, Since I was using external authentication (via LDAP), there is nothing here
about moving the authentication database.
The following instructions were designed so that I could cut and paste it to reduce
mistakes
and speed the conversion. All comments are proceeded by a # Sometimes the
instructions
call for editing files with vi, in this case, the next set of lines separated by
#--------
are what should be the new contents of the edited file.
----------------------------------------------------------------------------------
#Cleanup newserver
# On new server shutdown services
# The cyrus and pwcheck scripts are ones I put together to control cyrus and pwcheck.
# basically this step was just to kill all existing services on the new box
# your steps will differ.
/etc/rc.d/init.d/cyrus stop
/etc/rc.d/init.d/pwcheck stop
/etc/rc.d/init.d/qmailctl stop
/etc/rc.d/init.d/httpd stop
# Cleanup any existing mail directories, quotas, seen and subscription files, etc that
might be
#left over from testing
rm -f /var/imap/proc/*
rm -f /var/imap/mailboxes*
rm -f /var/imap/db/*.*
rm -f /var/imap/deliverdb/*.db
rm -f /var/imap/deliverdb/db/*
# delete subscriptions and seen files
cd /var/imap/user
for dir in `ls -d ?`; do rm -f /var/imap/user/$dir/* ; done
# delete quotas
cd /var/imap/quota
for dir in `ls -d ?`; do rm -f /var/imap/quota/$dir/* ; done
# delete sieve scripts (note this is not the default sieve directory)
cd /var/imap/sieve
for dir in `ls -d ?`; do rm -Rf /var/imap/sieve/$dir/* ; done
#Delete the mail spool (note we use multiple partitions in /var/spool/imap)
cd /var/spool/imap
for dir in `ls -d ?`; do rm -Rf /var/spool/imap/$dir/* ; done
# cleanp ezmlm mailing lists
# (This step omited from this posting since this is ezmlm specific)
# done with cleanup
# At this point the new server should be completely cleaned up and
# ready for the copy
-------------------------------------------------------------
# Restore Overview
# We need to restore all items in
# /var/imap/user/
# /var/imap/quota/
# /var/imap/mailboxes (file)
# /var/spool/imap/
#NFS Setup. I mounted the new server's file systems on the old server
# because we had a problem with the nfs install on the old server, ideally,
# I would have mounted the old on the new because then I could have done
# it read only and not risked any mistakes.
#on newserver, setup an export for each mount point that will need to have files
copied to it.
vi /etc/exports
#-------------------------------
/var oldserver.oakton.edu(rw,no_root_squash)
/var/imap oldserver.oakton.edu(rw,no_root_squash)
/var/spool/imap/0 oldserver.oakton.edu(rw,no_root_squash)
/var/spool/imap/1 oldserver.oakton.edu(rw,no_root_squash)
/var/spool/imap/2 oldserver.oakton.edu(rw,no_root_squash)
/var/spool/imap/3 oldserver.oakton.edu(rw,no_root_squash)
/var/spool/imap/4 oldserver.oakton.edu(rw,no_root_squash)
#-------------------------------
/etc/rc.d/init.d/nfs stop
/etc/rc.d/init.d/nfslock stop
/etc/rc.d/init.d/portmap stop
/etc/rc.d/init.d/portmap start
/etc/rc.d/init.d/nfslock start
/etc/rc.d/init.d/nfs start
# to reload (if you make future changes to /etc/exports )
#exportfs -a
#on old server mount nfs shares from new server
mkdir -p /mnt/newserver/var
mount -t nfs newserver.oakton.edu:/var /mnt/newserver/var
mount -t nfs newserver.oakton.edu:/var/imap /mnt/newserver/var/imap
mount -t nfs newserver.oakton.edu:/var/spool/imap/0 /mnt/newserver/var/spool/imap/0
mount -t nfs newserver.oakton.edu:/var/spool/imap/1 /mnt/newserver/var/spool/imap/1
mount -t nfs newserver.oakton.edu:/var/spool/imap/2 /mnt/newserver/var/spool/imap/2
mount -t nfs newserver.oakton.edu:/var/spool/imap/3 /mnt/newserver/var/spool/imap/3
mount -t nfs newserver.oakton.edu:/var/spool/imap/4 /mnt/newserver/var/spool/imap/4
# On Old Server
#increase quotas to allow mail to be delivered (We often have users over quota,
# and their mail is hanging out in the queue waiting for either the user to clean up
# their mailbox or for 10 days to elapse so it can bounce
# by increasing the quota, I allowed all this mail to be delivered and did not have
# to move the MTA's queue (We were going to increase the quota on the new
# server anyway). Note this script could have been writen to use the IMAP::Admin
# module instead of Net::Telnet, but we already had the basic framework and
# IMAP::Admin was not a part of cyrus 1.5
# get a listof all existing quotas
ls /var/imap/quota > /root/currentusers.txt
#create script
vi /root/adjquota.pl
---------------------------
#!/usr/bin/perl -w
use Net::Telnet;
use File::Find;
$hostname = "oldserver.oakton.edu";
$username = "<cyrus admin>";
$passwd = "password";
$quota = 60000;
## Open up a new session to imap server
$imap = new Net::Telnet (Telnetmode => 0);
$imap->open(Host => $hostname, Port => 143);
## Read connection message.
$line = $imap->getline;
die $line unless $line =~ /^\* OK/;
print "connected to imap server ok\n";
## Send admin user name.
$imap->print(". login $username $passwd");
$line = $imap->getline;
die $line unless $line =~ /^\. OK/;
print "logged in to imap server ok\n";
## Read each entry in file
open (P, "currentusers.txt") or die "Can't find the currentusers.txt file\n";
while (<P>) {
chomp;
$imap->print(qq{. setquota $_ (storage $quota)});
$line = $imap->getline;
die "$_ " . $line unless $line =~ /^\. OK/;
print "quota root set to $quota for $_\n";
}
close(P);
$imap->print(". logout");
exit;
-------------------------------
chmod 700 /root/adjquota.pl
/root/adjquota.pl
# Signal qmail to re-attempt delivery (Your MTA will work differently)
ps -ef | grep qmail-send
#Get process id
kill -SIGALRM <PID>
#check queue
qmHandle -s
qmHandle -L | less
#Adjust quotas if necessary to allow delivery
cyradm -u <cyrus admin> localhost
lqm user.<username>
sql user.<username> 65000
quit
#repeat SIGALRM and queue check until no mail is queued locally
#shutdown services on old server
Stop all mail services (shutdown qmail)
Comment out imapd line in /etc/inetd.conf - this disables people's ability to use IMAP
to access their
email. Then kill -HUP <inetd pid>
#Copy mailboxes file (on old server)
cp /var/imap/mailboxes /mnt/newserver/var/imap/mailboxes
#on old server
# (in Cyrus 1.5 quotas were all in one directory, in 2.0.16, the are hashed by the
first letter
# of the username, this script will put all of them in the correct directory (unless
they begin with a
# number or other character :-)
vi /root/CopyImapQuotas.sh
-------------------------------
# To copy quotas
for letter in a b c d e f g h i j k l m n o p q r s t u v w x y z
do
echo $letter
cp /var/imap/quota/user.$letter* /mnt/newserver/var/imap/quota/$letter/
done
---------------------------------
chmod 700 CopyImapQuotas.sh
/root/CopyImapQuotas.sh
# Same thing for subscriptions
vi /root/CopyImapSub.sh
-------------------------------
# To copy subscriptions
for letter in a b c d e f g h i j k l m n o p q r s t u v w x y z
do
echo $letter
cp /var/imap/user/$letter*.sub /mnt/newserver/var/imap/user/$letter/
done
---------------------------------
chmod 700 CopyImapSub.sh
/root/CopyImapSub.sh
# Create CopyImapSpool.sh script on old server and execute
# Note that this script is more complicated than others might need
# because we were trying to redistribute users into a new partitioning
# structure. On the old server a user's mail would have been in a partition
# based on the first letter of their name. For example, my directory would
#have been /var/spool/imap/j/user/jwade
#On the new server, we wanted to evenly distribute users into 5 partitions
#so that we could keep the file system size managable and evenly distribute
#disk space usage between the mount points.
# The idhash program that I use in the following script is a little program I wrote
#C (I also have a perl version) that calculates the partition number by adding
# up the ascii values of all the characters in the user's name, dividing by five
# and using the remainder. (modulus operator) This fairly simple hash function
# gives a pretty good distribution of users.
#Our script to add users uses this same "idhash" program to assing new users
#to partitions in the same way.
vi /root/CopyImapSpool.sh
------------------------------------------------------
# To copy users
# make sure directories exist
for num in 0 1 2 3 4
do
mkdir /mnt/newserver/var/spool/imap/$num/user/
done
# copy mail
for letter in a b c d e f g h i j k l m n o p q r s t u v w x y z
do
echo $letter
cd /var/spool/imap/$letter/user
for uid in *
do
echo $letter $uid `/root/idhash -n $uid`
cp -R /var/spool/imap/$letter/user/$uid
/mnt/newserver/var/spool/imap/`/root/idhash -n $uid`/user/
done
done
-----------------------------------
chmod 700 CopyImapSpool.sh
/root/CopyImapSpool.sh
# for 35GB of mail (1,668,019 files), this process took 6 hours 32 minutes. This was
far faster than when
# we tested it doing a backup and restore from tape
#on old server copy ezmlm files
(Step omitted here for brevity
#----------------------------------------------------------------------
#Steps to complete After Restore
# on newserver
#Next sections are to fix up file ownership, hopefully the umask took
# care of permissions
#fix subscrptions
chown -R cyrus:mail /var/imap/user/*
#fix quotas
chown -R cyrus:mail /var/imap/quota/*
#fix mail spool
chown -R cyrus:mail /var/spool/imap/*
#fix ezmlm
# (step omitted here for brevity)
# fix Mailboxes file
# I needed to do three things with the mailboxes file, change everyone's partitions
#to match our new partitioning structure (switch from a-z to 0-4), change the cyrus
admin user's permissions
#from "d" to"c" so that we could delete mailboxes (This was a change between
# 1.5.x and 2.0.x) and convert the flat text mailboxes file into the Berkeley DB that
2.0.x
# needs
cp /var/imap/mailboxes /var/imap/mailboxes.old
# the following perl script is what fixes the partition (using that same idhash
program)
# and fixes the permissions for the cyrus admin user, I can cheat here, since I
# know the cyrus admin user's acl is the last entry on each line in our mailboxes
file.
vi /root/fixmailboxes.pl
------------------------------------
while (<>) {
$line = $_;
# find login id
/^user\.(\w+)/;
# calculate new partition
$idhash = `idhash -n $1`;
# fix partition replace a-z with calculated 0-4
$line =~ s/(^user\.\w+[^\t]*\t)([a-z])/$1$idhash/ ;
# fix permissions for cyrus admin user
$line =~ s/d\t$/c\t/;
print $line;
}
-----------------------------------
perl -w /root/fixmailboxes.pl < /var/imap/mailboxes.old > /var/imap/mailboxes
# The following steps are used to convert the old malbox text file (after cleanup)
# to the new berkeley db
# for those who need more info about ctl_mboxlist, see the man page
#basically, the two options that apply to conversions are -u to convert
#from text file to db and -d to dump from db to text file
chown cyrus:mail /var/imap/mailboxes
su cyrus
cd /var/imap
/usr/cyrus/bin/ctl_mboxlist -u < mailboxes
/usr/cyrus/bin/ctl_mboxlist -c
exit
# Disconnect on old server
cat /etc/mtab
umount -t nfs newserver.oakton.edu:/var/imap
umount -t nfs newserver.oakton.edu:/var/spool/imap/0
umount -t nfs 1newserver.oakton.edu:/var/spool/imap/1
umount -t nfs newserver.oakton.edu:/var/spool/imap/2
umount -t nfs newserver.oakton.edu:/var/spool/imap/3
umount -t nfs newserver.oakton.edu:/var/spool/imap/4
umount -t nfs newserver.oakton.edu:/var
#shutdown rpc and nfs on new server (for security)
/etc/rc.d/init.d/nfs stop
/etc/rc.d/init.d/nfslock stop
/etc/rc.d/init.d/portmap stop
chkconfig --level 123456 nfs off
chkconfig --level 123456 nfslock off
chkconfig --level 123456 portmap off
chkconfig --list | less
#verify copy (do on both boxes)
#Verify count of subscriptions
#on old server
ls /var/imap/user/* | wc -l
#on new server
ls /var/imap/user/?/* | wc -l
#Verify count of quotas
#on old server
ls /var/imap/quota/* | wc -l
#on new server
ls /var/imap/quota/?/* | wc -l
#Verify count of spool dirs
#on old server
ls -ld /var/spool/imap/?/user/* | wc -l
#on new server
ls -ld /var/spool/imap/?/user/* | wc -l
#check count of files
#on old server
ls -R /var/spool/imap/?/user/* | wc -l
#on new server
ls -R /var/spool/imap/?/user/* | wc -l
# restart services
/etc/rc.d/init.d/pwcheck start
/etc/rc.d/init.d/cyrus start
qmailctl start
/etc/rc.d/init.d/httpd start
# test everything !!!!!
#if it is really bad, you can just go back to the old server at this point
# Change IP Address and hostname. Verify files with :
#convert name and ip
#Since we wanted to use the same dns name and ip address for the new server,
#at this point, we powered off the old server
#to make sure I found all references to the host name and ip address, I had previosly
searched the
#file system fir them
#on new server before copy
for dir in bin boot command dev etc home lib mnt nsr opt package root sbin service tmp
usr var
do
echo checking $dir
find /$dir -type f -exec grep -l "192\.168\.5\.105" {} \;
done
for dir in bin boot command dev etc home lib mnt nsr opt package root sbin service
tmp
usr var
do
echo checking $dir
find /$dir -type f -exec grep -l "newserver" {} \;
done
#this found the following files
# edit each and replace new ip (192.168.5.105) with old ip (192.168.5.5)
/etc/sysconfig/network-scripts/ifcfg-eth0
/etc/hosts
#these two are from qmail
/etc/smtp.txt
/etc/smtp.cdb
# edit following files and replace "newserver" with "oldserver" in
/etc/sysconfig/network
/etc/hosts
#qmail specific
/var/qmail/control/me
/var/qmail/control/locals
/var/qmail/control/rcpthosts
# paths omitted below for web stuff
websieve.conf
easysieve.pl
horde/imp/config/defaults.php3
#reboot
shutdown -r now
#At this point queued mail on our mail relay started to be delivered, and everything
was fine.
#I have omtted a couple of steps here that were specific to imp (moving prefs and
addressbook) and ezmlm
# ( moving mailing lists)
I still wonder if I should have reconstructed all the mailboxes (with
/usr/cyrus/bin/reconstruct) and fixed the quotas (/usr/cyrus/bin/quota -f ) before
proceeding, but I did not
Hope this was helpful,
John
