Delayed delete, restoring deleted mailboxes

[Leena Heino]

What is the recommended process to restore deleted messages or mailboxes 
when delayed delete is in use?

-- 
   Leena Heino  University of Tampere / Computer Centre
   ( liinu at uta.fi )  ( http://www.uta.fi/laitokset/tkk )

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Delayed delete, restoring deleted mailboxes

[Adam Tauno Williams]

On Tue, 2009-03-17 at 13:26 +0200, Leena Heino wrote:
> What is the recommended process to restore deleted messages or mailboxes 
> when delayed delete is in use?

Use the unexpunge command;  the man page isn't bad, and I have examples
and some notes in the Cyrus chapter of WMOGAG
.
-- 
OpenGroupware developer: awill...@whitemice.org

OpenGroupare & Cyrus IMAPd documenation @



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Delayed delete, restoring deleted mailboxes

[Joseph Brennan]

--On Tuesday, March 17, 2009 7:57 -0400 Adam Tauno Williams 
 wrote:

> On Tue, 2009-03-17 at 13:26 +0200, Leena Heino wrote:
>> What is the recommended process to restore deleted messages or mailboxes
>> when delayed delete is in use?
>
> Use the unexpunge command;  the man page isn't bad, and I have examples
> and some notes in the Cyrus chapter of WMOGAG
> .


Unexpunge for messages.

For entire mailboxes, use rename.  You can find the name by doing a
"lm *string*" in cyradm.  It will start "DELETED." and end with an
extra string.

Joseph Brennan
Lead Email Systems Engineer
Columbia University Information Technology


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Delayed delete, restoring deleted mailboxes

[Leena Heino]

On Tue, 17 Mar 2009, Adam Tauno Williams wrote:

> On Tue, 2009-03-17 at 13:26 +0200, Leena Heino wrote:
>> What is the recommended process to restore deleted messages or mailboxes
>> when delayed delete is in use?
>
> Use the unexpunge command;  the man page isn't bad, and I have examples
> and some notes in the Cyrus chapter of WMOGAG
> .

The unexpunge command was the first thing I tried and I read it's man 
page. But what puzzles me is that it seems that the deleted mailboxes 
are not shown in unexpunge commands output.

The mailbox meta files and mailbox's messages are shown in the spool dir 
under the DELETED prefix eg. in the directory:
/cyrus_spool/u/DELETED/user/foo/sent-mail/HEXSTRING/

What is the correct procedure the restore this mailbox so that user is 
able see the messages and access it once again.

-- 
   Leena Heino  University of Tampere / Computer Centre
   ( liinu at uta.fi )  ( http://www.uta.fi/laitokset/tkk )

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Delayed delete, restoring deleted mailboxes

[Kenneth Marshall]

On Tue, Mar 17, 2009 at 03:06:15PM +0200, Leena Heino wrote:
> On Tue, 17 Mar 2009, Adam Tauno Williams wrote:
> 
> > On Tue, 2009-03-17 at 13:26 +0200, Leena Heino wrote:
> >> What is the recommended process to restore deleted messages or mailboxes
> >> when delayed delete is in use?
> >
> > Use the unexpunge command;  the man page isn't bad, and I have examples
> > and some notes in the Cyrus chapter of WMOGAG
> > .
> 
> The unexpunge command was the first thing I tried and I read it's man 
> page. But what puzzles me is that it seems that the deleted mailboxes 
> are not shown in unexpunge commands output.
> 
> The mailbox meta files and mailbox's messages are shown in the spool dir 
> under the DELETED prefix eg. in the directory:
> /cyrus_spool/u/DELETED/user/foo/sent-mail/HEXSTRING/
> 
> What is the correct procedure the restore this mailbox so that user is 
> able see the messages and access it once again.
> 
You will need to move the mailbox back to the original location in
the user's tree. unexpunge works on messages not entire folders.

Cheers,
Ken


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: ctl_cyrusdb questions, looking for answers

[Michal Hlavinka]

> ...
> > 1) Listing of database files "checkpointed"
> > 2) Description of checkpoint format
> >   a) "ls /var/lib/imap" suggest two sets of checkpoint database are
> > retained.
>
> I'm not really sure here. Cyrus-imapd checkpoints databases and archives
> some of them (mboxlist_db, quota_db, annotation_db). Archives are put and
> rotated into the db.backup[1,2] dirs.

so are other databases (other then mboxlist_db, quota_db, annotation_db) 
checkpointed or not? 

>
> >   b) Description of checkpoint configuration options, if any, to control
> > checkpoint sets
> > 3) Checkpointed image backup requirements
>
> Not exactly sure what you mean here. I think you only need the database in
> db.backup? and of course clean the environment before restoring such a db.

AFAIK only that three databases are copied to the db.backupX directory. So if 
there are other checkpointed dbs they stay in the original place and I need to 
backup them?


> > 5) Appropriate recovery procedure when invocation with "-r" vs. "-r -x'
> > is required.
>
> I don't remember to have used -x in the last 10 years using cyrus-imapd. I
> really don't know what the cleanup part means. Is it cleaning up unused
> space in a db file or does it mean handling of open transactions?

thats what I'm wondering :)

>
> Regards,
> Simon

Thanks for the answers!

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: ctl_cyrusdb questions, looking for answers

[Michal Hlavinka]

Thanks for the answers!

> ...
>
> > The information that man does not provide is
> > 1) Listing of database files "checkpointed"
>
> I've always assumed all-of-them

excerpt from the source:

...
/* if we need to archive this db, add it to the list */
if (dblist[i].archive) {
...dblist[i] is added to archive_files list...
}

2 = (*(dblist[i].env))->sync(); 
for some dbs (afaik for those who has .archive=1) this is empty 
function

r2 = (*(dblist[i].env))->archive((const char**) archive_files,
 backup1);
... but dbs with .archive=0 are not in archive_files list
...
this dbs have .archive=1 : mboxlist_db, quota_db, annotation_db
other have .archive=0



>
> > 2) Description of checkpoint format
> >   a) "ls /var/lib/imap" suggest two sets of checkpoint database are
> > retained.
>
> ? Not following you here.  It checkpoints the databases,  not certain
> what "two sets" of what your seeing.

it's related to prev. question. The archive function (used only for some dbs) 
copies them to db.backupX directory. Other dbs are maybe somehow checkpointed 
by the sync call (that is non-empty function for them). So it seems not all 
dbs are checkpointed or maybe not all of them are in db.backupX ?




Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: ctl_cyrusdb questions, looking for answers

[Adam Tauno Williams]

> > Not exactly sure what you mean here. I think you only need the database in
> > db.backup? and of course clean the environment before restoring such a db.
> AFAIK only that three databases are copied to the db.backupX directory. So if 
> there are other checkpointed dbs they stay in the original place and I need 
> to 
> backup them?

As I understand it: there is no reason to backup deliver.db,
statuscache.db, or tls_sessions.db.  You can dump those without
consequence and they will be rebuilt on the fly.
-- 
OpenGroupware developer: awill...@whitemice.org

OpenGroupare & Cyrus IMAPd documenation @



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Is user lock-out required for backend xfers?

[Matt Selsky]

Quoting Wesley Craig :

> I'm pretty sure this is because neither the frontends nor the clients
> are able to intelligently handle the kinds of transient errors that
> are returned.  Frontend improvements might be possible.  Client
> improvements are Sisyphean.

What could the frontends do that would be better?


-- 
Matt

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: ctl_cyrusdb questions, looking for answers

[Bron Gondwana]

On Tue, Mar 17, 2009 at 09:37:37AM -0400, Adam Tauno Williams wrote:
> > > Not exactly sure what you mean here. I think you only need the database in
> > > db.backup? and of course clean the environment before restoring such a db.
> > AFAIK only that three databases are copied to the db.backupX directory. So 
> > if 
> > there are other checkpointed dbs they stay in the original place and I need 
> > to 
> > backup them?
> 
> As I understand it: there is no reason to backup deliver.db,
> statuscache.db, or tls_sessions.db.  You can dump those without
> consequence and they will be rebuilt on the fly.

Yes - we delete these as part of our startup scripts (prefer not to
have any BDB files lying around after a restart)

Hey "checkpoint" is a massively overloaded term.  In skiplist,
a "checkpoint" is when the file gets rewritten in order.  It's
a complete rewrite of the file.  Otherwise, skiplist is append
only, so deleted or replaced records take up space.

The algorithm for checkpoints is currently:

tid->logend > (2 * db->logstart + SKIPLIST_MINREWRITE)

Which basically means checkpoint when the file is over half
garbage.  I think that's actually a little too rare, especially
with SKIPLIST_MINREWRITE = 16834, it means .seen files have a
lot of junk in them!

The "checkpoint" that ctl_cyrusdb does on the other hand, is
totally different.  It's just copying those files into the
backup directories.  It doesn't actually run a db checkpoint,
at least not on skiplist.  There's no way to trigger one
externally.

I have patches in the FastMail set that tune SKIPLIST_MINREWRITE
lower, and the amount of junk required lower.  They also do a
checkpoint every time we run recovery (recovery is basically
going through and recreating all the links between records, 
rolling back any uncompleted transaction as well).

I think the "checkpoint on recovery" code will go into 2.3.15 -
it makes sense to rewrite the file when it's hot.

Bron.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Delayed delete, restoring deleted mailboxes

[Bron Gondwana]

On Tue, Mar 17, 2009 at 03:06:15PM +0200, Leena Heino wrote:
> On Tue, 17 Mar 2009, Adam Tauno Williams wrote:
> 
> > On Tue, 2009-03-17 at 13:26 +0200, Leena Heino wrote:
> >> What is the recommended process to restore deleted messages or mailboxes
> >> when delayed delete is in use?
> >
> > Use the unexpunge command;  the man page isn't bad, and I have examples
> > and some notes in the Cyrus chapter of WMOGAG
> > .
> 
> The unexpunge command was the first thing I tried and I read it's man 
> page. But what puzzles me is that it seems that the deleted mailboxes 
> are not shown in unexpunge commands output.
> 
> The mailbox meta files and mailbox's messages are shown in the spool dir 
> under the DELETED prefix eg. in the directory:
> /cyrus_spool/u/DELETED/user/foo/sent-mail/HEXSTRING/

(that HEXSTRING is a hexified time_t from when it was deleted by the
way)

> What is the correct procedure the restore this mailbox so that user is 
> able see the messages and access it once again.

Yeah, this is a missing feature, for sure.  We use custom in-house stuff
to restore backups, which can also fetch files out of the DELETED
namespace - but the cyrus tools need to be extended to support it too.

Bron.


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Is user lock-out required for backend xfers?

[Wesley Craig]

On 17 Mar 2009, at 11:26, Matt Selsky wrote:
> Quoting Wesley Craig :
>> I'm pretty sure this is because neither the frontends nor the clients
>> are able to intelligently handle the kinds of transient errors that
>> are returned.  Frontend improvements might be possible.  Client
>> improvements are Sisyphean.
>
> What could the frontends do that would be better?

Lie?  Try again?  Not having any reason to look closely at the  
problem lately, I can't say for sure what the proper solution might  
be.  But the frontend *can* tell what's happening, i.e., there are  
some distinct states that the xfer'ing mailbox progresses through.   
For each of those cases, the frontend can probably do something  
smart, e.g., retry, make the MUA wait until the mailbox is again in  
an accessible state, something else that might become apparent  
looking at the code.

:wes

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html