Re: Public calendars and addressbooks (was RE: Backup compaction optimization in a block-level replication environment)

2019-11-21 Thread ellie timoney
On Wed, Nov 20, 2019, at 4:41 PM, Deborah Pickett wrote: > > I'm curious how these are working for you, or what sort of configuration > > and workflows leads to having #calendars and #addressbooks as top-level > > shared mailboxes?  I've only very recently started learning how our DAV bits > > work

Public calendars and addressbooks (was RE: Backup compaction optimization in a block-level replication environment)

2019-11-19 Thread Deborah Pickett
> I'm curious how these are working for you, or what sort of configuration and workflows leads to having #calendars and #addressbooks as top-level shared mailboxes?  I've only very recently started learning how our DAV bits work (they have previously been black-boxes for me), and so far have only s

Re: Backup compaction optimization in a block-level replication environment

2019-11-19 Thread ellie timoney
On Wed, Nov 20, 2019, at 11:06 AM, Deborah Pickett wrote: > On 2019-11-20 10:03, ellie timoney wrote: >>> foo also includes "#calendars" and "#addressbooks" on my server so there are weird characters to deal with. >>> >> Now that's an interesting detail to consider. >> > I should restate my ori

Re: Backup compaction optimization in a block-level replication environment

2019-11-19 Thread Deborah Pickett
On 2019-11-20 10:03, ellie timoney wrote: foo also includes "#calendars" and "#addressbooks" on my server so there are weird characters to deal with. Now that's an interesting detail to consider. I should restate my original message because I'm being fast and loose with the meaning of "contain

Re: Backup compaction optimization in a block-level replication environment

2019-11-19 Thread ellie timoney
On Tue, Nov 19, 2019, at 9:38 AM, Deborah Pickett wrote: > > Food for thought. Maybe instead of having one "%SHARED" backup, having one > > "%SHARED.foo" backup per top-level shared folder would be a better > > implementation? I haven't seen shared folders used much in practice, so > > it's in

Re: Backup compaction optimization in a block-level replication environment

2019-11-18 Thread Deborah Pickett
Food for thought. Maybe instead of having one "%SHARED" backup, having one "%SHARED.foo" backup per top-level shared folder would be a better implementation? I haven't seen shared folders used much in practice, so it's interesting to hear about it. Looking at your own data, if you had one "%S

Re: Backup compaction optimization in a block-level replication environment

2019-11-17 Thread ellie timoney
> Related: I had to apply the patch described in > (https://www.mail-archive.com/info-cyrus@lists.andrew.cmu.edu/msg47320.html), > "backupd IOERROR reading backup files larger than 2GB", because during > initial population of my backup, chunks tended to by multiple GB in size > (my %SHARED user ba

Re: Backup compaction optimization in a block-level replication environment

2019-11-15 Thread Deborah Pickett
Further progress report: with small chunks, compaction takes about 15 times longer.  It's almost as if there is an O(n^2) complexity somewhere, looking at the rate that the disk file grows.  (Running perf on a compaction suggests that 90% of the time ctl_backups is doing compression, decompress

Re: Backup compaction optimization in a block-level replication environment

2019-11-14 Thread Deborah Pickett
On 2019-11-11 11:10, ellie timoney wrote: This setting might be helpful: Thanks, I saw that setting but didn't really think through how it would help me.  I'll experiment with it and report back. That would be great, thanks! Progress report: I started with very large chunks (minimum 64 MB, m

Re: Backup compaction optimization in a block-level replication environment

2019-11-10 Thread ellie timoney
On Fri, Nov 8, 2019, at 1:35 PM, Deborah Pickett wrote: > I didn't know if copying > the filesystem of a (paused) Cyrus replica was a supported way of > backing up, but now I do. Yeah, as long as there are no cyrus processes running, the database/index files can just be copied about and won't b

Re: Backup compaction optimization in a block-level replication environment

2019-11-07 Thread Deborah Pickett
On 2019-11-08 09:13, ellie timoney wrote: I'm not sure if I'm just not understanding, but if the chunk offsets were to remain the same, then there's no benefit to compaction? A (say) 2gb file full of zeroes between small chunks is still the same 2gb on disk as one that's never been compacted a

Re: Backup compaction optimization in a block-level replication environment

2019-11-07 Thread ellie timoney
I'm not sure if I'm just not understanding, but if the chunk offsets were to remain the same, then there's no benefit to compaction? A (say) 2gb file full of zeroes between small chunks is still the same 2gb on disk as one that's never been compacted at all! And if you don't use the compaction

Backup compaction optimization in a block-level replication environment

2019-11-06 Thread Deborah Pickett
(Sorry, that's a lot of big words.  I'll try explaining what I want to do.) On my LAN I have a Cyrus IMAP server (3.0.11), and a dedicated Cyrus backup server (patched with Ellie's shared-mailbox and 64-bit fseek fixes).  These are connected by a nice fat link so backups happen fast and often.