I wrote about this before in https://cyrus.topicbox.com/groups/devel/Tf0643c8d6e381e30-M6197e56c655115223444ce89/proposal-update-replication-protocol-with-a-renametree-command-atomic-deep-renames
*Atomic filesystem steps** * But I hadn't finished entirely thinking through how to know where we got to during the file moves. In particular, I said this: > - if both paths exist - may need smarts to merge / repair them (disk full > part way through?) - for path under source, move / replace into destination > since source is likely to be more correct (it's untouched except for the > header updates once we start the rename) There's actually a better way :) It's this: 1) copy/hardlink every file EXCEPT cyrus.header 2) write a new cyrus.header.NEW 3) rename cyrus.header.NEW to cyrus.header 4) clean up the source Which means there is an atomic switchover point: a) if cyrus.header does NOT exist, then guaranteed cleanup hasn't started, so copy/link everything from the source again. b) if cyrus.header DOES exist, then everything is in place already, so clean up the source. This is 100% totally robust, and the rename is atomic. Yay. *Atomic everything* Right now I was just proposing this for renames, but we should do it for everything. Mailbox deletion and mailbox creation should go through the same steps: 1) transactionally create a mailboxes.db entry saying "this change is coming" 2) do the change 3) transactionally update mailboxes.db to say that it's been done Which means that a create, a rename which needs to create parent mailboxes, whatever - they're all a multi-mailbox atomic multistep process - just sometimes there's only one member. Atomic delete will involve setting MBTYPE_DELETED|MBTYPE_MOVING until the files are removed, and then clearing the MOVING flag. *Partition moves** * I had a problem with partition moves, because there's only one N key, so how do you tell that it's a rename? The answer there is "insert a previous record into the I key". Likewise for everything else, everything is always in the I key. Easy/peasy. Bron. -- Bron Gondwana, CEO, Fastmail Pty Ltd br...@fastmailteam.com ------------------------------------------ Cyrus: Devel Permalink: https://cyrus.topicbox.com/groups/devel/T62d9c7ca534890fe-Mf32a1e2b58219bec66f9ffbd Delivery options: https://cyrus.topicbox.com/groups/devel/subscription
