Backup using ZFS Snapshots
Hello, I am running a Subversion 1.9.3 server on Ubuntu 16.04. I currently use svnadmin hotcopy to safely backup the SVN repositories, but since the repositories are now hosted on a ZFS dataset, I would like to utilize ZFS's snapshot capabilities to create atomic, point-in-time backups of the repositories. My plan to do this is as follows: 1. create zfs snapshot 2. clone zfs snapshot and mount at a temporary location 3. run svnadmin hotcopy from the mounted clone to safely create a backup 4. umount and destroy clone My only concern is if a commit is in-progress when the zfs snapshot occurs, would svnadmin hotcopy still be able to safely handle creating the backup? Is this a safe procedure for creating backups? Thanks, Andrew Martin
Re: Backup using ZFS Snapshots
- Original Message - > From: "Stefan Sperling" > To: "amartin" > Cc: "users" > Sent: Tuesday, December 13, 2016 3:29:50 PM > Subject: Re: Backup using ZFS Snapshots > On Tue, Dec 13, 2016 at 03:17:53PM -0600, Andrew Martin wrote: >> Hello, >> >> I am running a Subversion 1.9.3 server on Ubuntu 16.04. I currently use >> svnadmin hotcopy to safely backup the SVN repositories, but since the >> repositories are now hosted on a ZFS dataset, I would like to utilize ZFS's >> snapshot capabilities to create atomic, point-in-time backups of the >> repositories. My plan to do this is as follows: >> >> 1. create zfs snapshot >> 2. clone zfs snapshot and mount at a temporary location >> 3. run svnadmin hotcopy from the mounted clone to safely create a backup >> 4. umount and destroy clone >> >> My only concern is if a commit is in-progress when the zfs snapshot occurs, >> would svnadmin hotcopy still be able to safely handle creating the backup? >> >> Is this a safe procedure for creating backups? > > You don't need to hotcopy at all if you have filesystem snapshots. > Check out 'svnadmin freeze'. It was made for this use case. Thanks for the quick reply - 'svnadmin freeze' looks useful for this purpose. Is there a way to freeze the repository with a command and then unfreeze it with another command? I'd ideally like to run several commands on different servers while the repository is frozen.
Re: Backup using ZFS Snapshots
- Original Message - > From: "Mark Phippard" > To: "amartin" > Cc: "users" > Sent: Tuesday, December 13, 2016 3:35:37 PM > Subject: Re: Backup using ZFS Snapshots > On Tue, Dec 13, 2016 at 4:17 PM, Andrew Martin wrote: > >> Hello, >> >> I am running a Subversion 1.9.3 server on Ubuntu 16.04. I currently use >> svnadmin hotcopy to safely backup the SVN repositories, but since the >> repositories are now hosted on a ZFS dataset, I would like to utilize ZFS's >> snapshot capabilities to create atomic, point-in-time backups of the >> repositories. My plan to do this is as follows: >> >> 1. create zfs snapshot >> 2. clone zfs snapshot and mount at a temporary location >> 3. run svnadmin hotcopy from the mounted clone to safely create a backup >> 4. umount and destroy clone >> >> My only concern is if a commit is in-progress when the zfs snapshot occurs, >> would svnadmin hotcopy still be able to safely handle creating the backup? >> > > svnadmin hotcopy will not have a problem with an in-process commit. That > is kind of one of the points of that command. > > > >> Is this a safe procedure for creating backups? >> > > I know very little about ZFS but I do not understand why you would do > this. If it can take snapshots, then why wouldn't you just take a snapshot > of the actual live repository, why would you want to copy it? > > Setting that aside, I think hotcopy is a very good way to do backups ... > but it is not clear what value would come from taking a snapshot of the > backup given that version control history is immutable etc. Also, once you > create the initial hotcopy, you can use the hotcopy --incremental option to > just copy the newer revisions. Do that from a post-commit hook and you can > always have a backup up to the latest commit. > > To me snapshots would seem like a way to do backups without using hotcopy. > I am not sure the value of combining the two ... but it should work > technically if you have "reasons". > > Mark The reason I want to do it this way is that I want to "pull" the backups to the backup server rather than "pushing" them from the Subversion server. In order to do that, I plan on mounting the clone over NFS on the backup server and then using "svnadmin hotcopy" to pull any new data onto the backup server. Yes, I could just mount the live repository, but I have a number of repositories to backup and using the ZFS snapshot ensures that they are all from a single point-in-time. It sounds like "svnadmin hotcopy" will just ignore any in-progress commit, so even if the ZFS snapshot happened in the middle of a commit, it would still result in a consistent backup. Thanks, Andrew
Re: Backup using ZFS Snapshots
- Original Message - > From: "Mark Phippard" > To: "amartin" > Cc: "users" > Sent: Tuesday, December 13, 2016 3:57:04 PM > Subject: Re: Backup using ZFS Snapshots > On Tue, Dec 13, 2016 at 4:47 PM, Andrew Martin wrote: > >> >> >> - Original Message - >> > From: "Mark Phippard" >> > To: "amartin" >> > Cc: "users" >> > Sent: Tuesday, December 13, 2016 3:35:37 PM >> > Subject: Re: Backup using ZFS Snapshots >> >> > On Tue, Dec 13, 2016 at 4:17 PM, Andrew Martin >> wrote: >> > >> >> Hello, >> >> >> >> I am running a Subversion 1.9.3 server on Ubuntu 16.04. I currently use >> >> svnadmin hotcopy to safely backup the SVN repositories, but since the >> >> repositories are now hosted on a ZFS dataset, I would like to utilize >> ZFS's >> >> snapshot capabilities to create atomic, point-in-time backups of the >> >> repositories. My plan to do this is as follows: >> >> >> >> 1. create zfs snapshot >> >> 2. clone zfs snapshot and mount at a temporary location >> >> 3. run svnadmin hotcopy from the mounted clone to safely create a backup >> >> 4. umount and destroy clone >> >> >> >> My only concern is if a commit is in-progress when the zfs snapshot >> occurs, >> >> would svnadmin hotcopy still be able to safely handle creating the >> backup? >> >> >> > >> > svnadmin hotcopy will not have a problem with an in-process commit. That >> > is kind of one of the points of that command. >> > >> > >> > >> >> Is this a safe procedure for creating backups? >> >> >> > >> > I know very little about ZFS but I do not understand why you would do >> > this. If it can take snapshots, then why wouldn't you just take a >> snapshot >> > of the actual live repository, why would you want to copy it? >> > >> > Setting that aside, I think hotcopy is a very good way to do backups ... >> > but it is not clear what value would come from taking a snapshot of the >> > backup given that version control history is immutable etc. Also, once >> you >> > create the initial hotcopy, you can use the hotcopy --incremental option >> to >> > just copy the newer revisions. Do that from a post-commit hook and you >> can >> > always have a backup up to the latest commit. >> > >> > To me snapshots would seem like a way to do backups without using >> hotcopy. >> > I am not sure the value of combining the two ... but it should work >> > technically if you have "reasons". >> > >> > Mark >> >> The reason I want to do it this way is that I want to "pull" the backups to >> the backup server rather than "pushing" them from the Subversion server. In >> order to do that, I plan on mounting the clone over NFS on the backup >> server >> and then using "svnadmin hotcopy" to pull any new data onto the backup >> server. >> Yes, I could just mount the live repository, but I have a number of >> repositories >> to backup and using the ZFS snapshot ensures that they are all from a >> single >> point-in-time. >> >> It sounds like "svnadmin hotcopy" will just ignore any in-progress commit, >> so >> even if the ZFS snapshot happened in the middle of a commit, it would still >> result in a consistent backup. >> >> > Yes, hotcopy will leave you with a consistent repository at the state it > was in when the backup started. Any commits that are in process will not > be copied/included in the backup. The problem with hotcopy, is that unless > you craft a method where you can use --incremental, you are copying the > entire repository every time you do a backup even though only a small > number of new files may have been created. The SVN repository format is > write-once. Only new files get added as new revisions get created. So the > repository can be backed up very efficiently if you just keep capturing the > new revisions being created. > Yes I plan on using --incremental to make the backup very quick. Basically the steps will be: 1. make snapshot on fileserver 2. clone snapshot to read-only clone, accessible by backup server over NFS 3. mount clone on backup server 4. run svnadmin hotcopy --incremental on clone 5. unmount and delete clone Since I'll be pointing "svnadmin hotcopy" at the same target directory on the backup server each time this is run, I should be able to to take advantage of --incremental to make this fast. > If all of your activity happens via Apache there might be easier ways to make > all your repositories read only during a backup window and you can also always > use the start-commit hook as an easy way to make repositories read only. It's tempting to just stop apache during the backup, but I need to continue to provide read-only access during the backup window, so apache needs to stay on. Yes I could update the authz file and change all entries from rw to r, but that seems trickier given that I have a large authz file.
Re: Backup using ZFS Snapshots
- Original Message - > From: "Mark Phippard" > To: "amartin" > Cc: "users" > Sent: Thursday, December 15, 2016 8:40:43 AM > Subject: Re: Backup using ZFS Snapshots > On Thu, Dec 15, 2016 at 9:25 AM, Andrew Martin wrote: > >> > If all of your activity happens via Apache there might be easier ways to >> make >> > all your repositories read only during a backup window and you can also >> always >> > use the start-commit hook as an easy way to make repositories read only. >> >> It's tempting to just stop apache during the backup, but I need to >> continue to >> provide read-only access during the backup window, so apache needs to stay >> on. >> > > I was thinking of two approaches: > > 1. Have a "read-only" Apache httpd.conf that you swap into place and do a > graceful restart and then swap back at end. This configuration would use > some variant of these directives I took from the svnbook: > > # Authorization: Authenticated users only for non-read-only > #(write) operations; allow anonymous reads > >Require valid-user > > > You would use this concept, not this exact configuration. You would want > basically be configuring the server to only allow read options. > > > 2. The easier approach is the start-commit hook. Just have a single master > hook-script that all repositories are symlinked to. Assuming you do not > use this hook for anything else, you can just have it in place with content > like: > > exit 0; > > Then when you are doing a backup you change it to something like: > > echo "Server is in read-only mode for backup. It should be available again > within N minutes" > exit 1; > > That said, based on the approach you are taking, I do not think you need to > do any of this. Your ZFS snapshot of the filesystem can happen while a > commit is happening since the ultimate backup will be done with an svnadmin > hotcopy and that command will not care if the snapshot grabbed an > in-progress commit. > Thanks for the clarification - those both seem like good approaches as well, but like you said the ZFS snapshot + "svnadmin hotcopy" should be sufficient for my particular use case. Thanks again for the help! Andrew
Use authz to allow user access to only a specific subdirectory
Hello, I am running Subversion 1.7.7 on Ubuntu 12.04 using apache2 + dav_svn + mod-auth-ntlm-winbind for authentication. In my apache config, I specify an authz file for access control: AuthzSVNAccessFile /path/to/authz In the repository, I have several different sub-directories: /trunk/a /trunk/b /trunk/c I would like to grant user1 read access to only /trunk/a, but not access to /trunk/b or /trunk/c (or other future sibling directories that are created). The examples in the SVN Book show how to grant access if the user already has access to the parent directory, but not how to grant access to only a specific sub-directory. Using the following authz syntax doesn't work (user1 is denied access to /trunk/a): [Repo:/] * = user2 = r [Repo:/trunk/a] user1 = r How can I grant access only to a specific sub-directory? Thanks, Andrew Martin
