On 02/20/15 10:21, Philip Martin wrote:
You are making things very hard for yourself by running 1.6.11, it is
very old and no longer maintained, unless your distribution is
backporting fixes. A newer version will have fewer bugs and better
performance.
You'll get no disagreement from me on that. At my last shop we made
sure to upgrade to the next major version any time the one we were
running went out of maintenance.
1.6 has a hotcopy bug
http://subversion.tigris.org/issues/show_bug.cgi?id=3596
the effects are hard to predict but one possibility is corrupt revisions
when committing to the hotcopy. You could delete the file
db/rep-cache.db from the hotcopy and see if this fixes the problem. In
1.8 the hotcopy bug is fixed, 1.8 also attempts to detect and avoid
problems caused by a corrupt rep-cache.db.
This was indeed the issue. We have a cron job that runs every five
minutes that wries a timestamp into a file and checks it in in order to
exercise our CI server. This is pretty much guaranteed to fire at least
once during the hotcopy, ensuring that the rep cache will be
inconsistent with the remainder of the hotcopy. Truncating the hotcopy
cache on the slave before starting the server allowed the svnsync to
proceed as expected.
Thank you very, very much for your help,
Tony.
Tony Sweeney <swee...@addr.com> writes:
Hi,
Mise en scene:
We have three Subversion servers, version 1.6.11 running on
CentOS6. One is the live master and and the others are hot standbys in
different datacenters. The intent is that these are kept in sync with
the master using svnsync from a post-commit hook. One of the slaves
is already set up this way and has been since it was deployed by one
of my predecessors at this company. This is syncing nicely. The
other slave instance was also originally set up this way but was then
modified to instead use a block level filesystem mirroring technology,
again by one
of my predecessors. We have since concluded that the
mirroringsolution was not sufficiently trustworthy, and so we would
like to put it back to the svnsync method that it had run in the past.
The live master has in excess of 366,000 revisions and new revisions
come in at ~500 commits/day.
One solution would be to simply create an empty repo, set the
necessary r0 revprops and hook scripts and sync from scratch.
However, we estimate that this would take 10-15 days due to the size
of the repository, so we would like to short-circuit this by starting
from a hotcopy. We take hotcopies nightly, save them to a backup
server and 'svnadmin verify' them there. I am trying to bring this
slave back on sync using the method described here:
http://www.cardinalpath.com/how-to-use-svnsync-to-create-a-mirror-backup-of-your-subversion-repository/
But using a hotcopy rather than a dump.
The master hotcopy is restored in the correct location on the slave
and the hook scripts fixed up using Puppet (master and slave
necessarily have different hook script setups). Permissions are fixed
up to reflect that the hotcopy is taken as root but the server is
running under Apache. I set the svn:sync-last-merged-rev revprop on
r0 to reflect the newest revision on the hotcopy. At this point (or
so I thought) I should be able to replicate over the changes since
last night by running svnsync manually and then enable it in the hook
script.
The problem is that that the revisions seem to be corrupted on
transfer to this slave.
From the master:
[root@uk-svn-1-new yoyodyne]# /usr/bin/svnsync --non-interactive --trust-server-
cert sync https://uk-svn-2-new.uk.yoyodyne.lan/svn/yoyodyne
--sync-username svnsync --sync-password XXXXXXXX --source-username
svnsync --source-password XXXXXXXX --config-dir
/repos/svn/yoyodyne/hooks/certs
Transmitting file data ..
Committed revision 367219.
Copied properties for revision 367219.
Transmitting file data .
Committed revision 367220.
Copied properties for revision 367220.
Transmitting file data .
Committed revision 367221.
Copied properties for revision 367221.
Transmitting file data .svnsync: Corrupt representation '367220 0 41 29949930e5
5f203ff7027917e765e6ca7d 2654a46e5fff1f98b747a77ebf600c10c895fbcc367219-7wnu/_6'
You have new mail in /var/spool/mail/root
[root@uk-svn-1-new yoyodyne]#
In fact, checking on the slave, they have all come across corrupt:
[root@uk-svn-2-new svn]# svnadmin verify -r 367200:HEAD /repos/svn/yoyodyne
* Verified revision 367200.
* Verified revision 367201.
* Verified revision 367202.
* Verified revision 367203.
* Verified revision 367204.
* Verified revision 367205.
* Verified revision 367206.
* Verified revision 367207.
* Verified revision 367208.
* Verified revision 367209.
* Verified revision 367210.
* Verified revision 367211.
* Verified revision 367212.
* Verified revision 367213.
* Verified revision 367214.
* Verified revision 367215.
* Verified revision 367216.
* Verified revision 367217.
* Verified revision 367218.
svnadmin: Corrupt representation '367219 0 50
633695f7c627c997b8b4065db2d4cb4db53c
1f15b1d8cea246d731676f43b2df4c74a1710793 367218-7wnt/_d'
svnadmin: Malformed representation header
You have mail in /var/spool/mail/root
[root@uk-svn-2-new svn]#
Both the master and working slave verify all revisions clean at all
times. I've tried multiple hotcopies and every time the first
revision that it tries to sync comes across as corrupt.
On the master the revision file looks as follows:
[root@uk-svn-1-new yoyodyne]# head -30 db/revs/367/367219
DELTA 366873 14907 1619
SVN??@
?=??h?&ng serialVersionUID = 1;
ENDREP
DELTA 366873 16539 1063
SVN?J?7?V?)|???"?Y?? ??private boolean hasMoreResults = trueif(results.size() <
requiredResults){
hasMoreResults = false;
}
hasMoreResults;
}
}
ENDREP
id: 1j-366873.0-228442.r367219/295
type: file
pred: 1j-366873.0-228442.r366873/30632
count: 1
text: 367219 81 183 2999 2880cca7c6c56dea580d5c212595846c dc0cc36f718b468fc08817
88900f096d5cd4913f 367218-7wnq/_e
props: 366873 30580 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
cpath:
/MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/DBServiceQueryThread.java
copyroot: 228442 /MIME/MADb
id: 1h-366873.0-228442.r367219/692
type: file
pred: 1h-366873.0-228442.r366873/32681
count: 1
text: 367219 0 50 6336 95f7c627c997b8b4065db2d4cb4db53c 1f15b1d8cea246d731676f43
b2df4c74a1710793 367218-7wnq/_d
props: 366873 32629 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
cpath:
/MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/CachedRowList.java
copyroot: 228442 /MIME/MADb
[root@uk-svn-1-new yoyodyne]#
The working slave looks as follows:
[root@us-svn-1 yoyodyne]# head -30 db/revs/367/367219
DELTA 366873 14907 1619
SVN??@
?=??h?&ng serialVersionUID = 1;
ENDREP
DELTA 366873 16539 1063
SVN?J?7?V?)|???"?Y?? ??private boolean hasMoreResults = trueif(results.size() <
requiredResults){
hasMoreResults = false;
}
hasMoreResults;
}
}
ENDREP
id: 1j-366873.0-228442.r367219/295
type: file
pred: 1j-366873.0-228442.r366873/30631
count: 1
text: 367219 81 183 2999 2880cca7c6c56dea580d5c212595846c dc0cc36f718b468fc08817
88900f096d5cd4913f 367218-7yyk/_e
props: 366873 30579 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
cpath:
/MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/DBServiceQueryThread.java
copyroot: 228442 /MIME/MADb
id: 1h-366873.0-228442.r367219/692
type: file
pred: 1h-366873.0-228442.r366873/32680
count: 1
text: 367219 0 50 6336 95f7c627c997b8b4065db2d4cb4db53c 1f15b1d8cea246d731676f43
b2df4c74a1710793 367218-7yyk/_d
props: 366873 32628 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
cpath: /MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/Cach
edRowList.java
copyroot: 228442 /MIME/MADb
[root@us-svn-1 yoyodyne]#
But on the failing slave:
[root@uk-svn-2-new yoyodyne]# head -30 db/revs/367/367219
id: 1j-366873.0-228442.r367219/0
type: file
pred: 1j-366873.0-228442.r366873/30632
count: 1
text: 367219 81 183 2999 2880cca7c6c56dea580d5c212595846c dc0cc36f718b468fc08817
88900f096d5cd4913f 367218-7wnt/_e
props: 366873 30580 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
cpath:
/MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/DBServiceQueryThread.java
copyroot: 228442 /MIME/MADb
id: 1h-366873.0-228442.r367219/395
type: file
pred: 1h-366873.0-228442.r366873/32681
count: 1
text: 367219 0 50 6336 95f7c627c997b8b4065db2d4cb4db53c 1f15b1d8cea246d731676f43
b2df4c74a1710793 367218-7wnt/_d
props: 366873 32629 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
cpath:
/MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/CachedRowList.java
copyroot: 228442 /MIME/MADb
PLAIN
K 19
CSVQueryThread.java
V 37
file 1e-366873.0-228442.r366873/31862
K 18
CachedRowList.java
V 35
file 1h-366873.0-228442.r367219/395
K 25
DBServiceQueryThread.java
V 33
[root@uk-svn-2-new yoyodyne]#
Each time I've tries this the failure mode appears to be the same.
The initial 'DELTA <E2><80><A6> ENDREP' sections are missing from the
copied over file, it starts with the 'id:' line directory. Is there
anything I can do to debug this further?
Tony.
