fsverify.py unable to fix invalid svndiff header
Platform: debian 6.0.1 "squeeze", subversion 1.6.16dfsg-1+b1, fsfsverify.py from http://svn.apache.org/repos/asf/subversion/trunk/contrib/server-side python 2.6.6-3+squeeze6 (information about the repository at the end of this mesage) Yesterday I discovered that my subversion repository is broken. Trying to do "svnadmin dump" breaks at revision 683: sb@somehost:/tmp/svnrepo$ svnadmin dump /tmp/svnrepo/svn/ >/dev/null * Dumped revision 0. * Dumped revision 1. ... * Dumped revision 681. * Dumped revision 682. svnadmin: Decompression of svndiff data failed I found some threads on this list of people with the same error message suggesting to use fsfsverify.py. I assumed that the trunk version from the repository would be the newest and most stable so that's the version I've tried. I've done all of my experiments with fsfsverify.py, and one attempt at using fsfsfixer, on a copy of the repository, created with the command: (cd /tmp/svnrepo; rm -rf svn; cp -a /usr/local/svn/ .) The first run of fsfverify.py /tmp/svnrepo/svn/db/revs/0/683 reports the following: ... NodeRev Id: 4-683.0.r683/12232 type: file text: DELTA 683 1910 890 1548 30aadd70339eb7b41ab6eb20a6220202 prop: PLAIN 683 12138 81 0 685faf6a4230ea531a31a12e4b3cb565 cpath: /trunk/someprog/metamodeller/Makefile.msvc copyroot: 0 / Error InvalidCompressedStream: Invalid compressed data stream at offset 2247 (Error -3 while decompressing: incorrect data check, ) Try running with -f to fix the revision When I run fsfsverify.py -f /tmp/svnrepo/svn/db/revs/0/683 I see: ... NodeRev Id: 4-683.0.r683/12232 type: file text: DELTA 683 1910 890 1548 30aadd70339eb7b41ab6eb20a6220202 prop: PLAIN 683 12138 81 0 685faf6a4230ea531a31a12e4b3cb565 cpath: /trunk/someprog/metamodeller/Makefile.msvc copyroot: 0 / Copy 893 bytes from offset 1920 Write 893 bytes at offset 1916 Fixed? :-) Re-run fsfsverify without the -f option Which looks good. However, fsfverify.py /tmp/svnrepo/svn/db/revs/0/683 reports: ... NodeRev Id: 4-683.0.r683/12232 type: file text: DELTA 683 1910 890 1548 30aadd70339eb7b41ab6eb20a6220202 prop: PLAIN 683 12138 81 0 685faf6a4230ea531a31a12e4b3cb565 cpath: /trunk/someprog/metamodeller/Makefile.msvc copyroot: 0 / Traceback (most recent call last): File "/tmp/scripts/fsfsverify.py", line 1190, in options.dumpWindows) File "/tmp/scripts/fsfsverify.py", line 581, in verify svndiff = Svndiff(f, self.length) File "/tmp/scripts/fsfsverify.py", line 472, in __init__ (self.startingOffset) __main__.InvalidSvndiffHeader: Invalid svndiff header at offset 1916 This is also the same error message I get on all subsequent runs of fsfverify -f. I have tried 15 runs (all with the same output), since the author of fsfsverify.py, said in an article on this list, that he once had to run it 14 times. All assistance on this issue will be welcome. Thanks! - Steinar Some information on the repository: - The size of the repository is 22MB - The repository has 1794 revisions - The repository was converted from CVS to a BDB svn repository, with cvs2svn in May 2007 - In february I discovered that the repository was BDB (I had assumed it was FSFS) because of a BDB upgrade problem nntp://news.gmane.org/gmane.comp.version-control.subversion.user/86556 http://thread.gmane.org/gmane.comp.version-control.subversion.user/86556 - According to the articles in the above thread, I was able to dump the repository, and create a new FSFS repository in February 2009 - I have used svnmerge to track merges between different branches - The server installation of svn has been whatever has been on debian stable for the repository's entire lifetime - The clients have been a multitude of different svn clients, with different versions on different platforms - I have also used many different versions of svnmerge, with many different versions of python - svnmerge has not been used on the parts of the repository that has problems - Revision 683 is from December 2005, ie. well before the transition from CVS to subversion (has been through the transition from CVS to svn, and the 2009 transition from BDB to FSFS) - The svn checkouts outside of the problem area seems to work just fine - The area that has the problem isn't important to me (I can't even remember what it was about), but history both before and after, outside of the problem area, is (it's the history of my home directory config files, ie. .bashrc, .emacs and the like)
Re: fsverify.py unable to fix invalid svndiff header
On Sat, May 14, 2011 at 10:58:12AM +0200, Steinar Bang wrote: > Platform: > debian 6.0.1 "squeeze", > subversion 1.6.16dfsg-1+b1, > fsfsverify.py from > http://svn.apache.org/repos/asf/subversion/trunk/contrib/server-side > python 2.6.6-3+squeeze6 > > (information about the repository at the end of this mesage) > > Yesterday I discovered that my subversion repository is broken. Trying > to do "svnadmin dump" breaks at revision 683: > sb@somehost:/tmp/svnrepo$ svnadmin dump /tmp/svnrepo/svn/ >/dev/null > * Dumped revision 0. > * Dumped revision 1. > ... > * Dumped revision 681. > * Dumped revision 682. > svnadmin: Decompression of svndiff data failed > > > I found some threads on this list of people with the same error message > suggesting to use fsfsverify.py. I assumed that the trunk version from > the repository would be the newest and most stable so that's the version > I've tried. > > I've done all of my experiments with fsfsverify.py, and one attempt at > using fsfsfixer, on a copy of the repository, created with the command: > (cd /tmp/svnrepo; rm -rf svn; cp -a /usr/local/svn/ .) > > > The first run of fsfverify.py /tmp/svnrepo/svn/db/revs/0/683 reports the > following: > ... > NodeRev Id: 4-683.0.r683/12232 > type: file > text: DELTA 683 1910 890 1548 30aadd70339eb7b41ab6eb20a6220202 > prop: PLAIN 683 12138 81 0 685faf6a4230ea531a31a12e4b3cb565 > cpath: /trunk/someprog/metamodeller/Makefile.msvc > copyroot: 0 / > > > Error InvalidCompressedStream: Invalid compressed data stream at offset 2247 > (Error -3 while decompressing: incorrect data check, sl:0 tl:1548 cil: 317 cdl: 561 il:315 dl:729 whl:8 wl:886>) > > Try running with -f to fix the revision > > > When I run fsfsverify.py -f /tmp/svnrepo/svn/db/revs/0/683 I see: > ... > NodeRev Id: 4-683.0.r683/12232 > type: file > text: DELTA 683 1910 890 1548 30aadd70339eb7b41ab6eb20a6220202 > prop: PLAIN 683 12138 81 0 685faf6a4230ea531a31a12e4b3cb565 > cpath: /trunk/someprog/metamodeller/Makefile.msvc > copyroot: 0 / > > Copy 893 bytes from offset 1920 > Write 893 bytes at offset 1916 > Fixed? :-) Re-run fsfsverify without the -f option > > > Which looks good. However, fsfverify.py /tmp/svnrepo/svn/db/revs/0/683 > reports: The script probably took a wrong guess. Hopefully this is the known corruption problem with a duplicate block of data in the revision file. Can you check if the original revision file (i.e. not modified by fsfsverify.py) somewhere contains a data block which contains data that matches the data around byte offset 1916? Usually the spot where the corruption appears (offset 1916 in your case) contains an incomplete representation, but the representation data in the duplicated block is good. One of way of locating the duplicate block is to open the file in a hex editor and search the entire file for hex strings that occur around or after 1916. Try to locate boundaries of representations, which look as follows: https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_fs_fs/structure A representation begins with a line containing either "PLAIN\n" or "DELTA\n" or "DELTA \n", where , , and give the location of the delta base of the representation and the amount of data it contains (not counting the header or trailer). If no base location is given for a delta, the base is the empty stream. After the initial line comes raw svndiff data, followed by a cosmetic trailer "ENDREP\n". So if you find a duplicated block of data you should be able to fix this problem by copying representation data from the duplicate block to the corrupted location. DO NOT change any byte offsets in the file while doing this. If you cannot squeeze the data in because it would overlap with subsequent data you're out of luck but I've never seen this happen. Usually there is enough room to fit the data, but you might have to add padding. Any dummy byte will do, I usually use 0x42. Another possibility is that offset 2247 is wrong. In this case the expected svndiff data is probably located elsewhere and the offset in the representation header should be adjusted. This is of course not an easy task and it is unfortunate that people keep running into this problem. The source of the problem is not yet known :( If you have any further questions just ask. If you cannot get it fixed at all but can share the revision file privately I will have a go at it. Good luck!
Vulnerability in APR: CVE-2011-0419
To interested persons: Apache Subversion uses the Apache Portable Runtime (APR) to provide platform-specific and other utility services. APR announced the availability of APR 1.4.4, which addresses CVE-2011-0419, a potential unconstrained recursion bug in the apr_fnmatch(). An attacker could potentially exploit this issue to cause the target machine to exhaust stack memory or use excessive CPU. Prior to Subversion 1.6.16, Subversion used the compromised function on untrusted data in mod_dav_svn, exposing it to this flaw. In Subversion 1.6.16, mod_dav_svn was changed to avoid the use of apr_fnmatch(), eliminating this attack vector for Subversion. Thus, Subversion systems are only vulnerable if they are running *both* APR < 1.4.4 and Subversion < 1.6.16. It is recommended that users upgrade one or both of these components as soon as is convenient. To read more about the APR 1.4.4 release, see http://www.apache.org/dist/apr/Announcement1.x.html - The Subversion Team
.svn/entries format
We develop web interfaces and would like a release string displayed on the web page. This information must be read real time. We need to obtain information that would normally be provided by 'svn info'. The page, the apache user, does not have access to this program, but he can read the project’s .svn/entries file. We can guess at the information needed but was wondering if the format is published. Strictly an ro affair. (saw a previous nanny post) The env/server is Linux/apache. There are a hundred ways to do this, but allowing perl to read this file seems easiest. Admin will not allow apache user access to svn. Also, is there something outside the svn application, a ‘working directory only’ version of svn info, (svninfo?), that we could use to obtain local working directory “info”. This would not require the security demanded by svn proper and would save us from having to read this file.
Re: subversion 1.4.2
On Fri, May 13, 2011 at 11:32 AM, Eric Manceau wrote: > Hello, > > I am struggling with an old version of subversion : 1.4.2 > Is it safe for the repository to adopt the 1.6.16 version? It's also *wise* to upgrade, especially if your older repo is using the older and notoriously fragile Berkeley DB. Doing a dump and reload to a clean, new, working repo with the same UUID manually set should be invisibble to your clients. Do review the upgrade notes from the "Red Book" at http://svnbook.red-bean.com/. And be aware that any working copy you touch with a Subversion 1.6 client will no longer be compatible with Subversion 1.4 clients. > Moreover I want to split an old repos in 2 new repos without losing revision > of my files : > > Old Repo > |--data > |-- test > > New Repo 1 > | --data > > New Repo 2 > | --data > > Svndumpfilter with 1.4 did not help and the svn command as well so did the > tools evolve in the 1.6 version in order to achieve this goal? > > Thank you for your answer. They're better. The new repos will have to be distinct, with twho new uuid's. Why not build a new, upgraded repo first, and then split *that*?
Re: subversion 1.4.2
Guten Tag Eric Manceau, am Freitag, 13. Mai 2011 um 17:50 schrieben Sie: > As you can see we want to gather all tests of several differents repos > into a unique repo but we want to keep revision of the original repos > and it seems to be hard to achieve. > Maybe impossible? If disk space is not a problem, just copy your existing repository and delete unneeded folders during svn as a new revision. Upgrade subversion, dump your existing repository, create two new ones using svn 1.6.x and load your dump into both new repos. The new repository format will save you space anyway in most situations. Mit freundlichen Grüßen, Thorsten Schöning -- Thorsten Schöning AM-SoFT IT-Systeme - Hameln | Potsdam | Leipzig Telefon: Potsdam: 0331-743881-0 E-Mail: tschoen...@am-soft.de Web: http://www.am-soft.de AM-SoFT GmbH IT-Systeme, Konsumhof 1-5, 14482 Potsdam Amtsgericht Potsdam HRB 21278 P, Geschäftsführer: Andreas Muchow
Re: fsverify.py unable to fix invalid svndiff header
> Stefan Sperling : > The script probably took a wrong guess. > Hopefully this is the known corruption problem with a duplicate block of > data in the revision file. > Can you check if the original revision file (i.e. not modified by > fsfsverify.py) somewhere contains a data block which contains data > that matches the data around byte offset 1916? "offset 1916", is that "byte number 1916 in the 683 ref file"? Is that 1916 decimal, or hexadecimal? I'm assuming decimal for now. > Usually the spot where the corruption appears (offset 1916 in your case) > contains an incomplete representation, but the representation data in the > duplicated block is good. > One of way of locating the duplicate block is to open the file in a > hex editor and search the entire file for hex strings that occur > around or after 1916. Ok, opening the file in emacs hexl mode: `M-x hexl-find-file /tmp/svnrepo/svn/db/revs/0/683 RET' > Try to locate boundaries of representations, which look as follows: > https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_fs_fs/structure > A representation begins with a line containing either "PLAIN\n" or > "DELTA\n" or "DELTA \n", where , , > and give the location of the delta base of the representation > and the amount of data it contains (not counting the header or > trailer). If no base location is given for a delta, the base is the > empty stream. After the initial line comes raw svndiff data, followed > by a cosmetic trailer "ENDREP\n". `M-j 1916 RET', takes me here: 0770: 4e44 5245 500a 4445 4c54 410a 5356 4e01 NDREP.DELTA.SVN. 0780: 8c0c 823d 8431 823b 8140 4c00 8844 .=.1.;.@L..D 0790: 4ca8 404e 0048 8100 bf45 820d 9945 820d L.@N.H...E...E.. ... The cursor is positioned on the "5" starting "5356" in the first line, and on the "S" of "SVN". Does that make sense? So I should search for "ENDREP.DELTA.SVN"? The 683 revfile contains 15 istances of that string, but I have no idea of which ones are relevant or not. > So if you find a duplicated block of data you should be able to fix this > problem by copying representation data from the duplicate block to the > corrupted location. So what I'm looking for isn't exactly "ENDREP.DELTA.SVN", but what follows this text...? I tried searching for "x^.Rmo.@", but the one at the cursor is the only occurrence in the file. At least the only aligned so that search will find it. Doesn't look like hexl-mode has the possibility to search for a byte sequence. Maybe I should get myself a proper hex editor? > DO NOT change any byte offsets in the file while doing this. If you > cannot squeeze the data in because it would overlap with subsequent > data you're out of luck but I've never seen this happen. Usually > there is enough room to fit the data, but you might have to add > padding. Any dummy byte will do, I usually use 0x42. The meaning of life, the universe and everything? I thought that was 42 decimal...? :-) > Another possibility is that offset 2247 is wrong. In this case the > expected svndiff data is probably located elsewhere and the offset > in the representation header should be adjusted. Right... that's the first error that fsfsverify.py tries to fix? `M-j 2247 RET' takes the cursor over the "7" in "5878" in the first line: 08c0: 8550 8b57 8585 5978 5e1d 526d 6fda 400c .P.W..Yx^.Rmo.@. 08d0: fe1c ff0a 17f8 00d5 12fa a27d 4145 2a90 ...}AE*. 08e0: 8466 0da4 2361 6353 2414 ee0e b871 89a3 .f..#acS$q.. That wasn't as easily recognizable, as the 1916 one, though. Not as recognizable as a boundary at least. What are the things I should look for a duplicate of? The bytes following the troublesome position? And how many? > This is of course not an easy task and it is unfortunate that people > keep running into this problem. The source of the problem is not yet > known :( If you have any further questions just ask. If you cannot > get it fixed at all but can share the revision file privately I will > have a go at it. I think I need help with this one. I'll send you the revision file privately. Thanks! - Steinar
Re: fsverify.py unable to fix invalid svndiff header
On Sat, 14 May 2011 19:54 +0200, "Steinar Bang" wrote: > > Stefan Sperling : > > > The script probably took a wrong guess. > > > Hopefully this is the known corruption problem with a duplicate block of > > data in the revision file. > > > Can you check if the original revision file (i.e. not modified by > > fsfsverify.py) somewhere contains a data block which contains data > > that matches the data around byte offset 1916? > > "offset 1916", is that "byte number 1916 in the 683 ref file"? > Is that 1916 decimal, or hexadecimal? I'm assuming decimal for now. > Yes. > > Usually the spot where the corruption appears (offset 1916 in your case) > > contains an incomplete representation, but the representation data in the > > duplicated block is good. > > > One of way of locating the duplicate block is to open the file in a > > hex editor and search the entire file for hex strings that occur > > around or after 1916. > > Ok, opening the file in emacs hexl mode: > `M-x hexl-find-file /tmp/svnrepo/svn/db/revs/0/683 RET' > > > Try to locate boundaries of representations, which look as follows: > > > https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_fs_fs/structure > > A representation begins with a line containing either "PLAIN\n" or > > "DELTA\n" or "DELTA \n", where , , > > and give the location of the delta base of the representation > > and the amount of data it contains (not counting the header or > > trailer). If no base location is given for a delta, the base is the > > empty stream. After the initial line comes raw svndiff data, followed > > by a cosmetic trailer "ENDREP\n". > > `M-j 1916 RET', takes me here: > 0770: 4e44 5245 500a 4445 4c54 410a 5356 4e01 NDREP.DELTA.SVN. > 0780: 8c0c 823d 8431 823b 8140 4c00 8844 .=.1.;.@L..D > 0790: 4ca8 404e 0048 8100 bf45 820d 9945 820d L.@N.H...E...E.. > ... > > The cursor is positioned on the "5" starting "5356" in the first line, > and on the "S" of "SVN". > > Does that make sense? > > So I should search for "ENDREP.DELTA.SVN"? The 683 revfile contains 15 > istances of that string, but I have no idea of which ones are relevant > or not. > No, you should be looking for the sequence of bytes starting at offset 1916. So, the bytes are: 53564e018c0c823d (for example, 'xxd -s 1916 -l 10 -ps' will tell you that) And please don't try searching for the text representation on the right hand side of the hex view! (You've been here long enough that I assume you know that, but you said 'search for "ENDREP.DELTA.SVN"', which is incorrect in so many ways...) > > So if you find a duplicated block of data you should be able to fix this > > problem by copying representation data from the duplicate block to the > > corrupted location. > > So what I'm looking for isn't exactly "ENDREP.DELTA.SVN", but what > follows this text...? > > I tried searching for "x^.Rmo.@", but the one at the cursor is the only > occurrence in the file. At least the only aligned so that search will > find it. Doesn't look like hexl-mode has the possibility to search for > a byte sequence. Maybe I should get myself a proper hex editor? > > > DO NOT change any byte offsets in the file while doing this. If you > > cannot squeeze the data in because it would overlap with subsequent > > data you're out of luck but I've never seen this happen. Usually > > there is enough room to fit the data, but you might have to add > > padding. Any dummy byte will do, I usually use 0x42. > > The meaning of life, the universe and everything? I thought that was 42 > decimal...? :-) > > > Another possibility is that offset 2247 is wrong. In this case the > > expected svndiff data is probably located elsewhere and the offset > > in the representation header should be adjusted. > > Right... that's the first error that fsfsverify.py tries to fix? > > `M-j 2247 RET' takes the cursor over the "7" in "5878" in the first > line: > 08c0: 8550 8b57 8585 5978 5e1d 526d 6fda 400c .P.W..Yx^.Rmo.@. > 08d0: fe1c ff0a 17f8 00d5 12fa a27d 4145 2a90 ...}AE*. > 08e0: 8466 0da4 2361 6353 2414 ee0e b871 89a3 .f..#acS$q.. > > That wasn't as easily recognizable, as the 1916 one, though. Not as > recognizable as a boundary at least. > > What are the things I should look for a duplicate of? The bytes > following the troublesome position? And how many? > > > This is of course not an easy task and it is unfortunate that people > > keep running into this problem. The source of the problem is not yet > > known :( If you have any further questions just ask. If you cannot > > get it fixed at all but can share the revision file privately I will > > have a go at it. > > I think I need help with this one. I'll send you the revision file > privately. > > Thanks! > > > - Steinar > > Good luck, Daniel
Re: fsverify.py unable to fix invalid svndiff header
On Sat, 14 May 2011 21:10 +0300, "Daniel Shahaf" wrote:
>
>
> On Sat, 14 May 2011 19:54 +0200, "Steinar Bang" wrote:
> > > Stefan Sperling :
> >
> > > The script probably took a wrong guess.
> >
> > > Hopefully this is the known corruption problem with a duplicate block of
> > > data in the revision file.
> >
> > > Can you check if the original revision file (i.e. not modified by
> > > fsfsverify.py) somewhere contains a data block which contains data
> > > that matches the data around byte offset 1916?
> >
> > "offset 1916", is that "byte number 1916 in the 683 ref file"?
> > Is that 1916 decimal, or hexadecimal? I'm assuming decimal for now.
> >
>
> Yes.
>
"Yes, it's decimal". Because the node-rev indicated offset 1910 (decimal) and
fsfsverify spoke about offset 1916, which corresponds to '1910 +
strlen("DELTA\n")' (and to the 0x77C offset of the "SVN\1" in your hex dump) so
well that I'm not going to bother check that fsfsverify doesn't print offsets
in hex. :-)
Re: fsverify.py unable to fix invalid svndiff header
Steinar Bang wrote on Sat, May 14, 2011 at 20:37:42 +0200: > > Daniel Shahaf : > > > No, you should be looking for the sequence of bytes starting at offset > > 1916. So, the bytes are: > > > 53564e018c0c823d > > > (for example, 'xxd -s 1916 -l 10 -ps' will tell you that) > > Um... how do one determine the length of the sequence? You are using 10 > bytes I guess...? > No, I wasn't using "10" bytes, I was using an arbitrary number of bytes. "10" is not a magic number in this context. > Anyway, the sequence you are showing about, starting at 1916 is the only > one in the file, both when searching forward and backwards, in ghex. > > > And please don't try searching for the text representation on the > > right hand side of the hex view! (You've been here long enough that I > > assume you know that, but you said 'search for "ENDREP.DELTA.SVN"', > > which is incorrect in so many ways...) > > Yes, I was just trying my way around hexl-mode, and it proved to have > too limited search capabilities. And I was also hoping that there would > be human readable strings that could be used to identify the contents. > The thing starting with bytes 0x53564e01 is an svndiff1 stream --- a zlib-compressed binary delta. (Those four bytes are three magic bytes and one format-number byte.) It's probably the least human-readable part in Subversion... > Re: 2247 > > >> I tried searching for "x^.Rmo.@", but the one at the cursor is the only > >> occurrence in the file. At least the only aligned so that search will > >> find it. Doesn't look like hexl-mode has the possibility to search for > >> a byte sequence. Maybe I should get myself a proper hex editor? > > I installed ghex, and there I've searched for "78 5E 1D 52 6D 6F DA", > and there is only one occurrence of that strng in the file. Hmm. I'm assuming Stefan will have a look, but if for some reason he doesn't then I'll have a look at your original mail and see if I mis-advised you regarding what duplication to look for.
Re: fsverify.py unable to fix invalid svndiff header
Daniel Shahaf wrote on Sat, May 14, 2011 at 22:05:59 +0300: > an svndiff1 stream Documented in: ^/subversion/trunk/notes/svndiff
Re: fsverify.py unable to fix invalid svndiff header
> Steinar Bang : > Stefan Sperling : >> Try to locate boundaries of representations, which look as follows: >> https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_fs_fs/structure >>A representation begins with a line containing either "PLAIN\n" or >>"DELTA\n" or "DELTA \n", where , , >>and give the location of the delta base of the representation >>and the amount of data it contains (not counting the header or >>trailer). If no base location is given for a delta, the base is the >>empty stream. After the initial line comes raw svndiff data, followed >>by a cosmetic trailer "ENDREP\n". > `M-j 1916 RET', takes me here: > 0770: 4e44 5245 500a 4445 4c54 410a 5356 4e01 NDREP.DELTA.SVN. > 0780: 8c0c 823d 8431 823b 8140 4c00 8844 .=.1.;.@L..D > 0790: 4ca8 404e 0048 8100 bf45 820d 9945 820d L.@N.H...E...E.. > ... So... trying to interpret this with the description that Stefan gave, and that Daniel referred to. Here we have "DELTA\n" (the "0A" is "\n"), which begins a representation according to the above description. But this is where I get a bit lost. Is what I'm looking for what is between the "0A" and the next "ENDREP"? (and what will I do with it once I've got it? (but that's for later. let's first try to find it)). Anyway, the 10 byte sequence that Daniel gave: 53564e018c0c823d only has a single occurrence in that file, according to ghex. What now?
Re: fsverify.py unable to fix invalid svndiff header
> Daniel Shahaf : > No, you should be looking for the sequence of bytes starting at offset > 1916. So, the bytes are: > 53564e018c0c823d > (for example, 'xxd -s 1916 -l 10 -ps' will tell you that) Um... how do one determine the length of the sequence? You are using 10 bytes I guess...? Anyway, the sequence you are showing about, starting at 1916 is the only one in the file, both when searching forward and backwards, in ghex. > And please don't try searching for the text representation on the > right hand side of the hex view! (You've been here long enough that I > assume you know that, but you said 'search for "ENDREP.DELTA.SVN"', > which is incorrect in so many ways...) Yes, I was just trying my way around hexl-mode, and it proved to have too limited search capabilities. And I was also hoping that there would be human readable strings that could be used to identify the contents. Re: 2247 >> I tried searching for "x^.Rmo.@", but the one at the cursor is the only >> occurrence in the file. At least the only aligned so that search will >> find it. Doesn't look like hexl-mode has the possibility to search for >> a byte sequence. Maybe I should get myself a proper hex editor? I installed ghex, and there I've searched for "78 5E 1D 52 6D 6F DA", and there is only one occurrence of that strng in the file.
Re: fsverify.py unable to fix invalid svndiff header
> Steinar Bang : > Um... how do one determine the length of the sequence? You are using > 10 bytes I guess...? Never mind. It's from that structure description you posted. I'll study that some more.
Re: fsverify.py unable to fix invalid svndiff header
> Steinar Bang : > Steinar Bang : >> Um... how do one determine the length of the sequence? You are using >> 10 bytes I guess...? > Never mind. It's from that structure description you posted. Er sorry! Not you, but Stefan Sperling. I didn't look at the From headers of the emails.
Re: fsverify.py unable to fix invalid svndiff header
> Steinar Bang : > Steinar Bang : > Stefan Sperling : >>> Try to locate boundaries of representations, which look as follows: >>> https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_fs_fs/structure >>> A representation begins with a line containing either "PLAIN\n" or >>> "DELTA\n" or "DELTA \n", where , , >>> and give the location of the delta base of the representation >>> and the amount of data it contains (not counting the header or >>> trailer). If no base location is given for a delta, the base is the >>> empty stream. After the initial line comes raw svndiff data, followed >>> by a cosmetic trailer "ENDREP\n". >> `M-j 1916 RET', takes me here: >> 0770: 4e44 5245 500a 4445 4c54 410a 5356 4e01 NDREP.DELTA.SVN. >> 0780: 8c0c 823d 8431 823b 8140 4c00 8844 .=.1.;.@L..D >> 0790: 4ca8 404e 0048 8100 bf45 820d 9945 820d L.@N.H...E...E.. >> ... > So... trying to interpret this with the description that Stefan gave, > and that Daniel referred to. Here we have "DELTA\n" (the "0A" is "\n"), > which begins a representation according to the above description. > But this is where I get a bit lost. Just analyzing a little bit further from the /subversion/notes/svndiff description that Daniel mentioned. The first part after "DELTA\n" is "SVN" followed by a one byte format version number, in this case 0x01. And then comes the hard part: After the header come one or more windows, until the document ends. (So the decoder must have external context indicating when there is no more svndiff data.) Then comes the description of a window, and that is, as Daniel said, probably not human readable, or even recognizable in a hex editor, because the window may be zlib compressed (or so I understand it). Hm... hard this is. The dark side strong in this one.
Re: fsverify.py unable to fix invalid svndiff header
You probably don't need to know anything at all about svndiff --- if previous instances of the bug are representative, then you're free to treat consider the svndiff stream an opaque binary data. That is, previously what happened is that fsfs wrote A_1 A_2 B_1 B_2 C_1 C_1 C_2 D_1 D_2 E_1 E_2 On Sat, 14 May 2011 21:24 +0200, "Steinar Bang" wrote: > > Steinar Bang : > > > Steinar Bang : > > Stefan Sperling : > > >>> Try to locate boundaries of representations, which look as follows: > > >>> https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_fs_fs/structure > >>> A representation begins with a line containing either "PLAIN\n" or > >>> "DELTA\n" or "DELTA \n", where , , > >>> and give the location of the delta base of the representation > >>> and the amount of data it contains (not counting the header or > >>> trailer). If no base location is given for a delta, the base is the > >>> empty stream. After the initial line comes raw svndiff data, followed > >>> by a cosmetic trailer "ENDREP\n". > > >> `M-j 1916 RET', takes me here: > >> 0770: 4e44 5245 500a 4445 4c54 410a 5356 4e01 NDREP.DELTA.SVN. > >> 0780: 8c0c 823d 8431 823b 8140 4c00 8844 .=.1.;.@L..D > >> 0790: 4ca8 404e 0048 8100 bf45 820d 9945 820d L.@N.H...E...E.. > >> ... > > > So... trying to interpret this with the description that Stefan gave, > > and that Daniel referred to. Here we have "DELTA\n" (the "0A" is "\n"), > > which begins a representation according to the above description. > > > But this is where I get a bit lost. > > Just analyzing a little bit further from the /subversion/notes/svndiff > description that Daniel mentioned. > > The first part after "DELTA\n" is "SVN" followed by a one byte format > version number, in this case 0x01. > > And then comes the hard part: > After the header come one or more windows, until the document ends. > (So the decoder must have external context indicating when there is no > more svndiff data.) > > Then comes the description of a window, and that is, as Daniel said, > probably not human readable, or even recognizable in a hex editor, > because the window may be zlib compressed (or so I understand it). > > Hm... hard this is. The dark side strong in this one. > >
Re: fsverify.py unable to fix invalid svndiff header
On Sat, 14 May 2011 22:46 +0300, "Daniel Shahaf" wrote: > You probably don't need to know anything at all about svndiff --- if previous > instances of the bug are representative, then you're free to treat consider > the svndiff stream an opaque binary data. > > That is, previously what happened is that fsfs wrote > > A_1 A_2 B_1 B_2 C_1 C_1 C_2 D_1 D_2 E_1 E_2 > instead of the correct A_1 A_2 B_1 B_2 C_1 C_2 D_1 D_2 E_1 E_2 where again A_i,B_i don't stand for anything in particular other than "portions of the rev file that should be written in sequence". Daniel (since when does shift+enter send in the webmail?) > On Sat, 14 May 2011 21:24 +0200, "Steinar Bang" wrote: > > > Steinar Bang : > > > > > Steinar Bang : > > > Stefan Sperling : > > > > >>> Try to locate boundaries of representations, which look as follows: > > > > >>> https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_fs_fs/structure > > >>> A representation begins with a line containing either "PLAIN\n" or > > >>> "DELTA\n" or "DELTA \n", where , , > > >>> and give the location of the delta base of the representation > > >>> and the amount of data it contains (not counting the header or > > >>> trailer). If no base location is given for a delta, the base is the > > >>> empty stream. After the initial line comes raw svndiff data, followed > > >>> by a cosmetic trailer "ENDREP\n". > > > > >> `M-j 1916 RET', takes me here: > > >> 0770: 4e44 5245 500a 4445 4c54 410a 5356 4e01 NDREP.DELTA.SVN. > > >> 0780: 8c0c 823d 8431 823b 8140 4c00 8844 .=.1.;.@L..D > > >> 0790: 4ca8 404e 0048 8100 bf45 820d 9945 820d L.@N.H...E...E.. > > >> ... > > > > > So... trying to interpret this with the description that Stefan gave, > > > and that Daniel referred to. Here we have "DELTA\n" (the "0A" is "\n"), > > > which begins a representation according to the above description. > > > > > But this is where I get a bit lost. > > > > Just analyzing a little bit further from the /subversion/notes/svndiff > > description that Daniel mentioned. > > > > The first part after "DELTA\n" is "SVN" followed by a one byte format > > version number, in this case 0x01. > > > > And then comes the hard part: > > After the header come one or more windows, until the document ends. > > (So the decoder must have external context indicating when there is no > > more svndiff data.) > > > > Then comes the description of a window, and that is, as Daniel said, > > probably not human readable, or even recognizable in a hex editor, > > because the window may be zlib compressed (or so I understand it). > > > > Hm... hard this is. The dark side strong in this one. > > > > >
Error connecting to an Apache svn server remotely
I set up an apache SVN server on an Opensuse Linux server within the university department network. Access to it from within the university network is completely fine. One day when I tried to access from outside the university using the linux svn client, I got one of these two errors (nondeterministic): kzhu@tux:~$ svn list http://www.xxx.edu/mysvn svn: OPTIONS of 'http://www.xxx.edu/mysvn': Could not parse response status line. (http://www.xxx.edu) kzhu@tux:~$ svn list http://www.www.edu/mysvn svn: PROPFIND of '/mysvn': Could not read status line: connection was closed by server. (http://www.xxx.edu ) I looked at the apache access log on the server, and see these: xxx.xxx.xxx.xxx - - [15/May/2011:13:26:04 +0800] "OPTIONS /mysvn HTTP/1.1" 401 1286 "-" "SVN/1.6.15 (r1038135) neon/0.25.5" xxx.xxx.xxx.xxx - kzhu [15/May/2011:13:26:05 +0800] "OPTIONS /mysvn HTTP/1.1" 200 188 "-" "SVN/1.6.15 (r1038135) neon/0.25.5" I also looked at the access log of successful access to the serve from within the university network, I also see these two lines, but there are additional lines that indicates the successful retrieval of file listing: yyy.yyy.yyy.yyy - - [15/May/2011:13:29:34 +0800] "OPTIONS /mysvn HTTP/1.1" 401 1286 "-" "SVN/1.6.16 (r1073529) neon/0.29.3" yyy.yyy.yyy.yyy - kzhu [15/May/2011:13:29:34 +0800] "OPTIONS /mysvn HTTP/1.1" 200 188 "-" "SVN/1.6.16 (r1073529) neon/0.29.3" yyy.yyy.yyy.yyy - kzhu [15/May/2011:13:29:34 +0800] "PROPFIND /mysvn HTTP/1.1" 207 647 "-" "SVN/1.6.16 (r1073529) neon/0.29.3" yyy.yyy.yyy.yyy - kzhu [15/May/2011:13:29:34 +0800] "PROPFIND /mysvn/!svn/vcc/default HTTP/1.1" 207 399 "-" "SVN/1.6.16 (r1073529) neon/0.29.3" yyy.yyy.yyy.yyy - kzhu [15/May/2011:13:29:34 +0800] "PROPFIND /mysvn/!svn/bln/97 HTTP/1.1" 207 452 "-" "SVN/1.6.16 (r1073529) neon/0.29.3" ... omitted more lines. I looked at the firewall on the opensuse server (in yast) and make sure that the firewall is turned off, but it doesn't help! I also tried accessing from different places outside the university and I got the same error. I'm out of ideas! Please help!!
