Repository became corrupt on commit
Hi, Yesterday one of our repositories became corrupt when someone committed a simple text file. In the end we deleted the file and re-added it and it has been fine since then. We are using SVN 1.9.3 on the server. This is one of many repositories and the first time we have encountered such a situation. A verify showed this error: * Error verifying revision 728. svnadmin: E160013: Filesystem path 'trunk/scripts/script.py' is neither a file nor a directory The content of that file is what had replaced what used to be the trunk folder. When in that state it was not possible to checkout the repository or browse its contents (i.e. it was corrupt). We could see the log messages for the level above trunk but still could not check it out. Now that we have deleted then re-committed exactly the same file, the repository is back to normal and it seems like an unreproducable bug at this stage. Is this a known issue? Regards, Alan
Re: Repository became corrupt on commit
Hi Daniel,
Sorry, I meant to reply to all. For the benefit of others, here is what I said:
---
Thanks for the response.
> where dump-noderev.pl is [1]? The output should be just a few rfc822-like
> headers specifying the offset and checksum of the file contents representation
> ("rep"). One of the headers should be "type: file".
I'll take a look at this as soon as I can.
> Sorry, I don't follow. Can you explain again what the contents of the
> file is and what's its significance? Has the server been restarted
> between the two commit attempts? Is it svnserve (in which mode,
> -i/-t/-d/-T/service) or mod_dav_svn (which MPM)?
The file is a python script. It is just a text file. The first time it
was committed it corrupted the repository. We rolled back to a backup
and tried to commit the same file and it again broke the repository.
Only when we deleted, then added exactly the same file did the problem
go away. So it was reproducible until we rolled back and deleted the
file.
The server has not been restarted. It is mod_dav_dvn. I'm not sure
what you mean by MPM, sorry.
---
I'm afraid I'm new to perl so I may be doing something wrong, but I
checked out the entire infrastructure-puppet repository and ran the
command from within
~/infrastructure-puppet/modules/rootbin_asf/files/bin to get the
following output:
perl dump-noderev.pl /path/to/repository /trunk/scripts/script.py 728
Use of uninitialized value $noderev_id in split at dump-noderev.pl line 41.
Use of uninitialized value $txn_id in pattern match (m//) at
dump-noderev.pl line 42.
Use of uninitialized value $txn_id in pattern match (m//) at
dump-noderev.pl line 43.
Use of uninitialized value $REVISION in division (/) at dump-noderev.pl line 10.
Use of uninitialized value $REVISION in modulus (%) at dump-noderev.pl line 11.
Use of uninitialized value $REVISION in concatenation (.) or string at
dump-noderev.pl line 13.
Use of uninitialized value $REVISION in concatenation (.) or string at
dump-noderev.pl line 15.
Use of uninitialized value $REVISION in concatenation (.) or string at
dump-noderev.pl line 16.
Use of uninitialized value $file_offset in concatenation (.) or string
at dump-noderev.pl line 49.
xxd: 1024: No such file or directory
Is there a simple command that I can run rather than this script?
I tried to find out what our MPM is but I am not sure. I ran this command:
/usr/sbin/apache2 -l
Compiled in modules:
core.c
mod_so.c
mod_watchdog.c
http_core.c
mod_log_config.c
mod_logio.c
mod_version.c
mod_unixd.c
I'm not sure if that tells you anything but I don't think it is
prefork or worker.
Regards,
Alan
On Mon, Oct 22, 2018 at 4:54 PM Daniel Shahaf wrote:
>
> Please send that to the list, not just to me. :)
>
> (To save a round trip: an MPM is an httpd thing, see
> https://httpd.apache.org/docs/current/mpm.html . It's related to requests
> are distributed among httpd processes which factors in due to intra-process
> caches)
>
> Alan Spark wrote on Mon, 22 Oct 2018 16:50 +0100:
> > Hi Daniel,
> >
> > Thanks for the response.
> >
> > > where dump-noderev.pl is [1]? The output should be just a few rfc822-like
> > > headers specifying the offset and checksum of the file contents
> > > representation
> > > ("rep"). One of the headers should be "type: file".
> >
> > I'll take a look at this as soon as I can.
> >
> > > Sorry, I don't follow. Can you explain again what the contents of the
> > > file is and what's its significance? Has the server been restarted
> > > between the two commit attempts? Is it svnserve (in which mode,
> > > -i/-t/-d/-T/service) or mod_dav_svn (which MPM)?
> >
> > The file is a python script. It is just a text file. The first time it
> > was committed it corrupted the repository. We rolled back to a backup
> > and tried to commit the same file and it again broke the repository.
> > Only when we deleted, then added exactly the same file did the problem
> > go away. So it was reproducible until we rolled back and deleted the
> > file.
> >
> > The server has not been restarted. It is mod_dav_dvn. I'm not sure
> > what you mean by MPM, sorry.
> >
> > Regards,
> > Alan
> >
> > On Sat, Oct 20, 2018 at 6:04 PM Daniel Shahaf
> > wrote:
> > >
> > > Alan Spark wrote on Fri, 19 Oct 2018 16:04 +0100:
> > > > * Error verifying revision 728.
> > > > svnadmin: E160013: Filesystem path 'trunk/scripts/script.py' is
> > > > neither a file nor a directory
> > >
> > > E160013 is SVN_ERR_FS_CORRUPT, but this message is a new one.
> > >
> > > Ca
Re: Repository became corrupt on commit
Hi Daniel, I didn't get anywhere with the perl script. /usr/bin is definitely in the path and that is where svnlook is, and the SVNLOOK variable was already set... anyway I gave up and went for your alternative. Firstly I must apologise as I stated revision 728 in my original email but it looks like this was on one of our experimental copies of the repository that we no longer have. I do still have a copy of the broken repository at revision 719. With that in mind, I ran this: grep -a '^type:' /path/to/repository/db/revs/719 type: file type: dir I also ran on the last working revision: grep -a '^type:' /path/to/repository/db/revs/718 type: file type: dir type: dir type: dir type: dir type: dir I note that your path had a /0/ after /revs/ but this appears to be empty. I think I have confirmed that our MPM is prefork: a2query -M prefork I hope this helps. Let me know if you need me to check anything else. Regards, Alan On Tue, Oct 23, 2018 at 3:54 PM Daniel Shahaf wrote: > > Daniel Shahaf wrote on Tue, 23 Oct 2018 14:50 +: > > Alan Spark wrote on Tue, 23 Oct 2018 14:41 +0100: > > > perl dump-noderev.pl /path/to/repository /trunk/scripts/script.py 728 > > > Use of uninitialized value $noderev_id in split at dump-noderev.pl line > > > 41. > > > > Sorry about that; the script is obviously missing an error check. > > > > Looks like 'svnlook' does not exist in your $PATH. > > > > It's not obvious to non Perl speakers, but line 6 does the > > equivalent of: > > . > > if [ -z "$SVNLOOK" ]; then SVNLOOK=svnlook ; fi > > . > > so if you run the script with svn/svnadmin/svnlook in PATH, _or_ with > > $SVN, $SVNADMIN, $SVNLOOK set to those executables' full paths, then it > > should work. > > The above is the preferred way forward, but if you hit roadblocks with > that, there _is_ a crude alternative: > > % grep -a '^type:' /path/to/repository/db/revs/0/728 > > This will extract only one field from the rfc822-formatted node-rev > header. It's a poor substitute to the Perl script, but better than nothing. > > Cheers, > > Daniel
Re: Repository became corrupt on commit
Hi Daniel, > Odd. My first assumption is that the subshell on line 39 isn't behaving > as expected. If you run `svnlook tree --full-paths --show-ids -r 719 > /path/to/repository /trunk/scripts/script.py`, does print a path and a > node-rev id string (which looks like "a.b.c/d-e")? Here is the output that I got: svnlook tree --full-paths --show-ids -r 719 /path/to/repository /trunk/scripts/script.py svnlook: E160016: Failure opening '/trunk/scripts/script.py' svnlook: E160016: '/trunk' is not a directory in filesystem '6b6eeb5b-1909-4e6c-8b33-d77c8e70ace5' If I run this on 718 then I get this output: svnlook tree --full-paths --show-ids -r 718 /path/to/repository /trunk/scripts/script.py /trunk/scripts/script.py <3.0.r716/391789> > These all look correct. I assume that when you do a normal 'verify' > run, revisions 1:718 (inclusive) are verified successfully and 719 > errors out, correct? I.e., it prints "Verifying revision 718" and then > "Error verifying revision 719". That is correct: * Verified revision 718. * Error verifying revision 719. svnadmin: E160013: Filesystem path 'trunk/doc/EDITS.txt' is neither a file nor a directory > There's a slim chance that those "type:" lines had a NUL byte tacked on, > or something else that got lost in the translation to email. You can > rule out this remote possibility by piping the grep to xxd(1) (just > '| xxd', no option flags needed). Here is 718: grep -a '^type:' /path/to/repository/db/revs/718 | xxd : 7479 7065 3a20 6669 6c65 0a74 7970 653a type: file.type: 0010: 2064 6972 0a74 7970 653a 2064 6972 0a74 dir.type: dir.t 0020: 7970 653a 2064 6972 0a74 7970 653a 2064 ype: dir.type: d 0030: 6972 0a74 7970 653a 2064 6972 0a ir.type: dir. And 719: grep -a '^type:' /path/to/repository/db/revs/719 | xxd : 7479 7065 3a20 6669 6c65 0a74 7970 653a type: file.type: 0010: 2064 6972 0a > Otherwise, could you please confirm that the error you're getting on > r719 is the same one you posted for r728 (except for the revision > number, of course)? Yes, although I just noticed that it is a different file in this case but same error code: * Verified revision 718. * Error verifying revision 719. svnadmin: E160013: Filesystem path 'trunk/doc/EDITS.txt' is neither a file nor a directory I re-ran the svnlook commands on this file and get the same results: svnlook tree --full-paths --show-ids -r 719 /path/to/repository /trunk/doc/EDITS.txt svnlook: E160016: Failure opening '/trunk/doc/EDITS.txt' svnlook: E160016: '/trunk' is not a directory in filesystem '6b6eeb5b-1909-4e6c-8b33-d77c8e70ace5' And on 718: svnlook tree --full-paths --show-ids -r 718 /path/to/repository /trunk/doc/EDITS.txt /trunk/doc/EDITS.txt <166.0.r717/510746> Regards, Alan On Wed, Oct 24, 2018 at 3:08 PM Daniel Shahaf wrote: > > Alan Spark wrote on Wed, 24 Oct 2018 08:57 +0100: > > Hi Daniel, > > > > I didn't get anywhere with the perl script. /usr/bin is definitely in > > the path and that is where svnlook is, and the SVNLOOK variable was > > already set... anyway I gave up and went for your alternative. > > > > Odd. My first assumption is that the subshell on line 39 isn't behaving > as expected. If you run `svnlook tree --full-paths --show-ids -r 719 > /path/to/repository /trunk/scripts/script.py`, does print a path and a > node-rev id string (which looks like "a.b.c/d-e")? > > > Firstly I must apologise as I stated revision 728 in my original email > > but it looks like this was on one of our experimental copies of the > > repository that we no longer have. I do still have a copy of the > > broken repository at revision 719. With that in mind, I ran this: > > > > grep -a '^type:' /path/to/repository/db/revs/719 > > type: file > > type: dir > > > > I also ran on the last working revision: > > > > grep -a '^type:' /path/to/repository/db/revs/718 > > type: file > > type: dir > > type: dir > > type: dir > > type: dir > > type: dir > > > > These all look correct. I assume that when you do a normal 'verify' > run, revisions 1:718 (inclusive) are verified successfully and 719 > errors out, correct? I.e., it prints "Verifying revision 718" and then > "Error verifying revision 719". > > There's a slim chance that those "type:" lines had a NUL byte tacked on, > or something else that got lost in the translation to email. You can > rule out this remote possibility by piping the grep to xxd(1) (just > '| xxd', no option flags
