Hi David,
thanks for the report; the issue turned out to be quite serious.
The underlying bug has been present since at least as early as
1.8.0 but only surfaced with the added SHA1-collision detection
code in 1.8.18.
Under certain circumstances, yet-to-be-committed data would be
cached under the wrong key. The impact ranges from bogus errors
as you experienced them to silent repository corruption that only
a 'svnadmin verify' run would detect. So far, no such corruption
has been demonstrated but it remains a plausible scenario.
Luckily, the problem seems to be mostly restricted to 'svnadmin
load', with the "rep sharing" enable for the respective repo.
'svnadmin load -M 0' will also not have that issue.
I fixed the code and the fix should be shipped with the upcoming
1.8.19 release.
-- Stefan^2.
On 29.07.2017 14:37, Bert Huijben wrote:
This specific error message was added in the last release, so yes it is new in
the last versions. The last 1.9.x version also added it and I’m surprised that
you see the error on 1.8 and not 1.9 (or vice versa).
It tries to tell you that you have two files with an identical SHA-1 hash, but
different contents. A case that we accidentally didn’t catch before, that might
have caused some data loss under extreme circumstances.
This scenario is pretty unlikely to trigger, unless you specifically try to do
that. I’ll try to verify what exactly is in your dump file to see if you really
hit this scenario, or that we have a bug in our code that tries to catch this
scenario. Assuming you properly verified with 1.9.6 (and not 1.9.5 or earlier) I
assume that this is a bug in 1.8.18, as other scenarios are really extremely
unlikely.
I’m not sure I’ll have time to look into this before Monday though, so perhaps
one of the other developers beats me to it.
Bert
*From:*David Engel [mailto:den...@magnitude.com]
*Sent:* zaterdag 29 juli 2017 00:01
*To:* users@subversion.apache.org
*Subject:* Checksum mismatch bug in 1.8.18
Hi,
I think I’m running into a bug in svnadmin in the latest 1.8 release (1.8.18).
It is not present in 1.8.17 nor 1.9.5 nor 1.9.6. I’m guessing it’s related to
the recent changes around strict repsharing. I’m not subscribed to the mailing
list, so please CC me on any responses.
Here are the relevant details:
OS: Windows 10 and Windows 2012 R2
Subversion release: 1.8.18
Both SlikSVN release:
svnadmin, version 1.8.18-SlikSvn-1.8.18-X64 (SlikSvn/1.8.18) X64
compiled Jul 17 2017, 13:03:37 on x86_64-microsoft-windows6.2.9200
And Collabnet release:
svnadmin, version 1.8.18 (r1800620)
compiled Jul 7 2017, 05:51:59 on x86_64-microsoft-windows6.2.9200
Performing an svnadmin load results in the following error:
...
<<< Started new transaction, based on original revision 650
* editing path :
Dev/Common/Utils/External/StampVerData/MetaBuilderVersion.inf ... done.
* editing path :
Dev/Common/Utils/External/StampVerData/MetaBuilderVersionDBG.inf ... done.
* editing path : Dev/Common/Utils/External/StampVerData/OracleVersion.inf
...svnadmin: E160000: SHA1 of reps '-1 0 45 132 a6ea37d29094deece56250ebe167ce46
6a2d6bc4c7e58c0d2627a1b0dd5b23e542aec4d1 650-i2/_8' and '-1 136 45 132
a6ea37d29094deece56250ebe167ce46 6a2d6bc4c7e58c0d2627a1b0dd5b23e542aec4d1
650-i2/_8' matches (6a2d6bc4c7e58c0d2627a1b0dd5b23e542aec4d1) but contents differ
svnadmin: E160004: Filesystem is corrupt
svnadmin: E200014: Checksum mismatch while reading representation:
expected: a6ea37d29094deece56250ebe167ce46
actual: 5f696f5d0755f3bcb5e927bdfba4bfa8
In order to reproduce the error, you can use the attached VersionStamps3
repository dump file. It was taken with the same version of svnadmin from a
multi-GB repository that’s been around for over a decade and pruned
(svndumpfilter) to a small set of files and revisions in one folder while still
encountering the error. I first encountered the error another place very
quickly in an incremental load on top of a repo that was loaded from 1.8.14, but
that one was on a revision north of 120,000+. This is way easier to reproduce.
I tried filtering the dump further and dropping empty revisions, but the error
did not occur in those cases. The error also did not occur using 1.8.17,
1.9.5., and 1.9.6. It also did not occur in 1.8.18 if I set enable-rep-sharing
= false before performing the load.
Repro.bat script:
@echo off
:defineCommands
rem You might need to adjust these lines to point to your
rem compiled-from-source Subversion binaries, if using those:
for %%i in (svn.exe) do set SVN="%%~$PATH:i"
for %%i in (svnadmin.exe) do set SVNADMIN="%%~$PATH:i"
:defineUrls
rem Only supported access method: file:// <file:///>. If http:// or svn://, then
rem you'll have to configure it yourself first.
set URL=file:///%CD%/repos
set URL=%URL:\=/%
echo Base url for repo: %URL%
:cleanAllDirsAndCreateRepo
if exist repos rmdir /s /q repos
if exist import-me rmdir /s /q import-me
if exist wc rmdir /s /q wc
%SVNADMIN% create repos
pause
:prepareGreekTree
echo Making a Tree for import...
mkdir import-me\Dev\Common\Utils\External
echo Importing it...
cd import-me
%SVN% import -q -m "Initial import." %URL%
cd ..
rem This is where your reproduction recipe goes.
svnadmin --version
svnadmin load repos < VersionStamps3
goto :eof
Thanks,
David
