Re: Svn 1.9 repository 20% bigger than svn 1.8 repository
On 29 January 2016 at 02:40, Philip Martin wrote: > > rep-cache.db files are different, 39.7 MB in 1.9 and 58.4 MB in 1.8 > > The rep-cache is an SQLite file that contains a map of checksums to > revisions and if stops working then Subversion will continue to allow > commits but will do less, or no, deduplication. > > You can see how many entries there are with: > > sqlite3 repo/db/rep-cache.db "select count(*) from rep_cache order" > > You can see the entries with: > > sqlite3 repo/db/rep-cache.db "select hash,revision from rep_cache order > by revision" > > I suspect 1.9 will show fewer entries, or many refuse to return any > results at all if the file is corrupt. > You are correct, there are 341378 entries in 1.9 file vs 499837 in 1.8 And the failing de-duplication seems to be reason indeed. I have located one revision which is 112 MB in 1.9 repository but 15 kB in 1.8 repo. The commit is "add" (without history) of several files, some binary, some text, total of 900+ MB. Binary files are zip, xls and xlsx, ~92 MB, text files txt, xml, csv, ~840 MB How can I test if the 1.9 has really corrupt sqlite db file? For me it seems that it still gets updates (i.e. when I bring in new revisions with svnsync.) Gert
Re: BUG - SVN tries to connect to "akamai" - 15 second timeout - CRL - ctldl.windowsupdate.com
Cameron Sours wrote on Wed, Jan 27, 2016 at 20:32:45 +: > The windows Crypto API was attempting to connect to Windows Update to > retrieve Certificate revocation information (CRL - certificate > revocation list). The default timeout for CRL retrieval is 15 seconds. > The timeout for authentication on the server is 10 seconds; as 15 is > greater than 10, this fails. Thanks for sharing the problem and solution. > **Suggested SVN Changes:** > > 1.Enable Verbosity option on the command line for all operations The config flag from (3), once added, will be settable from the command-line through the --config-option flag. Is that what you mean? > 2.Add error ENUM name to stderr This is already done in debug builds (windows) and maintainer builds (unix). When dealing with release builds, you can convert E170013 codes to symbolic names with tools/dev/which_error.py. The E730054 code you saw is the sum of SOCECONNRESET = 10054 APR_OS_START_SYSERR = 72 (from tools/dev/aprerr.txt), which that script doesn't yet handle. > 3.Add config flag for Serf Library debug logging. > Cheers, Daniel
Re: Svn 1.9 repository 20% bigger than svn 1.8 repository
Gert Kello writes: > You are correct, there are 341378 entries in 1.9 file vs 499837 in 1.8 > > And the failing de-duplication seems to be reason indeed. I have located > one revision which is 112 MB in 1.9 repository but 15 kB in 1.8 repo. The > commit is "add" (without history) of several files, some binary, some text, > total of 900+ MB. Binary files are zip, xls and xlsx, ~92 MB, text files > txt, xml, csv, ~840 MB > > How can I test if the 1.9 has really corrupt sqlite db file? For me it > seems that it still gets updates (i.e. when I bring in new revisions with > svnsync.) Check the index with: sqlite3 db/rep-cache.db "pragma integrity_check" Compare the ordered list: sqlite3 db/rep-cache.db "select hash,revision from rep_cache order by revision" with the 1.8 list and you may be able to identify which revisions are missing from the 1.9 rep-cache. There isn't any way to fix the missing deduplication other than a dump/load, but it would be interesting to find out why it failed. Which method did you use to write during sync: file, svn or http? -- Philip Martin WANdisco
Re: Svn 1.9 repository 20% bigger than svn 1.8 repository
> I have a svn 1.9 repository, created with svnsync, that has ~15 > > revisions and size about 45 GB. > > 300kB/rev is quite large, like >1 MB of changes before > compression - on average. Are these office documents, > large xml / html files or simply many files per commit? > > The content is mixed. Quite many small, source code commits. But office documents and zip archives as well. There are even few extremely huge commits, biggest one is 3+GB, one 800+MB and one 500+MB (as per revision file size in db/revs folder) > > Due to some issues in svn-all-fast-export I > > wanted to have svn 1.8 version repository so I downgraded it by doing > > svnadmin (v 1.9) dump /svnadmin (v 1.8) load cycle. I was surprised that > > the size of v 1.8 repository is "only" 37.5 GB > > I tried to compare content of db\revs folder: some files are bigger in > 1.8 > > repo, some in 1.9 repo. > > For the record: you already said elsewhere in this > thread that you used 1.8 to create the 1.8 repo and > 1.9 for the 1.9. I also assume standard settings > as in "no fsfs.conf tweaks". > > Correct. > There is a simple way to compare the "content size" > your repositories. Run the 1.9 svnfsfs tool on both: > > svnfsfs stats -M 1000 /path/to/repo > /some/output/path > > It basically reads the whole repository, groups and > aggregates the item sizes and produces a long report. > Number of changes and node revision should be more > or less (exactly?) the same. If they are, you'll > be good. > > "Representation" size is where the numbers will differ. > Looking at the differences in detail, you should be able > to pin down one or two file extensions that account for > most of the increase. It would be interesting to learn > what is special about them ... > Yes, number of changes and number of node revision records are identical. Number of representation do differ (1.744.149 @1.8 vs 1.901.312 @1.9) The "nodes total", "directory noderevs" and "file noderevs" numbers are identical The "Largest representations:" sections shows that 1.9 has failed to de-duplicate several files (executables in this case) The "Extensions by number of representations:" shows that all extensions have bigger number of representations in 1.9 repo The size if representations is most increased for .exe and .pdf extensions, where .exe causes 5GB increase and .pdf 500MB. Several types cause increase ~300MB, "others" have +1GB The dump/load cycle into 1.9 is finished as well, now it is 36.2 GB (less compared to 1.8 which was 37.5 GB). Both 1.9->1.9 and 1.8->1.9 resulted almost identical repos when comparing files byte by byte (the exception is UUID file)... Which makes me wonder if I dumped the same rep twice. Too bad the windows cmd doesn't retain command history. Gert
Re: Svn 1.9 repository 20% bigger than svn 1.8 repository
On 29 January 2016 at 11:52, Philip Martin wrote: > > How can I test if the 1.9 has really corrupt sqlite db file? For me it > > seems that it still gets updates (i.e. when I bring in new revisions with > > svnsync.) > > Check the index with: > > sqlite3 db/rep-cache.db "pragma integrity_check" > Says "ok" > Compare the ordered list: > > sqlite3 db/rep-cache.db "select hash,revision from rep_cache order by > revision" > > with the 1.8 list and you may be able to identify which revisions are > missing from the 1.9 rep-cache. There isn't any way to fix the missing > They are identical up to rev 91450. And then there's a large gap between revisions 91450 and 155838. 155838 seems to be where new "svnsync" was started, there's gap of time stamps in revision files. 91450 does not seem to be special in term of sync sequence. About 60 revisions within same minute, previous two in same second, next one 4 seconds later (How does svnsync work? "find last rev at source and sync up to that" or "sync while next revision present in source"? If latter then first sync failed for some reason... If first then it probably ran till completion. But does it actually matter?) > deduplication other than a dump/load, but it would be interesting to > find out why it failed. Should I look at some svn oog information about the revisions where is started to fail? Anything special to look for? Or should we assume "antivirus or something else opened the file so that svnsync was unable to write it"? The sync for this rev has been executing in the middle of day so it is possible I tried to peek "how big the repo is at disk". > Which method did you use to write during sync: > file, svn or http? > file:/// Gert
Re: Svn 1.9 repository 20% bigger than svn 1.8 repository
Gert Kello writes: > They are identical up to rev 91450. And then there's a large gap between > revisions 91450 and 155838. 155838 seems to be where new "svnsync" was > started, there's gap of time stamps in revision files. > 91450 does not seem to be special in term of sync sequence. About 60 > revisions within same minute, previous two in same second, next one 4 > seconds later The rep-cache failure almost certainly explains the size difference. > (How does svnsync work? "find last rev at source and sync up to that" or > "sync while next revision present in source"? If latter then first sync > failed for some reason... If first then it probably ran till completion. > But does it actually matter?) svnsync uses revision properties on r0 that get updated as the sync proceeds: svnlook proplist -v --revprop -r0 path/to/repo > Should I look at some svn oog information about the revisions where is > started to fail? Anything special to look for? Or should we assume > "antivirus or something else opened the file so that svnsync was unable to > write it"? The sync for this rev has been executing in the middle of day so > it is possible I tried to peek "how big the repo is at disk". The SQLite code expects there to be contention for the file and has a timeout (10s as I recall). In your case something appears to have blocked access for an extended period of time. -- Philip Martin WANdisco
Re: Svn 1.9 repository 20% bigger than svn 1.8 repository
On Fri, Jan 29, 2016 at 12:42:53PM +0200, Gert Kello wrote: > Anything special to look for? Or should we assume > "antivirus or something else opened the file so that svnsync was unable to > write it"? > > > Which method did you use to write during sync: > > file, svn or http? > > > > file:/// This makes me wonder if you used file:// on top of a network drive of some kind? SQlite is known to have problems with network drives (as hinted at in http://sqlite.org/faq.html#q5).
Re: Svn 1.9 repository 20% bigger than svn 1.8 repository
> > Which method did you use to write during sync: > > > file, svn or http? > > > > > > > file:/// > > This makes me wonder if you used file:// on top of a network drive of > some kind? SQlite is known to have problems with network drives (as > hinted at in http://sqlite.org/faq.html#q5). > Well, it is virtual machine somewhere in our server rack. Really not sure what is the underlying hardware. But for the virtual machine itself it is "local" disk. Gert
Re: Svn 1.9 repository 20% bigger than svn 1.8 repository
I managed to replay only to Philip first, so I have a chance to write changed/better answer to list :) > Should I look at some svn oog information about the revisions where is > > started to fail? Anything special to look for? Or should we assume > > "antivirus or something else opened the file so that svnsync was unable > to > > write it"? The sync for this rev has been executing in the middle of day > so > > it is possible I tried to peek "how big the repo is at disk". > > The SQLite code expects there to be contention for the file and has a > timeout (10s as I recall). In your case something appears to have > blocked access for an extended period of time. > > The revision file write times: 91448 -> 13:16:44 91449 -> 13:16:44 91450 -> 13:16:44 91451 -> 13:16:48 91452 -> 13:16:48 91452 -> 13:17:00 91450 was the last one that has data is sqlite database. 91451 has nothing in 1.8 db neither 91452 has one entry in 1.8 db. So the 10 sec timeout is possible. However, I peeked into the event logs; seems like our server is logging all process starts/ends. I see that at 13:16:48, cmd.exe process was started. 13:16:49 a wermgr.exe is started - its windows error reporting tool. And 13:16.50 a rundll32.exe is started. The wermgr.exe ends at 13:16.50 and rundll32.exe at 13.16.55, and after that, next cmd.exe starts at 13:16:59 For me it seems those "cmd.exe" starts are related to svnsync, at least there are many of those during sync. I tried to find the error report generated but failed. Most likely do not know how to search it correctly. Gert
Re: Svn 1.9 repository 20% bigger than svn 1.8 repository
Gert Kello writes: > The revision file write times: > 91448 -> 13:16:44 > 91449 -> 13:16:44 > 91450 -> 13:16:44 > 91451 -> 13:16:48 > 91452 -> 13:16:48 > 91452 -> 13:17:00 > > 91450 was the last one that has data is sqlite database. > 91451 has nothing in 1.8 db neither > 91452 has one entry in 1.8 db. > > So the 10 sec timeout is possible. However, I peeked into the event logs; > seems like our server is logging all process starts/ends. I see that at > 13:16:48, cmd.exe process was started. 13:16:49 a wermgr.exe is started - > its windows error reporting tool. And 13:16.50 a rundll32.exe is started. > The wermgr.exe ends at 13:16.50 and rundll32.exe at 13.16.55, and after > that, next cmd.exe starts at 13:16:59 > > For me it seems those "cmd.exe" starts are related to svnsync, at least > there are many of those during sync. I tried to find the error report > generated but failed. Most likely do not know how to search it correctly. Subversion has repository handles and only attempts to open the rep-cache once for each repository handle, the result is cached in the handle. Running svnsync could open multiple handles but usually reuses a single handle. In the case of file:// the svnsync client simply reuses the handle. In the case of svn:// and http:// the svnsync client does the sync over one connection and the server caches the handle for the connection. If the rep-cache fails to open when first accessed then this failure can be cached for the entire svnsync run with the result that the entire svnsync run will do no deduplication. This is probably a good thing if the problem is a corrupt file that will always fail to work. It is less good for a transient problem cause by interference from some Windows anti-virus/indexing. -- Philip Martin WANdisco
'D:\Development\SVN\Releases\TortoiseSVN-1.9.2\ext\subversion\subversion\libsvn_client\cleanup.c' line 227: assertion failed (svn_dirent_is_absolute(dir_abspath))
--- Subversion Exception! --- Subversion encountered a serious problem. Please take the time to report this on the Subversion mailing list with as much information as possible about what you were trying to do. But please first search the mailing list archives for the error message to avoid reporting the same problem repeatedly. You can find the mailing list archives at http://subversion.apache.org/mailing-lists.html Subversion reported the following (you can copy the content of this dialog to the clipboard using Ctrl-C): In file 'D:\Development\SVN\Releases\TortoiseSVN-1.9.2\ext\subversion\subversion\libsvn_client\cleanup.c' line 227: assertion failed (svn_dirent_is_absolute(dir_abspath)) --- Aceptar ---
Re: 'D:\Development\SVN\Releases\TortoiseSVN-1.9.2\ext\subversion\subversion\libsvn_client\cleanup.c' line 227: assertion failed (svn_dirent_is_absolute(dir_abspath))
Hi, rosa scarz wrote: > Please take the time to report this on the Subversion mailing list > with as much information as possible about what > you were trying to do. You did not do that. > In file > > 'D:\Development\SVN\Releases\TortoiseSVN-1.9.2\ext\subversion\subversion\libsvn_client\cleanup.c' > line 227: assertion failed (svn_dirent_is_absolute(dir_abspath)) Update to 1.9.3 and try again. Andreas
Re: svn-all-fast-export crash (was: Re: Svn 1.9 repository 20% bigger than svn 1.8 repository)
> On Jan 28, 2016, at 1:54 AM, Gert Kello wrote: > > I have a svn 1.9 repository, created with svnsync, that has ~15 revisions > and size about 45 GB. Due to some issues in svn-all-fast-export I wanted to > have svn 1.8 version repository so I downgraded it by doing svnadmin (v 1.9) > dump /svnadmin (v 1.8) load cycle. Gert, I was going to mention that I'm also trying to use svn-all-fast-export with a repository of ~145000 revisions, created by svnsync 1.9 from an older repository (which was at one point converted from cvs), and that I'm also having issues, specifically svn-all-fast-export crashes after every import, and branches and tags don't get imported. Then I noticed that you already commented on the bug report I filed about that: https://github.com/svn-all-fast-export/svn2git/issues/5 I'll try doing svn-all-fast-export with Subversion 1.8 libraries instead to confirm if that fixes it for me too.
