Seeing very slow performance with svnadmin verify

[Eric Johnson]

Using: svnadmin --version
svnadmin, version 1.9.4 (r1740329)

The first time I ran into this, I simply added the -M option. Since the 
default is reported just "16", I added "-M 384", and noticed a 
significant speed bump. However, I've got a repository where even this 
big jump didn't fix the performance problems. Current behavior is to 
appear to pause significantly, then report two versions verified. The 
pause lasts on the order of thirty seconds.


Since I have on the order of 230,000 additional commits to check, I'm 
looking at 39 days before my repository finishes verifying. That's not 
really acceptable.


Any suggestions? Am I doing something wrong?

Thanks!

Eric.

Re: Seeing very slow performance with svnadmin verify

[Andreas Mohr]

On Tue, Nov 01, 2016 at 12:14:32AM -0700, Eric Johnson wrote:
> Since I have on the order of 230,000 additional commits to check, I'm
> looking at 39 days before my repository finishes verifying. That's not
> really acceptable.
> 
> Any suggestions? Am I doing something wrong?

You could try to
run strace and/or ltrace with sufficiently detailed cmdline options,
to try to figure out
which system call exactly is involved in the delay/expensive handling.

A 30s pause might perhaps hint at some network / DNS resolve things.



HTH,

Andreas Mohr

-- 
GNU/Linux. It's not the software that's free, it's you.

Re: Seeing very slow performance with svnadmin verify

[Stefan Sperling]

On Tue, Nov 01, 2016 at 12:14:32AM -0700, Eric Johnson wrote:
> Using: svnadmin --version
> svnadmin, version 1.9.4 (r1740329)
> 
> The first time I ran into this, I simply added the -M option. Since the
> default is reported just "16", I added "-M 384", and noticed a significant
> speed bump. However, I've got a repository where even this big jump didn't
> fix the performance problems. Current behavior is to appear to pause
> significantly, then report two versions verified. The pause lasts on the
> order of thirty seconds.
> 
> Since I have on the order of 230,000 additional commits to check, I'm
> looking at 39 days before my repository finishes verifying. That's not
> really acceptable.

I doubt this extrapolation makes sense, because the effort to verify a
revision depends on the nature of the changes made in the revision.
Perhaps some earlier revisions contain huge imports and are thus expensive
to verify?

> Any suggestions? Am I doing something wrong?

Please provide more information about this repository, in particular
the output of 'svnadmin info', and the number of directory entries,
average file sizes, etc. ('svnfsfs stats' will show these numbers).

There are a lot of variables involved so it's hard to give you good
advice without more detailed information. See here for more context:
http://subversion.apache.org/docs/release-notes/1.9.html#fsfs-improvements

There is now a --metadata-only option for 'svnadmin verify' but it's hard
to tell if this would help your particular case since it's unclear which
repository format you are using.
http://subversion.apache.org/docs/release-notes/1.9.html#svnadmin-verify

SVN Properties

[CM Analyst]

  Hello, In our environment, there is a need to identify one or more set of 
files witha custom attribute. I expect SVN properties is the way to go? For a 
given set of .exes, I want to attach anattribute using the SVN property called 
"RC: Release Candidate". My questions: 1) Is this approach a recommended use 
for the SVN'sproperty feature? 2) Is there a way to search to later retrieve 
all .exeswith the given property? In my testing so far, I created: $ svn 
propset Released "This SW component isavailable for release." fooproperty 
'Released' set on 'foo' $ svn proplist fooProperties on 'foo':  Released  
svn:mime-type However, I was not able to then search and locate thefile by its 
property. 3) What is a better way to indicate a given status/attribute on aset 
of files?
 Amad   

Re: SVN Properties

[Eric Johnson]

Hi Amad


On 11/1/16 10:26 AM, CM Analyst wrote:

Hello,
In our environment, there is a need to identify one or more set of 
files with a custom attribute. I expect SVN properties is the way to go?
For a given set of .exes, I want to attach an attribute using the SVN 
property called "RC: Release Candidate".

My questions:
1) Is this approach a recommended use for the SVN's property feature?
While there is nothing wrong with this use, given #2, it doesn't seem 
like a particularly useful path.


Also, contrary to the apparent purpose of your proposed label, the label 
will stick to the file even as you modify it. Which means that an update 
to the EXE, would still be marked "RC: Release Candidate", and unless 
you took an extra step in order to clear that setting. Since that step 
could be forgotten, unless automated, it seems like a less-than-robust 
mechanism.


Why not copy the exe to a "Release Candidate" folder in SVN?
2) Is there a way to search to later retrieve all .exes with the given 
property?
No. Also unclear whether you want to do this only for "latest", but also 
for "over time". If you want to do it over time, that means exhaustively 
searching through older releases.



In my testing so far, I created:
$ svn propset Released "This SW component is available for release." foo
property 'Released' set on 'foo'
$ svn proplist foo
Properties on 'foo':
Released
svn:mime-type
However, I was not able to then search and locate the file by its 
property.


Exactly.

3) What is a better way to indicate a given status/attribute on a set 
of files?


See my suggestion above about copying into a known folder. Would that 
satisfy your requirements?


Eric.


Amad

Re: SVN Properties

[CM Analyst]

Eric,
Thank you for your response. 
Actually, we do copy executables to a "release" directory in SVN. 
However, for that to happen, the release content has to be defined in advance, 
completed the formal test cycle. 
For some stakeholders, that's too late. Our challenge is really about how to 
determine release content.
They want the ability to "see" (determine the eligibility of an .exe) 
beforehand: after coding and testing is done but before SW is copied to the 
'release' directory. 
My thought was to see if I can create some naming scheme by which to tie 
disparate components together using the property feature. 
Perhaps that's not possible.
  

On Tuesday, November 1, 2016 2:36 PM, Eric Johnson  wrote:
 

 Hi Amad


On 11/1/16 10:26 AM, CM Analyst wrote:
> Hello,
> In our environment, there is a need to identify one or more set of 
> files with a custom attribute. I expect SVN properties is the way to go?
> For a given set of .exes, I want to attach an attribute using the SVN 
> property called "RC: Release Candidate".
> My questions:
> 1) Is this approach a recommended use for the SVN's property feature?
While there is nothing wrong with this use, given #2, it doesn't seem 
like a particularly useful path.

Also, contrary to the apparent purpose of your proposed label, the label 
will stick to the file even as you modify it. Which means that an update 
to the EXE, would still be marked "RC: Release Candidate", and unless 
you took an extra step in order to clear that setting. Since that step 
could be forgotten, unless automated, it seems like a less-than-robust 
mechanism.

Why not copy the exe to a "Release Candidate" folder in SVN?
> 2) Is there a way to search to later retrieve all .exes with the given 
> property?
No. Also unclear whether you want to do this only for "latest", but also 
for "over time". If you want to do it over time, that means exhaustively 
searching through older releases.

> In my testing so far, I created:
> $ svn propset Released "This SW component is available for release." foo
> property 'Released' set on 'foo'
> $ svn proplist foo
> Properties on 'foo':
> Released
> svn:mime-type
> However, I was not able to then search and locate the file by its 
> property.

Exactly.

> 3) What is a better way to indicate a given status/attribute on a set 
> of files?

See my suggestion above about copying into a known folder. Would that 
satisfy your requirements?

Eric.
>
> Amad



   

Re: SVN Properties

[Eric Johnson]

Hi Amad,

On 11/1/16 1:48 PM, CM Analyst wrote:

Eric,

Thank you for your response.

Actually, we do copy executables to a "release" directory in SVN.

However, for that to happen, the release content has to be defined in 
advance, completed the formal test cycle.
You could have another folder which is just "Pending RC". EXEs could 
then be moved from there to the destination


For some stakeholders, that's too late. Our challenge is really about 
how to determine release content.
I think Subversion properties don't get you there. Subversion properties 
speak to properties of the thing they're assigned to, where those 
properties are about the folder or file, not about what external systems 
/ people think of the folder / file. That's why "mimetype" is a perfect 
versioned property - it is a property about the file itself.


Why not have a version-controlled file which serves the purpose of 
identifying release content?


They want the ability to "see" (determine the eligibility of an .exe) 
beforehand: after coding and testing is done but before SW is copied 
to the 'release' directory.


My thought was to see if I can create some naming scheme by which to 
tie disparate components together using the property feature.


Perhaps that's not possible.
The properties are not the same as a "tagging" mechanism that you might 
find in other contexts.


Eric.




On Tuesday, November 1, 2016 2:36 PM, Eric Johnson  wrote:


Hi Amad


On 11/1/16 10:26 AM, CM Analyst wrote:
> Hello,
> In our environment, there is a need to identify one or more set of
> files with a custom attribute. I expect SVN properties is the way to go?
> For a given set of .exes, I want to attach an attribute using the SVN
> property called "RC: Release Candidate".
> My questions:
> 1) Is this approach a recommended use for the SVN's property feature?
While there is nothing wrong with this use, given #2, it doesn't seem
like a particularly useful path.

Also, contrary to the apparent purpose of your proposed label, the label
will stick to the file even as you modify it. Which means that an update
to the EXE, would still be marked "RC: Release Candidate", and unless
you took an extra step in order to clear that setting. Since that step
could be forgotten, unless automated, it seems like a less-than-robust
mechanism.

Why not copy the exe to a "Release Candidate" folder in SVN?
> 2) Is there a way to search to later retrieve all .exes with the given
> property?
No. Also unclear whether you want to do this only for "latest", but also
for "over time". If you want to do it over time, that means exhaustively
searching through older releases.

> In my testing so far, I created:
> $ svn propset Released "This SW component is available for release." foo
> property 'Released' set on 'foo'
> $ svn proplist foo
> Properties on 'foo':
> Released
> svn:mime-type
> However, I was not able to then search and locate the file by its
> property.

Exactly.

> 3) What is a better way to indicate a given status/attribute on a set
> of files?

See my suggestion above about copying into a known folder. Would that
satisfy your requirements?

Eric.

>
> Amad

Re: Subversion Exception - cleanup

[Stefan]

On 10/28/2016 16:12, Dan Atkinson wrote:
> Hi Johan,
>
> The repository is on a local drive and I have several other
> repositories on the same drive as well that were unaffected.
>
> I do some tasks (update/tag) on the repository via the command line. I
> just checked and the version of SVN used on the command line is
> 1.8.15.14429.
>
> In any case, when I restarted my machine this morning, my repo was no
> longer corrupt.
>
> I don't know what caused it, or why it was fine again this morning,
> but this is no longer an issue for me.
>
> Kind regards, Dan
>
> On 28 October 2016 at 14:43, Johan Corveleyn  > wrote:
>
> [ Added Dan Atkinson to cc, because he asked us to :-). Dan, see
> below. ]
>
> On Fri, Oct 28, 2016 at 12:51 AM, Stefan  > wrote:
> > On 10/27/2016 16:36, Dan Atkinson wrote:
> >> Hi there,
> >>
> >> Firstly, I am not subscribed to this mailing list so would
> appreciate
> >> being explicitly CC'd in any responses. :-)
> >>
> >> When I attempted to update my repository, I received the
> following update:
> >>
> >> Error: The working copy database at 'D:\Work\SVN\trunk' is corrupt.
> >> Error: Try a 'Cleanup'. If that doesn't work you need to do a fresh
> >> checkout.
> >>
> >> When I attempted a cleanup, I received the following message:
> >>
> >> ---
> >> Subversion Exception!
> >> ---
> >> Subversion encountered a serious problem.
> >> Please take the time to report this on the Subversion mailing list
> >> with as much information as possible about what
> >> you were trying to do.
> >> But please first search the mailing list archives for the error
> message
> >> to avoid reporting the same problem repeatedly.
> >> You can find the mailing list archives at
> >> http://subversion.apache.org/mailing-lists.html
> 
> >>
> >> Subversion reported the following
> >> (you can copy the content of this dialog
> >> to the clipboard using Ctrl-C):
> >>
> >> In file
> >> 
> 
> 'D:\Development\SVN\Releases\TortoiseSVN-1.9.4\ext\subversion\subversion\libsvn_client\cleanup.c'
> >>  line 227: assertion failed (svn_dirent_is_absolute(dir_abspath))
> >> ---
> >> OK
> >> ---
> >>
> >>
> >> ---
> >> TortoiseSVN
> >> ---
> >> Cleanup failed to process the following paths:
> >> D:\Work\SVN\trunk
> >> The working copy database at 'D:\Work\SVN\trunk' is corrupt.
> >> Try a 'Cleanup'. If that doesn't work you need to do a fresh
> checkout.
> >> ---
> >> OK
> >> ---
> >>
> >> My version information is:
> >> TortoiseSVN 1.9.4, Build 27285 - 64 Bit , 2016/04/24 13:59:58
> >> Subversion 1.9.4, -release
> >> apr 1.5.2
> >> apr-util 1.5.4
> >> serf 1.3.8
> >> OpenSSL 1.0.2g  1 Mar 2016
> >> zlib 1.2.8
> >> SQLite 3.12.1
> >>
> >> I am running Windows 10 v1511 (OS Build 10586.601).
> >>
> >> Our SVN server (VisualSVN Server) is running SVN 1.7.6 (r1370777)
> >>
> >> I did try to search the archives for similar problems but
> received a
> >> 403 from Google when I attempted it.
> >>
> >> If you require any further information, please let me know
> directly.
> >>
> >> Kind regards, Dan Atkinson
> >
> > In case of a WC DB corruption I'd say your best bet is to do a fresh
> > checkout and copy over locally modified files from the old
> working copy
> > to the new one.
> >
> > You can also try to debug/troubleshoot the database corruption
> and try
> > to repair the database (sqlite3 -> pragma integrity_check) but
> unless
> > you have a reason to try to save your working copy, there's
> little point
> > to do so, IMO.
> >
> > Although I doubt it's related to the corruption you ran into: Your
> > VisualSVN Server is quite outdated. I assume you are running
> Visual SVN
> > Server 2.5.6. I'd certainly suggest you to upgrade your server
> to 2.5.26
> > at least (which will bring SVN up to 1.7.21). The upgrade should
> be as
> > simple as clicking through the installer.
> >
> > If possible, you'd even upgrade to a later one which will bring
> SVN up
> > to 1.8 or 1.9 (currently VisualSVN Server 3.5.6 is the
> recommended version).
>
> It would of course be interesting to know how the working copy could
> have ended up being corrupt. Any idea how it got corrupted, Dan?
>
I just recalled that we had cases of WC db corruption in our office too
on two machines, when they ran into a bluescreen. It was suggested th