Problems compiling 1.7.0 on redhat el4 64bit

[michael_rytting]

For a while I was downloading and running the development build of subversion 
1.7.0.  At one revision of the code I started having an issue where svn would 
immediately segfault.  At that time I stopped using 1.7.0 assuming the issue 
would be fixed before release.  Unfortunately I just downloaded the beta2 
release of 1.7.0 and I'm having the same problem.  Interestingly enough I can 
successfully build beta2 on one of our 32bit systems, but I can't use that 
build on our 64 bit systems because it tries to link against the 32bit version 
of libmagic.

It's important to note that I don't get any errors compiling the design.  It's 
just that after I install it I get a segfault when I try and run svn.  
Ironically, if I run svn with no options or invalid options it successfully 
runs, but as soon as I add a valid option like "svn help" it segfaults.  I've 
spent multiple hours trying to compile with different options to my 
LD_LIBRARY_PATH/PATH variables and I've tried to mix up how I build apr, 
apr-util, serf, and neon, but it always results in the same problem.

I am using a very vanilla build process.

Wget
Tar
./get-deps.sh
./configure -prefix=$HOME/local
make && make install

As I've noted I have successfully built a development version of 1.7.0 on our 
64bit systems in the past.

-Mike

RE: Problems compiling 1.7.0 on redhat el4 64bit

[michael_rytting]

Like I've said, I was able to build subversion 1.7.0 on our 64bit systems in 
the past.  The build flow hasn't changed.  I've put a lot of time into trying 
different options to the build flow and I'm pretty sure that isn't the culprit.

As for your second recommendation, it doesn't look like libmagic can be 
disabled.  The configure script doesn't read the without_libmagic variable 
anywhere.  And the configure script doesn't properly deal with the "no" 
argument.   I just ended up deleting the entire libmagic section in the 
configure script and built subversion that way.  Now at least I can build on a 
32bit system and run the 32bit version of svn on our 64bit systems.

-Original Message-
From: Stefan Sperling [mailto:s...@elego.de] 
Sent: Tuesday, August 09, 2011 8:35 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: users@subversion.apache.org
Subject: Re: Problems compiling 1.7.0 on redhat el4 64bit

On Mon, Aug 08, 2011 at 04:19:06PM -0600, michael_rytt...@agilent.com wrote:
> For a while I was downloading and running the development build of subversion 
> 1.7.0.  At one revision of the code I started having an issue where svn would 
> immediately segfault.  At that time I stopped using 1.7.0 assuming the issue 
> would be fixed before release.  Unfortunately I just downloaded the beta2 
> release of 1.7.0 and I'm having the same problem.  Interestingly enough I can 
> successfully build beta2 on one of our 32bit systems, but I can't use that 
> build on our 64 bit systems because it tries to link against the 32bit 
> version of libmagic.
> 
> It's important to note that I don't get any errors compiling the design.  
> It's just that after I install it I get a segfault when I try and run svn.  
> Ironically, if I run svn with no options or invalid options it successfully 
> runs, but as soon as I add a valid option like "svn help" it segfaults.  I've 
> spent multiple hours trying to compile with different options to my 
> LD_LIBRARY_PATH/PATH variables and I've tried to mix up how I build apr, 
> apr-util, serf, and neon, but it always results in the same problem.
> 
> I am using a very vanilla build process.
> 
> Wget
> Tar
> ./get-deps.sh
> ./configure -prefix=$HOME/local
> make && make install
> 

This is most likely a problem with your build, not with Subversion itself. 
Maybe your build is getting 32bit and 64bit libraries mixed up?

> As I've noted I have successfully built a development version of 1.7.0 on our 
> 64bit systems in the past.

Note that libmagic support was added quite recently (in alpha2).
Does the problem disappear if you pass the --with-libmagic=no option to 
configure?

RE: Problems compiling 1.7.0 on redhat el4 64bit

[michael_rytting]

Unfortunately, some of us don't have a choice what version of linux we have to 
run on.  I end up compiling almost all the dependencies for subversion myself.  
I guess I'll need to track down exactly what change in the development process 
started breaking things.  For now, we have a solution where I've compiled on 
32bit with support for libmagic manually removed.

As a side note, it would be nice to actually be able to add a 
--without-libmagic and have the configure script work without hacking.  The 
release notes even say libmagic is optional, but the way I see it currently, 
the only we the default configure won't use it is if your build machine doesn't 
have libmagic installed.

-Original Message-
From: Nico Kadel-Garcia [mailto:nka...@gmail.com] 
Sent: Tuesday, August 09, 2011 12:59 PM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: users@subversion.apache.org
Subject: Re: Problems compiling 1.7.0 on redhat el4 64bit

On Mon, Aug 8, 2011 at 6:19 PM,   wrote:
> For a while I was downloading and running the development build of 
> subversion 1.7.0.  At one revision of the code I started having an 
> issue where svn would immediately segfault.  At that time I stopped 
> using 1.7.0 assuming the issue would be fixed before release.  
> Unfortunately I just downloaded the beta2 release of 1.7.0 and I'm having the 
> same problem.
> Interestingly enough I can successfully build beta2 on one of our 
> 32bit systems, but I can't use that build on our 64 bit systems 
> because it tries to link against the 32bit version of libmagic.

RHEL 4 is, frankly, way too old to support with new tools. It's initial release 
was over six years, and far too many core libraries, such as swig and sqlite 
and python, have not gotten updated to match subversion requirements.

I'm trying to rebundle an SRPM for 1.6.17 for the RPMforge repository, and no 
way am I burning the cycles to get it back to RHEL 4. The
1.7.0beta3 releases will compile well enough on RHEL 5, especially if you build 
from an SRPM for 1.6.x first to get most of the dependencies resolved.

RHEL 5, and Scientific Linux 5, have subversion-1.6.11 available from the 
upstream vendor. Would that be enough for you?

RE: Problems compiling 1.7.0 on redhat el4 64bit

[michael_rytting]

Ok, I've tracked down which revision caused the problem.  It happened in rev 
1104160.  Stefan2 made a change to utf.c to speed up UTF8 conversion.  Ever 
since this change went in I am seeing subversion crash when I compile on 64bit 
el4.

Just for kicks and giggles I updated to the HEAD revision of 
http://svn.apache.org/repos/asf/subversion/trunk.  Did a reverse merge on rev 
1104160, cleaned up the conflicts and built it and viola!  No more crashing 
when built on 64bit el4.

-Mike

RE: Problems compiling 1.7.0 on redhat el4 64bit

[michael_rytting]

I am attaching a patch file which I used to change utf.c in the head revision 
to get things to play nice with el4 64bit.  "make check" does come back clean 
with this version.

-Mike

P.S. why isn't "make check" structured so that the -j option to make would 
work.  It would be nice to use multiple threads to speed up the run.

From: Mark Phippard [mailto:markp...@gmail.com]
Sent: Tuesday, August 09, 2011 2:56 PM
To: RYTTING,MICHAEL (A-ColSprings,ex1); Subversion Development
Cc: users@subversion.apache.org
Subject: Re: Problems compiling 1.7.0 on redhat el4 64bit


On Tue, Aug 9, 2011 at 4:49 PM, 
mailto:michael_rytt...@agilent.com>> wrote:

Ok, I've tracked down which revision caused the problem.  It happened in rev 
1104160.  Stefan2 made a change to utf.c to speed up UTF8 conversion.  Ever 
since this change went in I am seeing subversion crash when I compile on 64bit 
el4.

Just for kicks and giggles I updated to the HEAD revision of 
http://svn.apache.org/repos/asf/subversion/trunk.  Did a reverse merge on rev 
1104160, cleaned up the conflicts and built it and viola!  No more crashing 
when built on 64bit el4.

Thanks.  Moving the thread to dev@

Can someone look at this and possibly fix it?

If you are not on users@, and to summarize this thread.  SVN builds fine on 
RHEL 4 64-bit but crashes immediately on any command, even svn help.  I saw the 
same problem with our builds.

--
Thanks

Mark Phippard
http://markphip.blogspot.com/

RE: Problems compiling 1.7.0 on redhat el4 64bit

[michael_rytting]

And of course I forget to actually attach the file.

From: Mark Phippard [mailto:markp...@gmail.com]
Sent: Tuesday, August 09, 2011 2:56 PM
To: RYTTING,MICHAEL (A-ColSprings,ex1); Subversion Development
Cc: users@subversion.apache.org
Subject: Re: Problems compiling 1.7.0 on redhat el4 64bit


On Tue, Aug 9, 2011 at 4:49 PM, 
mailto:michael_rytt...@agilent.com>> wrote:

Ok, I've tracked down which revision caused the problem.  It happened in rev 
1104160.  Stefan2 made a change to utf.c to speed up UTF8 conversion.  Ever 
since this change went in I am seeing subversion crash when I compile on 64bit 
el4.

Just for kicks and giggles I updated to the HEAD revision of 
http://svn.apache.org/repos/asf/subversion/trunk.  Did a reverse merge on rev 
1104160, cleaned up the conflicts and built it and viola!  No more crashing 
when built on 64bit el4.

Thanks.  Moving the thread to dev@

Can someone look at this and possibly fix it?

If you are not on users@, and to summarize this thread.  SVN builds fine on 
RHEL 4 64-bit but crashes immediately on any command, even svn help.  I saw the 
same problem with our builds.

--
Thanks

Mark Phippard
http://markphip.blogspot.com/


patch
Description: patch

RE: Problems compiling 1.7.0 on redhat el4 64bit

[michael_rytting]

I am attaching a stack trace with symbols enabled.

I'm using gcc.  The default in the makefile.

I am using the apr that you get from the get-deps.sh script. APR_HAS_THREADS is 
defined.

If I disable optimizations by doing "make CFLAGS=-O0" the program no longer 
crashes.

-Original Message-
From: Philip Martin [mailto:philip.mar...@wandisco.com] 
Sent: Tuesday, August 09, 2011 3:34 PM
To: Mark Phippard
Cc: RYTTING,MICHAEL (A-ColSprings,ex1); Subversion Development; 
users@subversion.apache.org
Subject: Re: Problems compiling 1.7.0 on redhat el4 64bit

Mark Phippard  writes:

> On Tue, Aug 9, 2011 at 4:49 PM,  wrote:
>
>> ** **
>>
>> Ok, I’ve tracked down which revision caused the problem.  It happened 
>> in rev 1104160.  Stefan2 made a change to utf.c to speed up UTF8 conversion.
>> Ever since this change went in I am seeing subversion crash when I 
>> compile on 64bit el4.
>>
>> ** **
>>
>> Just for kicks and giggles I updated to the HEAD revision of 
>> http://svn.apache.org/repos/asf/subversion/trunk.  Did a reverse 
>> merge on rev 1104160, cleaned up the conflicts and built it and 
>> viola!  No more crashing when built on 64bit el4.
>>
>
> Thanks.  Moving the thread to dev@
>
> Can someone look at this and possibly fix it?
>
> If you are not on users@, and to summarize this thread.  SVN builds 
> fine on RHEL 4 64-bit but crashes immediately on any command, even svn 
> help.  I saw the same problem with our builds.

It could be a compiler/optimiser bug.  Which compiler are you using?
Can you provide a stack trace of the crash (with debug symbols please)?
Does the crash go away if you compile utf.c without optimisation?  Which 
version of APR are you using?  Is APR_HAS_THREADS defined in apr.h?

--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com


stacktrace
Description: stacktrace

Re: Problems compiling 1.7.0 on redhat el4 64bit

[michael_rytting]

It is set to 1

On Aug 10, 2011, at 3:51 PM, "Philip Martin"  wrote:

>  writes:
> 
>> I am attaching a stack trace with symbols enabled.
> 
> Thanks!
> 
>> I'm using gcc.  The default in the makefile.
> 
> I think RHEL may have come with two different gcc, a 3 series and a 4
> series.  What version does 'gcc -v' show?
> 
>> I am using the apr that you get from the get-deps.sh
>> script. APR_HAS_THREADS is defined.
> 
> Just to check, APR_HAS_THREADS is defined to be 1?
> 
> It would be good to confirm that at runtime you really are using the
> version of libapr you compiled, run ldd on the svn executable.
> 
>> If I disable optimizations by doing "make CFLAGS=-O0" the program no
>> longer crashes.
> 
> That suggests it could be a compiler bug.
> 
>> #0  apr_xlate_conv_buffer (convset=0x2d6674752d6e7673, 
> 
> convset looks to be corrupt, that value is way bigger than the other
> pointer values.  It looks like ASCII, "-ftu-nvs", but that probably
> just means it's random.
> 
>>inbuf=0x7fbe8a "/cos/home/ryttingm", inbytes_left=0x7fb038, 
>>outbuf=0x54d5a8 "", outbytes_left=0x7fb030) at xlate/xlate.c:339
>> 339if (convset->ich != (iconv_t)-1) {
>> (gdb) 
>> Thread 1 (process 14949):
>> #0  apr_xlate_conv_buffer (convset=0x2d6674752d6e7673, 
>>inbuf=0x7fbe8a "/cos/home/ryttingm", inbytes_left=0x7fb038, 
>>outbuf=0x54d5a8 "", outbytes_left=0x7fb030) at xlate/xlate.c:339
>>status = 0
>> #1  0x002a965a4b95 in convert_to_stringbuf (node=0x2a965bcc30, 
> 
> node looks to be corrupt as well, that looks like a code value.
> 
> Looking at the rest of the stack trace I would say this is the first
> call to a utf8 conversion function that would have invoked
> get_xlate_handle_node and I suspect it is that function that is going
> wrong.  At a guess something to do with the use of atomic_swap, which
> makes it important to confirm the value of APR_HAS_THREADS.
> 
> -- 
> uberSVN: Apache Subversion Made Easy
> http://www.uberSVN.com

RE: Problems compiling 1.7.0 on redhat el4 64bit

[michael_rytting]

I tried running -O2 and -O1, both were crashing as well.  Only completely 
disabling optimizations will cause the segfault to stop.

-Original Message-
From: David Chapman [mailto:dcchap...@acm.org] 
Sent: Wednesday, August 10, 2011 5:39 PM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: philip.mar...@wandisco.com; markp...@gmail.com; d...@subversion.apache.org; 
users@subversion.apache.org
Subject: Re: Problems compiling 1.7.0 on redhat el4 64bit

On 8/10/2011 4:12 PM, michael_rytt...@agilent.com wrote:
> It is set to 1
>
> On Aug 10, 2011, at 3:51 PM, "Philip Martin"  
> wrote:
>
>>   writes:
>>
>>> If I disable optimizations by doing "make CFLAGS=-O0" the program no 
>>> longer crashes.
>> That suggests it could be a compiler bug.

Sorry to jump in so late, but I had significant problems with optimization for 
64-bit executables under CentOS 4.x.  If you can't upgrade, I suggest dialing 
down the optimization flags.  I had to drop from -O3 to -O2 so that Crypto++ 
5.5.2 would run, for example.  At some point I could drop support for 
RHEL/CentOS 4.x from my client's products and try going back to -O3, but the 
required testing hasn't been real high on the priority list.

Assuming the SVN configuration scripts can distinguish between the releases, I 
suspect that changing the default optimization flags for RHEL 4.x will be 
simpler than trying to fix the new UTF-8 code.  Have you tried intermediate 
levels of optimization?  We went to production with -O2, so it's not as if all 
64-bit optimizations were broken.

-- 
 David Chapman dcchap...@acm.org
 Chapman Consulting -- San Jose, CA

RE: Problems compiling 1.7.0 on redhat el4 64bit

[michael_rytting]

> > I'm using gcc.  The default in the makefile.
> 
> I think RHEL may have come with two different gcc, a 3 series and a 4 series. 
>  What version does 'gcc -v' show?

3.4.6

> It would be good to confirm that at runtime you really are using the version 
> of libapr you compiled, run ldd on the svn executable.

If it wasn't, changing the optimization level shouldn't have made a difference. 
 That being said, ldd reports that I am using the correct version.

libapr-1.so.0 => /cos/home/ryttingm/local/lib/libapr-1.so.0 (0x002a967b5000)

-Mike

RE: Problems compiling 1.7.0 on redhat el4 64bit

[michael_rytting]

No luck with that option, still crashing.

-Original Message-
From: Philip Martin [mailto:philip.mar...@wandisco.com] 
Sent: Wednesday, August 10, 2011 4:20 PM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: markp...@gmail.com; d...@subversion.apache.org; users@subversion.apache.org
Subject: Re: Problems compiling 1.7.0 on redhat el4 64bit

Philip Martin  writes:

>  writes:
>
>> If I disable optimizations by doing "make CFLAGS=-O0" the program no 
>> longer crashes.
>
> That suggests it could be a compiler bug.

The other flag you could try is to enable optimisation but to use 
-fno-strict-aliasing.

--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com

RE: Problems compiling 1.7.0 on redhat el4 64bit

[michael_rytting]

No luck, still crashing.

-Original Message-
From: Philip Martin [mailto:philip.mar...@wandisco.com] 
Sent: Monday, August 22, 2011 10:23 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: markp...@gmail.com; d...@subversion.apache.org; users@subversion.apache.org
Subject: Re: Problems compiling 1.7.0 on redhat el4 64bit

 writes:

> It is set to 1
>> 
>> Looking at the rest of the stack trace I would say this is the first 
>> call to a utf8 conversion function that would have invoked 
>> get_xlate_handle_node and I suspect it is that function that is going 
>> wrong.  At a guess something to do with the use of atomic_swap, which 
>> makes it important to confirm the value of APR_HAS_THREADS.

I wonder if it is connected to this APR bug:

https://issues.apache.org/bugzilla/show_bug.cgi?id=50731

In subversion/libsvn_subr/utf.c we declare

static volatile void *xlat_ntou_static_handle = NULL; static volatile void 
*xlat_uton_static_handle = NULL;

that is "pointers to volatile data", as required by APR, but we really want 
"volatile pointers to data".  Perhaps we should do something similar to 
svn_cache_config.c.  Would you try this patch built with optimisations enabled:

Index: subversion/libsvn_subr/utf.c
===
--- subversion/libsvn_subr/utf.c(revision 1160136)
+++ subversion/libsvn_subr/utf.c(working copy)
@@ -90,8 +90,8 @@
  * using atomic xchange ops, i.e. without further thread synchronization.
  * If the respective item is NULL, fallback to hash lookup.
  */
-static volatile void *xlat_ntou_static_handle = NULL; -static volatile void 
*xlat_uton_static_handle = NULL;
+static void * volatile xlat_ntou_static_handle = NULL; static void * 
+volatile xlat_uton_static_handle = NULL;
 
 /* Clean up the xlate handle cache. */
 static apr_status_t
@@ -182,11 +182,11 @@
  * the caller.
  */
 static APR_INLINE void*
-atomic_swap(volatile void **mem, void *new_value)
+atomic_swap(void * volatile * mem, void *new_value)
 {
 #if APR_HAS_THREADS
 #if APR_VERSION_AT_LEAST(1,3,0)
-   return apr_atomic_xchgptr(mem, new_value);
+   return apr_atomic_xchgptr((volatile void **)mem, new_value);
 #else
/* old APRs don't support atomic swaps. Simply return the
 * input to the caller for further proccessing. */


--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com

RE: Problems compiling 1.7.0 on redhat el4 64bit

[michael_rytting]

I did.  It didn't help.

-Mike

From: Mark Phippard [mailto:markp...@gmail.com]
Sent: Monday, August 22, 2011 10:48 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: philip.mar...@wandisco.com; d...@subversion.apache.org; 
users@subversion.apache.org
Subject: Re: Problems compiling 1.7.0 on redhat el4 64bit

Michael, did you ever get a chance trying the -fno-strict-aliasing flag rather 
than removing all optimizations?



On Mon, Aug 22, 2011 at 12:43 PM, 
mailto:michael_rytt...@agilent.com>> wrote:
No luck, still crashing.

-Original Message-
From: Philip Martin 
[mailto:philip.mar...@wandisco.com]
Sent: Monday, August 22, 2011 10:23 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: markp...@gmail.com; 
d...@subversion.apache.org; 
users@subversion.apache.org
Subject: Re: Problems compiling 1.7.0 on redhat el4 64bit
mailto:michael_rytt...@agilent.com>> writes:

> It is set to 1
>>
>> Looking at the rest of the stack trace I would say this is the first
>> call to a utf8 conversion function that would have invoked
>> get_xlate_handle_node and I suspect it is that function that is going
>> wrong.  At a guess something to do with the use of atomic_swap, which
>> makes it important to confirm the value of APR_HAS_THREADS.

I wonder if it is connected to this APR bug:

https://issues.apache.org/bugzilla/show_bug.cgi?id=50731

In subversion/libsvn_subr/utf.c we declare

static volatile void *xlat_ntou_static_handle = NULL; static volatile void 
*xlat_uton_static_handle = NULL;

that is "pointers to volatile data", as required by APR, but we really want 
"volatile pointers to data".  Perhaps we should do something similar to 
svn_cache_config.c.  Would you try this patch built with optimisations enabled:

Index: subversion/libsvn_subr/utf.c
===
--- subversion/libsvn_subr/utf.c(revision 1160136)
+++ subversion/libsvn_subr/utf.c(working copy)
@@ -90,8 +90,8 @@
 * using atomic xchange ops, i.e. without further thread synchronization.
 * If the respective item is NULL, fallback to hash lookup.
 */
-static volatile void *xlat_ntou_static_handle = NULL; -static volatile void 
*xlat_uton_static_handle = NULL;
+static void * volatile xlat_ntou_static_handle = NULL; static void *
+volatile xlat_uton_static_handle = NULL;

 /* Clean up the xlate handle cache. */
 static apr_status_t
@@ -182,11 +182,11 @@
 * the caller.
 */
 static APR_INLINE void*
-atomic_swap(volatile void **mem, void *new_value)
+atomic_swap(void * volatile * mem, void *new_value)
 {
 #if APR_HAS_THREADS
 #if APR_VERSION_AT_LEAST(1,3,0)
-   return apr_atomic_xchgptr(mem, new_value);
+   return apr_atomic_xchgptr((volatile void **)mem, new_value);
 #else
   /* old APRs don't support atomic swaps. Simply return the
* input to the caller for further proccessing. */


--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com



--
Thanks

Mark Phippard
http://markphip.blogspot.com/

RE: Really lousy performance with svn info --depth infinity

[michael_rytting]

Very well, we'll move it to users.  I just figured it was a bug in 1.7 since 
the performance of the svn info -depth infinity was SIGNIFICANTLY faster in 
1.6.x.  Currently it takes 1 minute with 1.7 and less than a second with 1.6.x.

-Mike

From: Bert Huijben [mailto:b...@qqmail.nl]
Sent: Thursday, September 01, 2011 9:43 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1); d...@subversion.apache.org
Subject: RE: Really lousy performance with svn info --depth infinity

These operations do very different things. The first reads your working copy to 
obtain a lot of details (unless you pass a revision; in that case it looks at 
the repository)
The second asks the repository what files there are below the current directory 
in the repository (using the revision of the current directory).

In my usage (repository hosted via http:// on the other side of the world) the 
second is much slower... But if you have a local repository the opposite might 
be true.

I think this question belongs on users{_AT_}subversion.apache.org though, as it 
is about using Subversion and not on Subversion development.

Bert

From: michael_rytt...@agilent.com [mailto:michael_rytt...@agilent.com]
Sent: donderdag 1 september 2011 17:26
To: d...@subversion.apache.org
Subject: Really lousy performance with svn info --depth infinity

Hi guys,

Quick question.  I would expect the following 2 commands to perform equivalent 
functions.

svn info -depth infinity
svn ls -depth infinity | xargs svn info

This issue is that the first command takes 1 minute to run while the second 
command takes 6 seconds to run.  I am running 1.7.0-rc2.

-Mike

Apparent "svn rm" scaling problem in 1.7.x

[michael_rytting]

I am starting to see some very bad performance with "svn rm" compared to the 
1.6.x line of subversion.  I have a directory that is full of files.  If I go 
into the directory and run "svn rm *", it is significantly slower than running 
svn rm on the whole directory.  While the difference in time taken is 
significant, the speed is still relatively acceptable for small working copies.

"svn rm dir/*"   3.8s  1.7.1
"svn rm dir"0.173s 1.7.1
"svn rm dir/*"1.008s 1.6.17

In this case 1.6.17 is nearly 4 times faster.  My working copy is nfs mounted 
and has 200 nodes.

When the working copy size starts to get larger, deleting the same directory as 
above, but within a larger working copy and things really start to slow down.

"svn rm dir/*"   6m15s  1.7.1
"svn rm dir"8.5s  1.7.1
"svn rm dir/*"1.14s   1.6.17

This working copy is also nfs mounted but I'm up to 23948 nodes (doing sqlite3 
.svn/wc.db "select count (*) from nodes").  The directory that I am deleting 
has 49 files.  The largest is file is 2.5k.  I am doing the comparison of the 
same directory of the same repository.  In one case I am only doing a partial 
checkout to keep the working copy size down where the second test I am doing a 
complete checkout.

We can also see that subversion 1.6.17 scales very well for working copy size.




Michael Rytting
Agilent Technologies
michael_rytt...@agilent.com
719-590-3708

RE: Apparent "svn rm" scaling problem in 1.7.x

[michael_rytting]

I did an additional benchmark doing "svn rm dir/*" on a local directory instead 
of an nfs directory.  It runs in 10.4s.  Is going from 10.4s to 6m15s 
acceptable when using a working copy on nfs vs local?  I am fine with a certain 
amount of slowdown when using nfs.  But, I don't see this kind of degradation 
on nfs vs local for other operations.  Svn rm seems particularly susceptible.

-Original Message-
From: Stefan Sperling [mailto:s...@elego.de] 
Sent: Monday, October 31, 2011 9:45 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: users@subversion.apache.org
Subject: Re: Apparent "svn rm" scaling problem in 1.7.x

On Mon, Oct 31, 2011 at 09:09:21AM -0600, michael_rytt...@agilent.com wrote:
> I am starting to see some very bad performance with "svn rm" compared to the 
> 1.6.x line of subversion.  I have a directory that is full of files.  If I go 
> into the directory and run "svn rm *", it is significantly slower than 
> running svn rm on the whole directory.  While the difference in time taken is 
> significant, the speed is still relatively acceptable for small working 
> copies.
> 
> "svn rm dir/*"   3.8s  1.7.1
> "svn rm dir"0.173s 1.7.1
> "svn rm dir/*"1.008s 1.6.17
> 
> In this case 1.6.17 is nearly 4 times faster.  My working copy is nfs mounted 
> and has 200 nodes.
> 

What happens here is that each deletion target uses one sqlite transaction.
AFAIK creating sqlite transactions on NFS is expensive.

In the recursive case, only one transaction is used (for 'dir').
In the non-recursive case, the shell expands the target list, and svn ends up 
using one transaction per target.

To fix this problem, we could change the implementation so that one sqlite 
transaction is used even for multiple targets.

> When the working copy size starts to get larger, deleting the same directory 
> as above, but within a larger working copy and things really start to slow 
> down.
> 
> "svn rm dir/*"   6m15s  1.7.1
> "svn rm dir"8.5s  1.7.1
> "svn rm dir/*"1.14s   1.6.17
> 
> This working copy is also nfs mounted but I'm up to 23948 nodes (doing
> sqlite3 .svn/wc.db "select count (*) from nodes").  The directory that 
> I am deleting has 49 files.  The largest is file is 2.5k.  I am doing 
> the comparison of the same directory of the same repository.  In one 
> case I am only doing a partial checkout to keep the working copy size 
> down where the second test I am doing a complete checkout.
> 
> We can also see that subversion 1.6.17 scales very well for working copy size.

Yes, 1.6 performed better on NFS than 1.7 does.
This is because of how sqlite behaves on NFS filesystems.
But 1.7 performs and scale an awful lot better on local disk than 1.6 did, so 
this was a tradeoff we were willing to make.
I am not sure how much additional performance can be squeezed out of sqlite on 
NFS. It has been said that not much can be done about it.
But hints are welcome in case anyone has an idea.

RE: Apparent "svn rm" scaling problem in 1.7.x

[michael_rytting]

I'm always willing to try patches.

-Original Message-
From: Stefan Sperling [mailto:s...@elego.de] 
Sent: Tuesday, November 01, 2011 2:49 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: users@subversion.apache.org
Subject: Re: Apparent "svn rm" scaling problem in 1.7.x

On Mon, Oct 31, 2011 at 10:12:04AM -0600, michael_rytt...@agilent.com
wrote:
> I did an additional benchmark doing "svn rm dir/*" on a local 
> directory instead of an nfs directory.  It runs in 10.4s.  Is going 
> from 10.4s to 6m15s acceptable when using a working copy on nfs vs 
> local?  I am fine with a certain amount of slowdown when using nfs.
> But, I don't see this kind of degradation on nfs vs local for other 
> operations.  Svn rm seems particularly susceptible.
> 

I've taken a stab at fixing this, see
http://svn.apache.org/viewvc?view=revision&revision=1195873

Would you be able to test a patch against 1.7.x?
There are some conflicts when the above commit is merged into the 1.7.x branch, 
so I'll have to prepare a special patch for 1.7.x.

RE: Apparent "svn rm" scaling problem in 1.7.x

[michael_rytting]

Not much of an improvement.  "svn rm dir/*" now takes 2m6s vs 7s for "svn rm 
dir".  

As a side note, I really think there is fundamentally something wrong of the 
performance of "svn rm" with large working copies.  Here are some example times.

svn rm7s
svn add   0.126s
svn st   2s
svn blame  0.2s
svn lock   0.12s
svn unlock  0.103s
svn log 0.089s
svn revert   0.133s
svn info   0.074s

I'm assuming that all these commands are doing some form of sqlite database 
transactions, but the rm transaction, in particular, is very slow.  Even when 
using a local working copy, I am seeing large discrepancies in the time it 
takes to run "svn rm" vs most other svn commands.  It's just since the local 
working copy is faster overall, you are less likely to notice the large 
discrepancy in performance.


-Original Message-
From: Stefan Sperling [mailto:s...@elego.de] 
Sent: Tuesday, November 01, 2011 8:35 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: users@subversion.apache.org
Subject: Re: Apparent "svn rm" scaling problem in 1.7.x

On Tue, Nov 01, 2011 at 06:45:59AM -0600, michael_rytt...@agilent.com wrote:
> I'm always willing to try patches.

Great! Please give this patch a spin:

svn diff -c 1196018 
https://svn.apache.org/repos/asf/subversion/branches/1.7.x-r1195873

RE: Apparent "svn rm" scaling problem in 1.7.x

[michael_rytting]

LOL!  I love the env variable.

Here is some similar data for a local working copy.  These are all run with the 
env variable set.  Again, svn rm is significantly slower than all other 
operations.

svn rm   0.35s
svn st 0.105s
svn blame  0.041s
svn unlock 0.056s
svn lock  0.053s
svn log   0.036s
svn info 0.014s

-Original Message-
From: Stefan Sperling [mailto:s...@elego.de] 
Sent: Tuesday, November 01, 2011 11:00 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: users@subversion.apache.org
Subject: Re: Apparent "svn rm" scaling problem in 1.7.x

On Tue, Nov 01, 2011 at 10:38:07AM -0600, michael_rytt...@agilent.com wrote:
> Not much of an improvement.  "svn rm dir/*" now takes 2m6s vs 7s for "svn rm 
> dir".  

Before the patch, we had:

"svn rm dir/*"   6m15s  1.7.1
"svn rm dir"8.5s  1.7.1
"svn rm dir/*"1.14s   1.6.17

So this patch cut about 4 minutes of runtime, which is somwhat significant but 
definitely not enough. But it's a step in the right direction.

> As a side note, I really think there is fundamentally something wrong of the 
> performance of "svn rm" with large working copies.  Here are some example 
> times.
> 
> svn rm7s
> svn add   0.126s
> svn st   2s
> svn blame  0.2s
> svn lock   0.12s
> svn unlock  0.103s
> svn log 0.089s
> svn revert   0.133s
> svn info   0.074s
> 
> I'm assuming that all these commands are doing some form of sqlite 
> database transactions, but the rm transaction, in particular, is very 
> slow.  Even when using a local working copy, I am seeing large 
> discrepancies in the time it takes to run "svn rm" vs most other svn 
> commands.  It's just since the local working copy is faster overall, 
> you are less likely to notice the large discrepancy in performance.

Yes, that looks bad. There might be a linear DB table scan involved in 'svn rm' 
that becomes noticable on large WCs.
Do you see this difference only on NFS or also on local disk?

How are you measuring the time?
Note that most commands take at least one second because Subversion waits for 
one second for filesystem timestamps to update after some operations. To cut 
this deliberate delay out of the equation, do this:
export SVN_I_LOVE_CORRUPTED_WORKING_COPIES_SO_DISABLE_SLEEP_FOR_TIMESTAMPS=yes

RE: Apparent "svn rm" scaling problem in 1.7.x

[michael_rytting]

Oh and to measure the time, I'm using a stopwatch and synchronizing the start 
of the command with pressing go on my stop watch.  I then push stop when the 
command completes.



P.S. Actually I'm using the linux time command :)  i.e. "time svn st "

-Original Message-
From: Stefan Sperling [mailto:s...@elego.de] 
Sent: Tuesday, November 01, 2011 11:00 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: users@subversion.apache.org
Subject: Re: Apparent "svn rm" scaling problem in 1.7.x

On Tue, Nov 01, 2011 at 10:38:07AM -0600, michael_rytt...@agilent.com wrote:
> Not much of an improvement.  "svn rm dir/*" now takes 2m6s vs 7s for "svn rm 
> dir".  

Before the patch, we had:

"svn rm dir/*"   6m15s  1.7.1
"svn rm dir"8.5s  1.7.1
"svn rm dir/*"1.14s   1.6.17

So this patch cut about 4 minutes of runtime, which is somwhat significant but 
definitely not enough. But it's a step in the right direction.

> As a side note, I really think there is fundamentally something wrong of the 
> performance of "svn rm" with large working copies.  Here are some example 
> times.
> 
> svn rm7s
> svn add   0.126s
> svn st   2s
> svn blame  0.2s
> svn lock   0.12s
> svn unlock  0.103s
> svn log 0.089s
> svn revert   0.133s
> svn info   0.074s
> 
> I'm assuming that all these commands are doing some form of sqlite 
> database transactions, but the rm transaction, in particular, is very 
> slow.  Even when using a local working copy, I am seeing large 
> discrepancies in the time it takes to run "svn rm" vs most other svn 
> commands.  It's just since the local working copy is faster overall, 
> you are less likely to notice the large discrepancy in performance.

Yes, that looks bad. There might be a linear DB table scan involved in 'svn rm' 
that becomes noticable on large WCs.
Do you see this difference only on NFS or also on local disk?

How are you measuring the time?
Note that most commands take at least one second because Subversion waits for 
one second for filesystem timestamps to update after some operations. To cut 
this deliberate delay out of the equation, do this:
export SVN_I_LOVE_CORRUPTED_WORKING_COPIES_SO_DISABLE_SLEEP_FOR_TIMESTAMPS=yes

RE: Apparent "svn rm" scaling problem in 1.7.x

[michael_rytting]

Perhaps I wasn't clear, the second set of runs where with a local working copy 
instead of an nfs mounted working copy.

From: Mark Phippard [mailto:markp...@gmail.com]
Sent: Tuesday, November 01, 2011 11:18 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: s...@elego.de; users@subversion.apache.org
Subject: Re: Apparent "svn rm" scaling problem in 1.7.x

On Tue, Nov 1, 2011 at 1:10 PM, 
mailto:michael_rytt...@agilent.com>> wrote:
LOL!  I love the env variable.

Here is some similar data for a local working copy.  These are all run with the 
env variable set.  Again, svn rm is significantly slower than all other 
operations.

svn rm   0.35s
svn st 0.105s
svn blame  0.041s
svn unlock 0.056s
svn lock  0.053s
svn log   0.036s
svn info 0.014s

But look how much it improved compared to how much the others improved?

svn rm7s
svn add   0.126s
svn st   2s
svn blame  0.2s
svn lock   0.12s
svn unlock  0.103s
svn log 0.089s
svn revert   0.133s
svn info   0.074s

Many of these commands are not even impacted by that variable.  That said, I do 
not get how this envvar can shave 7 seconds off the operation when it usually 
only sleeps for a second.

--
Thanks

Mark Phippard
http://markphip.blogspot.com/

RE: Apparent "svn rm" scaling problem in 1.7.x

[michael_rytting]

It's just one directory that has 49 files in it.

-Original Message-
From: Stefan Sperling [mailto:s...@elego.de] 
Sent: Tuesday, November 01, 2011 11:22 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: users@subversion.apache.org
Subject: Re: Apparent "svn rm" scaling problem in 1.7.x

On Tue, Nov 01, 2011 at 11:10:44AM -0600, michael_rytt...@agilent.com wrote:
> LOL!  I love the env variable.
> 
> Here is some similar data for a local working copy.  These are all run with 
> the env variable set.  Again, svn rm is significantly slower than all other 
> operations.
> 
> svn rm   0.35s
> svn st 0.105s
> svn blame  0.041s
> svn unlock 0.056s
> svn lock  0.053s
> svn log   0.036s
> svn info 0.014s

What effect does the env var have on 'svn rm dir/*'?
How many files and directories are you actually deleting via 'svn rm dir/*'?

RE: Apparent "svn rm" scaling problem in 1.7.x

[michael_rytting]

I'd have to do some research to get the options.  It's a proprietary filesystem.

That being said, I understand that nfs mounted working copies will degrade my 
performance.  I really think this is a more fundamental performance issue with 
svn rm that gets exacerbated with slow performance over nfs.

From: Tony Sweeney [mailto:tswee...@omnifone.com
Sent: Tuesday, November 01, 2011 11:32 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1); markp...@gmail.com
Cc: s...@elego.de; users@subversion.apache.org
Subject: RE: Apparent "svn rm" scaling problem in 1.7.x



From: michael_rytt...@agilent.com 
[mailto:michael_rytt...@agilent.com]
Sent: 01 November 2011 17:19
To: markp...@gmail.com
Cc: s...@elego.de; 
users@subversion.apache.org
Subject: RE: Apparent "svn rm" scaling problem in 1.7.x
Perhaps I wasn't clear, the second set of runs where with a local working copy 
instead of an nfs mounted working copy.


What are your NFS mount options?


From: Mark Phippard 
[mailto:markp...@gmail.com]
Sent: Tuesday, November 01, 2011 11:18 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: s...@elego.de; 
users@subversion.apache.org
Subject: Re: Apparent "svn rm" scaling problem in 1.7.x

On Tue, Nov 1, 2011 at 1:10 PM, 
mailto:michael_rytt...@agilent.com>> wrote:
LOL!  I love the env variable.

Here is some similar data for a local working copy.  These are all run with the 
env variable set.  Again, svn rm is significantly slower than all other 
operations.

svn rm   0.35s
svn st 0.105s
svn blame  0.041s
svn unlock 0.056s
svn lock  0.053s
svn log   0.036s
svn info 0.014s

But look how much it improved compared to how much the others improved?

svn rm7s
svn add   0.126s
svn st   2s
svn blame  0.2s
svn lock   0.12s
svn unlock  0.103s
svn log 0.089s
svn revert   0.133s
svn info   0.074s

Many of these commands are not even impacted by that variable.  That said, I do 
not get how this envvar can shave 7 seconds off the operation when it usually 
only sleeps for a second.

--
Thanks

Mark Phippard
http://markphip.blogspot.com/

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
__


No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.1834 / Virus Database: 2092/4589 - Release Date: 11/01/11

__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
__

RE: Apparent "svn rm" scaling problem in 1.7.x

[michael_rytting]

Benchmarking with 49 files is taking too long.  Here are some benchmarks of 
trying to delete a directory with 5 files.  I am using approximations because I 
am seeing variations in each run due to network traffic.

"svn rm dir/*" 1.6.17 ~0.15s
"svn rm dir" 1.7.1~10s
"svn rm dir/*" 1.7.1~50s 
"svn rm dir/*" 1.7.1.p1 ~30s (single transaction)
"svn rm dir/*" 1.7.1.p2 ~25s (sqlite change)
"svn rm dir/*" 1.7.1.p3 ~30s (c code change)

We can continue to benchmark deletes on multiple files, but I still think the 
culprit is the fundamental svn rm command is too slow.


-Original Message-
From: Stefan Sperling [mailto:s...@elego.de] 
Sent: Tuesday, November 01, 2011 12:30 PM
To: RYTTING,MICHAEL (A-ColSprings,ex1); users@subversion.apache.org
Subject: Re: Apparent "svn rm" scaling problem in 1.7.x

On Tue, Nov 01, 2011 at 07:00:57PM +0100, Stefan Sperling wrote:
> On Tue, Nov 01, 2011 at 11:31:35AM -0600, michael_rytt...@agilent.com wrote:
> > It's just one directory that has 49 files in it.
> 
> Good. Please try this patch in addition to the other one.
> It makes 'svn rm dir/*' with 49 files go down from about 4.20 seconds 
> to about 1.50 seconds for me (local disk).
> 
> Note that I am not going to commit this as is.
> It just tests whether the overhead of sorting paths in sqlite matters 
> much on NFS.

I went ahead and implemented sorting the deleted items list in C.
The following gives you a patch that applies cleanly on top of my previous 
1.7.x patch:

 svn diff -c1196191 https://svn.apache.org/repos/asf/subversion/trunk

Can you try that? Thanks.

Another performance issue with subversion 1.7.x and nfs mounted working copies

[michael_rytting]

In addition to slow "svn rm" commands we are seeing some pretty severe 
slowdowns for "svn ci" as well.  For example, here is an interesting situation. 
 I want to checkin a single file in my repository.  If I run "svn ci" from the 
root of my working copy it takes 1m11s to complete.  However if I instead run

cd 
svn st
cd subdir
svn ci
cd 
svn up

That whole set of commands takes 16s to run.  The actual change I am committing 
is adding or deleting a single line of a very small script.  I am trying to 
simulate what a top level "svn ci" script would do with the commands I have 
above.

Here are the stats for my working copy. As generated by my checkSVNSize.sh 
script.

digital/mixedsignal/analog_model/analog_model_vams/netlist/hh1a|210
5393
0
0

My checkSVNSize.sh script runs the following queries

sqlite3 .svn/wc.db "select parent_relpath, count(*) AS n from nodes group by 
parent_relpath order by n desc limit 1"
sqlite3 .svn/wc.db "select count (*) from nodes"
sqlite3 .svn/wc.db "select count (*) from nodes where op_depth > 0"
sqlite3 .svn/wc.db "select count (*) from actual_node"

Michael Rytting
Agilent Technologies
michael_rytt...@agilent.com
719-590-3708