Hi Andy, Thank you for your prompt reply.
I found the performance problem was caused by 21406 files inside a single
folder, total size of this folder
is 255M. (sorry for this bad layout, but this is not in my control)
I've done a little more test over this folder today, and found if we
launch svnserve with "--memory-cache-size=8192 --cache-txdeltas yes
--cache-fulltexts yes", the performance is much better.
Here is my test environment:
Server side:
svn version 1.7.7 (r1393599)
model name : Intel(R) Xeon(R) CPU L5410 @ 2.33GHz
/dev/sdb on /export type xfs (rw)
total used free shared buffers cached
Mem: 16467548 15698252 769296 0 256 13319084
Client side:
svn version 1.6.11 (r934486)
model name : Intel(R) Xeon(R) CPU E5520 @ 2.27GHz
tmpfs on /root/tmp type tmpfs (rw,size=4g)
total used free shared buffers cached
Mem: 12297388 9509928 2787460 0 663260 7344808
svn server is running on a SSD disk, and I'm checking out to tmpfs,
free memory is enough, so we are CPU-bound in these tests.
Case 1:
If we launch svnserve with default option, it takes 23 mins to checkout
this folder:
/usr/bin/time svn co svn://10.68.xx.xx/data/task
Checked out revision 89151.
185.41user 261.34system 23:00.37elapsed 32%CPU (0avgtext+0avgdata
449760maxresident)k
0inputs+0outputs (0major+28236minor)pagefaults 0swaps
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29149 root 20 0 162m 38m 1292 R 100.0 0.2 0:24.98 svnserve -d -r
/export/svn/repositories/
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21920 root 20 0 206m 7348 2908 S 3.4 0.1 0:01.46 svn
In this case bottleneck is server CPU.
Case 2:
With cache options, it takes only 8 mins.
/usr/bin/time svn co svn://10.68.xx.xx/data/task
Checked out revision 89151.
181.65user 253.39system 8:07.51elapsed 89%CPU (0avgtext+0avgdata
449744maxresident)k
0inputs+0outputs (0major+28235minor)pagefaults 0swaps
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29128 root 20 0 8312m 548m 1300 S 0.7 3.4 0:02.07 svnserve -d -r
/export/svn/repositories/ --memory-cache-size=8192 --cache-txdeltas yes
--cache-fulltexts yes
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21793 root 20 0 268m 59m 2908 R 89.1 0.5 3:36.61 svn
CPU usage on server side decreased dramatically when cache options is on,
memory cost is acceptable in my case
, Client CPU slowdown the checkout process this time.
I also tested to checkout this folder on server side machine, with svn
1.7.7, to tmpfs,
it takes only 16 seconds, this significant improvement must be attribute to
new working copy library (WC-NG) in svn 1.7?
Thank SVN dev team for your continuous efforts in optimization.
BTW. it's still painfully slow to checkout this repository to hard disk,
if disk IO will be optimized in future version of svn , it will be a good
news.
2012/10/12 Andy Levy <andy.l...@gmail.com>
> On Fri, Oct 12, 2012 at 11:06 AM, Wei Bai <baiwei...@gmail.com> wrote:
> > Hi, Thanks for replying.
> >
> >
> >
> > I'm using svnserve 1.7.5 on a Dell R610 server ( Xeon E5620*2/16GB)
> >
> > The server is running CentOS 5.5, SSD disk is used to increase IO
> > performance.
> >
> > The repository is very large: 100K+ files, 100K+ revisions, total size of
> > the work copy is about 1.5G.
> >
> >
> >
> > When I what to check out a new work copy on another linux machine, I
> found
> > it's painfully
> >
> > slow, it takes about 2 hours. Can anybody tell me if this speed is
> normal?
> >
> > I noticed when the svn checkout command is running on client side, there
> is
> > a svnserve process with 100% CPU usage on server side,
> >
> > does this means the concurrent performance of svn is very bad for
> checkout
> > command?
>
> Not necessarily. In my environment, I typically hit I/O constraints
> limiting my checkout performance long before I hit CPU constraints.
>
> If you're performing a checkout and seeing 100% CPU utilization on the
> server, you're CPU-bound. With a faster processor, you may see
> improved checkout performance. With an SSD (assuming the repository is
> on the SSD), you're probably not I/O bound.
>
> How's your memory utilization while checking out? Is it possible that
> you're memory constrained and swapping out to disk?
>
> Because of how Subversion stores revisions, it must look at past
> revisions to construct the revision you're requesting. Depending upon
> a number of factors, this may become CPU and/or memory bound. It's a
> trade-off of performance vs. storage efficiency, and there is no one
> optimal setting for everyone's repository. The developers have
> selected a value which works well enough for most people.
>
> > And this svnserve process will not disappear immediately if I kill the
> > client side svn process, it will run for a long time with 100% CPU usage,
> > this might be a problem?
>
> It depends on how you're killing the client. If the server doesn't
> realize that the client has terminated, it'll keep trying to perform
> the checkout.
>
> > Could anyone give some advice on how to optimize the performance of svn
> on a
> > large repository?
>
> First you need to determine your limiting factor. In this case, it
> looks like CPU (or memory, once you look into that).
>
