Dear SVN Buddies,
(directed to this mailing list from: http://subversion.tigris.org/issue-tracker.html) At our company we're using Subversion for some years now but are coping with a strange server instability for some months now. After a random period, the Subversion server stops responding and users cannot update/commit/navigate the Subversion repository. Restarting the Subversion service seems to resolve the problem. I'd like to explain the problem as much as possible below. Hopefully you can help me out here, or at least guide me to contacts that can help me with analyzing the problem. I did a lot of Google searched on the errors I received in the logfiles, but wasn't able to find any topics that could really help me out. Thanks in advance! Configuration - Subversion Server* Subversion Distribution CollabNet Subversion Edge (1.2) Software version 1.2.1-1406.49 Subversion version 1.6.12-1406.49 Operating System Windows Server 2003 SP2, 2Gb (*This is a virtual server, running on VMWare 8.) Configuration - Subversion Clients We're running mixed clients. Our company policy is to use SmartSVN as Subversion client, however, some people may be using different clients, and we're aware of at least people running Versions (Mac OS X) and TortoiseSVN (Windows). As we're also developing on Linux, command-line Subversion builds may be used as well. Subversion clients used mostly are 1.6 and 1.7 (we officially introduced SmartSVN 7 a few months ago after purchasing a bunch of licenses) but I recently found some project teams still running 1.5. Problem analysis Now it's getting interesting. I studied some logfiles in: \csvn\data\logs (That's where CollabNet Subverion stores the logfiles) error_[date].log: After the service failed to respond, I restarted the Subversion server and started analyzing this file. This file grew approx. 80Mb in less than 20 minutes. But I also encountered older error files that exceeded 200Mb, the largest being 3Gb. [Tue Nov 06 14:36:27 2012] [error] (OS 6)The handle is invalid. : Child 3880: Encountered too many errors accepting client connections. Possible causes: Unknown. Try using the Win32DisableAcceptEx directive. [Tue Nov 06 14:36:27 2012] [error] (OS 6)The handle is invalid. : Child 3880: Encountered too many errors accepting client connections. Possible causes: Unknown. Try using the Win32DisableAcceptEx directive. Some other logfile (a few days earlier) [Wed Oct 31 13:34:02 2012] [error] (OS 6)The handle is invalid. : Child 212: Encountered too many errors accepting client connections. Possible causes: Unknown. Try using the Win32DisableAcceptEx directive. [Wed Oct 31 13:34:02 2012] [error] (OS 6)The handle is invalid. : Child 212: Encountered too many errors accepting client connections. Possible causes: Unknown. Try using the Win32DisableAcceptEx directive. During this unresponsive periode, it writes approx. 200-300 lines per second to the error log. A lot. Before the lockup, some other errors can be found: [Wed Oct 31 09:43:47 2012] [error] [client 172.17.64.54] Provider encountered an error while streaming a REPORT response. [500, #0] [Wed Oct 31 09:43:47 2012] [error] [client 172.17.64.54] A failure occurred while driving the update report editor [500, #620018] [Wed Oct 31 09:43:47 2012] [error] [client 172.17.64.54] Provider encountered an error while streaming a REPORT response. [500, #0] [Wed Oct 31 09:43:47 2012] [error] [client 172.17.64.54] A failure occurred while driving the update report editor [500, #620018] These are written to the error log as well, approx. 25-30 lines per second, and it looks like it keeps writing to the logfile until it has received too many errors. >From what I know and have read, this may have to do with the DAV SVN module for Apache. This may be an error in the module, but I also found articles that say that these errors can still mean something went wrong within the Subversion part, who knows. I don't. :D access_[date].log: Interestingly, the access logfile proves that the client request that generates this error is repeating every second. Two different access logfiles: 172.17.96.37 - - [06/Nov/2012:10:53:15 +0100] "OPTIONS /svn/priva_ux/Priva/Projects HTTP/1.1" 401 401 172.17.96.37 - user1 [06/Nov/2012:10:53:15 +0100] "OPTIONS /svn/priva_ux/Priva/Projects HTTP/1.1" 200 - 172.17.96.37 - user1 [06/Nov/2012:10:53:15 +0100] "PROPFIND /svn/priva_ux/Priva/Projects HTTP/1.1" 207 712 172.17.96.37 - user1 [06/Nov/2012:10:53:15 +0100] "PROPFIND /svn/priva_ux/!svn/vcc/default HTTP/1.1" 207 411 172.17.96.37 - user1 [06/Nov/2012:10:53:15 +0100] "PROPFIND /svn/priva_ux/!svn/bln/230677 HTTP/1.1" 207 428 172.17.96.37 - user1 [06/Nov/2012:10:53:15 +0100] "PROPFIND /svn/priva_ux/!svn/bc/230677/Priva/Projects HTTP/1.1" 207 346 172.17.96.37 - user1 [06/Nov/2012:10:53:15 +0100] "PROPFIND /svn/priva_ux/Priva/Projects HTTP/1.1" 207 712 172.17.96.37 - user1 [06/Nov/2012:10:53:15 +0100] "REPORT /svn/priva_ux/!svn/vcc/default HTTP/1.1" 200 16041 172.17.64.17 - - [06/Nov/2012:17:24:18 +0100] "OPTIONS /svn/tcx_platform/trunk HTTP/1.1" 401 401 172.17.64.17 - user2 [06/Nov/2012:17:24:18 +0100] "OPTIONS /svn/tcx_platform/trunk HTTP/1.1" 200 - 172.17.64.17 - user2 [06/Nov/2012:17:24:18 +0100] "PROPFIND /svn/tcx_platform/trunk HTTP/1.1" 207 702 172.17.64.17 - user2 [06/Nov/2012:17:24:18 +0100] "PROPFIND /svn/tcx_platform/!svn/vcc/default HTTP/1.1" 207 418 172.17.64.17 - user2 [06/Nov/2012:17:24:18 +0100] "PROPFIND /svn/tcx_platform/!svn/bln/23230 HTTP/1.1" 207 434 172.17.64.17 - user2 [06/Nov/2012:17:24:18 +0100] "PROPFIND /svn/tcx_platform/!svn/bc/23230/trunk HTTP/1.1" 207 340 172.17.64.17 - user2 [06/Nov/2012:17:24:18 +0100] "PROPFIND /svn/tcx_platform/trunk HTTP/1.1" 207 702 172.17.64.17 - user2 [06/Nov/2012:17:24:18 +0100] "REPORT /svn/tcx_platform/!svn/vcc/default HTTP/1.1" 200 16021 Those requests seem to generate an error on the server. The clients keep repeating this eternally, until the server can't take it anymore. What did they do? What are they using ... I was able to back trace the IP addresses to the corresponding users. The users I spoke told me that they we're just doing their work, committing files, or updating their working copy, nothing special. Most of the users I spoke, but I'm not 100% sure, are using SmartSVN. This -may- be an indication, however, I'm not really sure that this specific client is doing something wrong. Test/spare server - Subversion Server* We're also having a spare/test server running. I copied a few repositories to that server, installed the latest version of Collabnet Subversion Edge, and did some experiments. Subversion Distribution CollabNet Subversion Edge (3.1.2) Software version 3.1.2-3272.97 Subversion version 1.7.6-3272.97 Operating System Windows Server 2003 SP2, 2Gb (*This is a virtual server, running on VMWare 8.) However, I was able to (randomly) reproduce this error on this server, using a SmartSVN 7 client. I'm not sure what this means, but the same errors are logged and even this server eventually locked up and required a service restart. Conclusion I'm almost out of options. Wasn't able to find too many Google hits on people having same log file errors, which does concern me. Further options to investigate: * Clean Windows installation * Other OS versions (Windows Server 2008, Linux distro's) * Reproduce errors with different client versions I'm not really sure what to do. I'd like to receive some advices on how to isolate this problem and work to a solution. Thanks in advance, Looking forward to hearing from you, Kind regards, Remko Seelig Strategic Professional Configuration Management Research & Development Priva Zijlweg 3 Postbus 18 2678 ZG De Lier T +31(0)174522 498 M +31(0)6 150 103 97 F +31 (0)174 522 700 www.priva.nl <http://www.priva.nl> remko.see...@priva.nl <mailto:remko.see...@priva.nl> ______________________________________________________________________ Priva B.V. A disclaimer is applicable to this email, please refer to www.priva.nl/disclaimer. Op deze email is een disclaimer van toepassing, ga naar www.priva.nl/disclaimer.
<<image001.jpg>>
