Subversion queries hanging, timing out
Hello, Lately we have been experiencing intermittent timeouts with our Subversion operations. It does not happen initially, but after a while it starts happening. Restarting Apache alleviates the problem, but it comes back after a time. As you can imagine, this wreaks havoc. Our operating environment: - server - Windows 2003 - Apache 2.2.13 - Subversion server 1.6.3 - Subversion client 1.6.6 - mod_auth_sspi 1.0.4-2.0.58 - 200+ very active users, ~74K files We have been doing a lot of things to try and mitigate the situation, but to no avail. Changes have included: - tweaking the memory module settings (WinNT MPM) - packing the shards - trimming hooks down to minimal activity - monitoring system resources for spikes (none found, plenty of headroom, no queueing, etc) - examining the error and access logs (nothing interesting found) One thing we cannot get much of a view into is the SSPI authentication module (mod_auth_sspi). It does not seem to have any instrumentation. Has anyone experienced timeouts or deadlocks with this module? Google isn't turning up anything interesting. I've viewed the SVN interactions in Wireshark. A normal sequence of operations is: 1. client: svn log request 2. server: 401, authorization required 3. client: send creds In the hang scenario, we see just the initial client request (#1). Does this help or hurt the theory that the mod_auth_sspi/AD interaction is causing the problem? My next idea is to allow anonymous read access to the repo, which may help prove that the authentication mechanism is someone responsible. If nothing else, it should improve the performance. Lastly, it might be worth mentioning that I have exposed the same SVN repo on two different endpoints in Apache. That is, I have two location elements (with different paths) but they both point to the same repo path. Is there any problem with doing this? Thanks for reading. Please let me know if you have any ideas. Regards, Dave Purrington
Re: Subversion queries hanging, timing out
Great feedback from both of you, thanks for the help. In the meantime, we've decided to stop using mod_auth_sspi. Hopefully, that will mitigate some of this. But we definitely have more than 150 users, so I'll take a look at the max_clients setting. Cheers! -dave On Thu, Jan 14, 2010 at 4:37 AM, wrote: > Hi, > > (sorry for the format - OutBarf again..) > > > From: Dave Purrington [mailto:dave.purring...@gmail.com] >Sent: Monday, January 11, 2010 8:25 PM >To: users@subversion.apache.org >Subject: Subversion queries hanging, timing out > > >Hello, > >Lately we have been experiencing intermittent timeouts with our > Subversion operations. It does not happen initially, but after a while > it starts happening.Restarting Apache alleviates the problem, but it > comes back after a time. As you can imagine, this wreaks havoc. > > [...] > > depending on how many users you have, you might want to check the > max_clients setting in Apache - I was bitten by this recently. The > default is 150 (AFAIK) and that's too small. Increase it somewhat (like > 500), reload Apache. This might help - in my case, it did. > > I think the reason behind that is clients keeping the connection to the > server alive and the server running out of slots. > > Cheers, > > Ulli > > > > > Please note: This e-mail may contain confidential information > intended solely for the addressee. If you have received this > e-mail in error, please do not disclose it to anyone, notify > the sender promptly, and delete the message from your system. > Thank you. > >
Re: Windows 7 entries file locking
FWIW, I'm on Windows 2008 Server R2 (the server version of Windows 7) with a large repo. I have Windows Search enabled as well as virus scanning. The only time I have seen this behavior is running Filemon or Procmon during svn operations. On Thu, Jan 14, 2010 at 4:59 AM, David Turner wrote: > From the TortoiseSVN lists: > > A few people have raised this: a check-out on Windows 7 often fails > because entries is unreadable. This is caused by antivirus software and/or > Windows indexing services competing for file locks. It's fairly easy to > reproduce with a large repository on Windows 7 if you have antivirus > software and are checking out into an indexed folder. > > Obviously those of us in a corporate environment don't have the option > of disabling our antivirus software. > > Is it possible to keep entries open with exclusive access for the > duration of the move from tmp/entries to entries? >
Re: Subversion queries hanging, timing out
Kudos! The module needs some TLC, I'm sure people will appreciate the work. Unfortunately, we've moved away from it and will not be moving back. It's too disruptive to my repo user community to keep changing our auth strategy. I wish you well! -dave On Fri, Jan 15, 2010 at 9:50 AM, Andrey Repin wrote: > Greetings, Dave Purrington! > > > Lately we have been experiencing intermittent timeouts with our > Subversion > > operations. It does not happen initially, but after a while it starts > > happening. Restarting Apache alleviates the problem, but it comes back > after > > a time. As you can imagine, this wreaks havoc. > > > Our operating environment: > > >- server - Windows 2003 > >- Apache 2.2.13 > >- Subversion server 1.6.3 > >- Subversion client 1.6.6 > >- mod_auth_sspi 1.0.4-2.0.58 > >- 200+ very active users, ~74K files > > Server: Apache/2.2.11 (Win32) mod_auth_sspi/1.0.5 SVN/1.6.1 PHP/5.2.2 DAV/2 > > http://www.nosq.com/blog/2008/06/fixing-mod_auth_sspi-and-ie-losing-post-data/ > > http://dev.nosq.com/downloads/mod_auth_sspi/mod_auth_sspi_1.0.5b-vc9-2.2.11.zip > > > -- > WBR, > Andrey Repin (anrdae...@freemail.ru) 15.01.2010, <17:48> > > Sorry for my terrible english... > >
Deleted directory committed to repo translated to ignored directory locally
The scenario goes like this:
1. Homer svn deletes a folder and commits to the repository.
2. Marge svn updates, and svn reports the folder Homer deleted ("D
")
3. Marge runs svn st, some folder is not reported as unversioned.
However, the folder is still on disk (unversioned, i.e. no .svn folder) and
is now ignored (revealed by "svn st --no-ignore"). What's odd is that I
can't even tell how it's being ignored. Marge doesn't have a matching
pattern in her global ignores, and svn propget svn:ignore on the parent
folder reveals nothing useful. All clients using 1.6.6 on Windows.
What's also strange is that there are plenty of cases where files or folders
are deleted and post update they show up as unversioned items. Why the
difference?
Sneh? Bug? Feature? Cosmic rays?
Thanks,
Dave
Re: Deleted directory committed to repo translated to ignored directory locally
Clever clever. I'm still trying to wrap my head around that idea. I'll look
again at the patterns.
The fact that no one has heard of this problem makes me think I've missed a
source of ignore patterns somewhere. Time to quadruple check. I also see
from another reply that the registry is a source of ignore patterns, so I'm
going to look there as well.
On Wed, Feb 3, 2010 at 2:59 AM, Felix Gilcher wrote:
>
> On Feb 3, 2010, at 12:02 AM, Ryan Schmidt wrote:
>
> >
> > On Feb 2, 2010, at 16:34, Dave Purrington wrote:
> >
> >> The scenario goes like this:
> >> • Homer svn deletes a folder and commits to the repository.
> >> • Marge svn updates, and svn reports the folder Homer deleted ("D
> ")
> >> • Marge runs svn st, some folder is not reported as unversioned.
> >> However, the folder is still on disk (unversioned, i.e. no .svn folder)
> and is now ignored (revealed by "svn st --no-ignore"). What's odd is that I
> can't even tell how it's being ignored. Marge doesn't have a matching
> pattern in her global ignores, and svn propget svn:ignore on the parent
> folder reveals nothing useful. All clients using 1.6.6 on Windows.
> >>
> >> What's also strange is that there are plenty of cases where files or
> folders are deleted and post update they show up as unversioned items. Why
> the difference?
> >
> > I've never heard of that happening. Do you have a reproduction recipe
> that can recreate this problem?
>
> Maybe it was ignored after it had been added to the repository (matched by
> some sort of wildcard pattern or the like). It would show up during regular
> commits since it's in the repo and items that are versioned cannot be
> ignored. After deleting however, it would no longer be versioned and thus
> matched by the ignore. So it would be interesting by which ignore rule it's
> matched and whether that rule was added after the folder.
>
> Another possibility would be that someone added the folder even though it
> was ignored from the beginning.
>
> felix
>
> --
> Felix Gilcher
>
> Bitextender GmbH
> Paul-Heyse-Str. 6
> D-80336 München
>
>
>
Re: Deleted directory committed to repo translated to ignored directory locally
Felix, you win the prize. I was looking for something to specifically match
one of the deleted folder names. There is a svn:ignore value of "*" in the
parent folder. Why someone thought this was a good idea, I have no clue.
Thanks for the help!
-Dave P.
On Wed, Feb 3, 2010 at 9:02 AM, Dave Purrington
wrote:
> Clever clever. I'm still trying to wrap my head around that idea. I'll look
> again at the patterns.
>
> The fact that no one has heard of this problem makes me think I've missed a
> source of ignore patterns somewhere. Time to quadruple check. I also see
> from another reply that the registry is a source of ignore patterns, so I'm
> going to look there as well.
>
>
>
> On Wed, Feb 3, 2010 at 2:59 AM, Felix Gilcher <
> felix.gilc...@bitextender.com> wrote:
>
>>
>> On Feb 3, 2010, at 12:02 AM, Ryan Schmidt wrote:
>>
>> >
>> > On Feb 2, 2010, at 16:34, Dave Purrington wrote:
>> >
>> >> The scenario goes like this:
>> >> • Homer svn deletes a folder and commits to the repository.
>> >> • Marge svn updates, and svn reports the folder Homer deleted ("D
>> ")
>> >> • Marge runs svn st, some folder is not reported as unversioned.
>> >> However, the folder is still on disk (unversioned, i.e. no .svn folder)
>> and is now ignored (revealed by "svn st --no-ignore"). What's odd is that I
>> can't even tell how it's being ignored. Marge doesn't have a matching
>> pattern in her global ignores, and svn propget svn:ignore on the parent
>> folder reveals nothing useful. All clients using 1.6.6 on Windows.
>> >>
>> >> What's also strange is that there are plenty of cases where files or
>> folders are deleted and post update they show up as unversioned items. Why
>> the difference?
>> >
>> > I've never heard of that happening. Do you have a reproduction recipe
>> that can recreate this problem?
>>
>> Maybe it was ignored after it had been added to the repository (matched by
>> some sort of wildcard pattern or the like). It would show up during regular
>> commits since it's in the repo and items that are versioned cannot be
>> ignored. After deleting however, it would no longer be versioned and thus
>> matched by the ignore. So it would be interesting by which ignore rule it's
>> matched and whether that rule was added after the folder.
>>
>> Another possibility would be that someone added the folder even though it
>> was ignored from the beginning.
>>
>> felix
>>
>> --
>> Felix Gilcher
>>
>> Bitextender GmbH
>> Paul-Heyse-Str. 6
>> D-80336 München
>>
>>
>>
>
Re: Action needed to get critical SVN related fix in Windows 7 SP1
Erik, what is the method for this kind of feedback? Is it enough to leave a message on the note? Or are you asking that we call our rep? Should we file a ticket about it? On Thu, Feb 4, 2010 at 4:16 PM, Erik Funkenbusch wrote: > I forgot to include the Technet url > http://social.technet.microsoft.com/Forums/fi-FI/w7itprogeneral/thread/df935a52-a0a9-4f67-ac82-bc39e0585148 > > > > > > > > > > Someone posted about what is essentially Subversion issue 3574 ( > http://subversion.tigris.org/issues/show_bug.cgi?id=3574) on the technet > forums and received a response from Neal Christensen, the NTFS Lead > Developer. He indicated that this is a known issue in Windows 7, but was > not scheduled for inclusion in SP1 because it was not receiving much > feedback. > > > > If you have access to send Microsoft feedback on this issue, it is in > everyone's best interest if as many people do so as they can to get this > issue included in SP1. This seriously affects those using Windows 7 as both > a server and client for Subversion. >
svnserve not picking up sasldb credential changes
I'm testing out using SASL with svnserve (1.6.1 from Collabnet on Windows 2003 server), and it is basically working. Here is my svn.config: pwcheck_method: auxprop auxprop_plugin: sasldb sasldb_path: x:/path/to/repo/conf/sasldb mech_list: DIGEST-MD5 I'm also able to create a user in said db using the saslpasswd2 utility, and authenticate using the new user's credentials. However, svnserve only seems to load the db once, at the first access attempt. Any changes (modified passwords, new users, etc) are not detected by the service. Old passwords remain viable. Once the service is restarted, however, the new changes are picked up. Is this a known problem? Is there some file I need to touch or something? Restarting the service on every change isn't a reasonable workaround for me. Thanks in advance, Dave P.
