Slow initial repo access (https method)
Hi, We are using svn 1.6.16 on the server and have noticed that there is a large pause in the inital https requests, (snip of wireshark shown bellow). Basically it looks like this is a server side issue but i am not sure where to look for logs (apache ?). 10.141.80.130 (server) 10.141.81.134 (client) No. TimeSourceDestination Protocol Info 1 0.0010.141.81.134 10.141.80.130 TCP 50186 > pcsync-https [SYN] Seq=0 Win=8192 Len=0 MSS=1460 SACK_PERM=1 No. TimeSourceDestination Protocol Info 2 0.00012510.141.80.130 10.141.81.134 TCP pcsync-https > 50186 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460 SACK_PERM=1 No. TimeSourceDestination Protocol Info 3 0.00016610.141.81.134 10.141.80.130 TCP 50186 > pcsync-https [ACK] Seq=1 Ack=1 Win=64240 Len=0 No. TimeSourceDestination Protocol Info 4 0.00030710.141.81.134 10.141.80.130 TCP 50186 > pcsync-https [PSH, ACK] Seq=1 Ack=1 Win=64240 Len=227 No. TimeSourceDestination Protocol Info 5 0.01304110.141.80.130 10.141.81.134 TCP pcsync-https > 50186 [ACK] Seq=1 Ack=228 Win=65308 Len=1460 No. TimeSourceDestination Protocol Info 6 0.01306810.141.80.130 10.141.81.134 TCP pcsync-https > 50186 [PSH, ACK] Seq=1461 Ack=228 Win=65308 Len=13 No. TimeSourceDestination Protocol Info 7 0.01308410.141.81.134 10.141.80.130 TCP 50186 > pcsync-https [ACK] Seq=228 Ack=1474 Win=64240 Len=0 No. TimeSourceDestination Protocol Info 8 0.02923410.141.81.134 10.141.80.130 TCP 50186 > pcsync-https [PSH, ACK] Seq=228 Ack=1474 Win=64240 Len=198 No. TimeSourceDestination Protocol Info 9 0.03349910.141.80.130 10.141.81.134 TCP pcsync-https > 50186 [PSH, ACK] Seq=1474 Ack=426 Win=65110 Len=266 No. TimeSourceDestination Protocol Info 10 0.22508010.141.81.134 10.141.80.130 TCP 50186 > pcsync-https [ACK] Seq=426 Ack=1740 Win=63974 Len=0 No. TimeSourceDestination Protocol Info 11 15.123199 10.141.81.134 10.141.80.130 TCP 50186 > pcsync-https [PSH, ACK] Seq=426 Ack=1740 Win=63974 Len=485 No. TimeSourceDestination Protocol Info 12 15.123238 10.141.81.134 10.141.80.130 TCP 50186 > pcsync-https [PSH, ACK] Seq=911 Ack=1740 Win=63974 Len=133 No. TimeSourceDestination Protocol Info 13 15.123401 10.141.80.130 10.141.81.134 TCP pcsync-https > 50186 [ACK] Seq=1740 Ack=1044 Win=64492 Len=0 No. TimeSourceDestination Protocol Info 14 15.124022 10.141.80.130 10.141.81.134 TCP pcsync-https > 50186 [PSH, ACK] Seq=1740 Ack=1044 Win=64492 Len=746 Note the giant 15 second wait between packets 10-11, and then fast afterwards. See the slow packets 10-11 in full bellow; Frame 10: 54 bytes on wire (432 bits), 54 bytes captured (432 bits) Ethernet II, Src: Dell_4e:1a:ce (00:1e:c9:4e:1a:ce), Dst: Dell_cf:17:71 (00:1e:c9:cf:17:71) Internet Protocol, Src: 10.141.81.134 (10.141.81.134), Dst: 10.141.80.130 (10.141.80.130) Version: 4 Header length: 20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00) Total Length: 40 Identification: 0x453c (17724) Flags: 0x02 (Don't Fragment) Fragment offset: 0 Time to live: 128 Protocol: TCP (6) Header checksum: 0x [incorrect, should be 0xfe71] Source: 10.141.81.134 (10.141.81.134) Destination: 10.141.80.130 (10.141.80.130) Transmission Control Protocol, Src Port: 50186 (50186), Dst Port: pcsync-https (8443), Seq: 426, Ack: 1740, Len: 0 Source port: 50186 (50186) Destination port: pcsync-https (8443) [Stream index: 0] Sequence number: 426(relative sequence number) Acknowledgement number: 1740(relative ack number) Header length: 20 bytes Flags: 0x10 (ACK) Window size: 63974 Checksum: 0xb73c [validation disabled] [SEQ/ACK analysis] Frame 11: 539 bytes on wire (4312 bits), 539 bytes captured (4312 bits) Ethernet II, Src: Dell_4e:1a:ce (00:1e:c9:4e:1a:ce), Dst: Dell_cf:17:71 (00:1e:c9:cf:17:71) Internet Protocol, Src: 10.141.81.134 (10.141.81.134), Dst: 10.141.80.130 (10.141.80.130) Version: 4 Header length: 20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00) Total Length: 52
RE: Slow initial repo access (https method)
Thanks for the info, enabling apache logging i can see where the pause is comming from,. its between the inital client connection and granting the access rights to my user. A full 15 seconds ! This is a fast quad core xeon server as well. [Thu Apr 21 09:32:30 2011] [info] Connection: Client IP: 10.141.81.134, Protocol: TLSv1, Cipher: DHE-RSA-AES256-SHA (256/256 bits) [Thu Apr 21 09:32:45 2011] [info] [client 10.141.81.134] Access granted: 'MFletcher' OPTIONS Play:/ How do i go about finding out why its taking so long to do the inital https / SSL stuff before granting access ? I realise this crosses the boundary between SVN/apache/OpenSSL so it might be tricky. regards, Matthew > -Original Message- > From: Daniel Shahaf [mailto:d...@daniel.shahaf.name] > Sent: 20 April 2011 18:21 > To: Matthew Fletcher; users@subversion.apache.org > Subject: Re: Slow initial repo access (https method) > > > > On Wed, 20 Apr 2011 14:06 +0100, "Matthew Fletcher" > wrote: > > Hi, > > > > We are using svn 1.6.16 on the server and have noticed that > there is a large pause in the inital https requests, (snip of > wireshark shown bellow). Basically it looks like this is a > server side issue but i am not sure where to look for logs (apache ?). > > I'd firstly try to understand what the two packets are on > either side of the large pause. So... > > * A dev who knows the protocol might be able to tell what > those packets are without even looking at your data; > * You might be able to decrypt the packets you posted; > * You might be able to reproduce the problem with non-ssl connections; > * You might be able to log the packets before they get > encrypted (or after decrypted). > ** Serck Controls Ltd, Rowley Drive, Coventry, CV3 4FH, UK A company registered in England Reg. No. 4353634 Tel: +44 (0) 24 7630 5050 Fax: +44 (0) 24 7630 2437 Web: www.serck-controls.com Admin: p...@serck-controls.co.uk A subsidiary of Schneider Electric. ** This email and files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the above. Any views or opinions presented are those of the author and do not necessarily represent those of Serck Controls Ltd. This message has been scanned for malware by Mailcontrol. www.Mailcontrol.com
RE: Slow initial repo access (https method)
Hi, Nothing as fancy as LDAP, just a basic file. AuthType Basic AuthBasicProvider file I wonder if it could be due to DNS name lookup issues, http://old.nabble.com/Using-SSL-between-Apache-proxy-and-Synapse-causes-consistent-10-second-delay-in-SSL-handshake-td16014406.html "Check if reverse DNS lookups are the same for both production and stating/integration. The 10 second delays are usually caused by reverse lookups when accepting connections, since we try to get the hostname for logging." I will look into that. regards Matthew > -Original Message- > From: Daniel Shahaf [mailto:d...@daniel.shahaf.name] > Sent: 21 April 2011 09:49 > To: Matthew Fletcher; users@subversion.apache.org > Subject: RE: Slow initial repo access (https method) > > The problem may be not openssl but rather your authentication > backend (as configured in httpd.conf). For example, if you > use LDAP authentication and your LDAP server is slow to > respond, that could account for some seconds' difference. > > On Thu, 21 Apr 2011 09:44 +0100, "Matthew Fletcher" > wrote: > > > > Thanks for the info, enabling apache logging i can see > where the pause is comming from,. its between the inital > client connection and granting the access rights to my user. > A full 15 seconds ! This is a fast quad core xeon server as well. > > > > [Thu Apr 21 09:32:30 2011] [info] Connection: Client IP: > > 10.141.81.134, Protocol: TLSv1, Cipher: DHE-RSA-AES256-SHA (256/256 > > bits) [Thu Apr 21 09:32:45 2011] [info] [client > 10.141.81.134] Access > > granted: 'MFletcher' OPTIONS Play:/ > > > > How do i go about finding out why its taking so long to do > the inital https / SSL stuff before granting access ? I > realise this crosses the boundary between SVN/apache/OpenSSL > so it might be tricky. > > > > > > regards, > > > > Matthew > > > > > > > > > -Original Message- > > > From: Daniel Shahaf [mailto:d...@daniel.shahaf.name] > > > Sent: 20 April 2011 18:21 > > > To: Matthew Fletcher; users@subversion.apache.org > > > Subject: Re: Slow initial repo access (https method) > > > > > > > > > > > > On Wed, 20 Apr 2011 14:06 +0100, "Matthew Fletcher" > > > wrote: > > > > Hi, > > > > > > > > We are using svn 1.6.16 on the server and have noticed that > > > there is a large pause in the inital https requests, (snip of > > > wireshark shown bellow). Basically it looks like this is a server > > > side issue but i am not sure where to look for logs (apache ?). > > > > > > I'd firstly try to understand what the two packets are on either > > > side of the large pause. So... > > > > > > * A dev who knows the protocol might be able to tell what those > > > packets are without even looking at your data; > > > * You might be able to decrypt the packets you posted; > > > * You might be able to reproduce the problem with non-ssl > > > connections; > > > * You might be able to log the packets before they get > encrypted (or > > > after decrypted). > > > > > > > > ** > > Serck Controls Ltd, Rowley Drive, Coventry, CV3 4FH, UK A company > > registered in England Reg. No. 4353634 > > Tel: +44 (0) 24 7630 5050 Fax: +44 (0) 24 7630 2437 > > Web: www.serck-controls.com Admin: p...@serck-controls.co.uk A > > subsidiary of Schneider Electric. > > > ** > > This email and files transmitted with it are confidential > and intended > > solely for the use of the individual or entity to whom they are > > addressed. If you have received this email in error please > notify the > > above. Any views or opinions presented are those of the > author and do > > not necessarily represent those of Serck Controls Ltd. > > > > This message has been scanned for malware by Mailcontrol. > > www.Mailcontrol.com > > >
RE: Slow initial repo access (https method)
Hi, Our IT guys have been quite helpful and checked the DNS setup (and showed it to me), looks fine. A test accessing the repo via IP address gives the same delay. regards Matthew J Fletcher | Serck Controls | United Kingdom | Lead Software Engineer Phone: +44 2476 305050 | Direct Dial: +44 2476 515089 Email: mfletc...@serck-controls.co.uk | Site: www.serck-controls.co.uk | Address: Rowley Drive, Coventry, CV3 4FH, United Kingdom > -Original Message- > From: Matthew Fletcher > Sent: 21 April 2011 09:55 > To: 'Daniel Shahaf'; users@subversion.apache.org > Subject: RE: Slow initial repo access (https method) > > > Hi, > > Nothing as fancy as LDAP, just a basic file. > > AuthType Basic > AuthBasicProvider file > > > I wonder if it could be due to DNS name lookup issues, > > http://old.nabble.com/Using-SSL-between-Apache-proxy-and-Synap > se-causes-consistent-10-second-delay-in-SSL-handshake-td16014406.html > > "Check if reverse DNS lookups are the same for both > production and stating/integration. The 10 second delays are > usually caused by reverse lookups when accepting connections, > since we try to get the hostname for logging." > > I will look into that. > > > regards > > Matthew > > > > > -----Original Message- > > From: Daniel Shahaf [mailto:d...@daniel.shahaf.name] > > Sent: 21 April 2011 09:49 > > To: Matthew Fletcher; users@subversion.apache.org > > Subject: RE: Slow initial repo access (https method) > > > > The problem may be not openssl but rather your > authentication backend > > (as configured in httpd.conf). For example, if you use LDAP > > authentication and your LDAP server is slow to respond, that could > > account for some seconds' difference. > > > > On Thu, 21 Apr 2011 09:44 +0100, "Matthew Fletcher" > > wrote: > > > > > > Thanks for the info, enabling apache logging i can see > > where the pause is comming from,. its between the inital client > > connection and granting the access rights to my user. > > A full 15 seconds ! This is a fast quad core xeon server as well. > > > > > > [Thu Apr 21 09:32:30 2011] [info] Connection: Client IP: > > > 10.141.81.134, Protocol: TLSv1, Cipher: > DHE-RSA-AES256-SHA (256/256 > > > bits) [Thu Apr 21 09:32:45 2011] [info] [client > > 10.141.81.134] Access > > > granted: 'MFletcher' OPTIONS Play:/ > > > > > > How do i go about finding out why its taking so long to do > > the inital https / SSL stuff before granting access ? I > realise this > > crosses the boundary between SVN/apache/OpenSSL so it might > be tricky. > > > > > > > > > regards, > > > > > > Matthew > > > > > > > > > > > > > -Original Message- > > > > From: Daniel Shahaf [mailto:d...@daniel.shahaf.name] > > > > Sent: 20 April 2011 18:21 > > > > To: Matthew Fletcher; users@subversion.apache.org > > > > Subject: Re: Slow initial repo access (https method) > > > > > > > > > > > > > > > > On Wed, 20 Apr 2011 14:06 +0100, "Matthew Fletcher" > > > > wrote: > > > > > Hi, > > > > > > > > > > We are using svn 1.6.16 on the server and have noticed that > > > > there is a large pause in the inital https requests, (snip of > > > > wireshark shown bellow). Basically it looks like this > is a server > > > > side issue but i am not sure where to look for logs (apache ?). > > > > > > > > I'd firstly try to understand what the two packets are > on either > > > > side of the large pause. So... > > > > > > > > * A dev who knows the protocol might be able to tell what those > > > > packets are without even looking at your data; > > > > * You might be able to decrypt the packets you posted; > > > > * You might be able to reproduce the problem with non-ssl > > > > connections; > > > > * You might be able to log the packets before they get > > encrypted (or > > > > after decrypted). > > > > > > > > > > > > > ** > > > Serck Controls Ltd, Rowley Drive, Coventry, CV3 4FH, UK A company > > > registered in England Reg. No. 4353634 > > > Tel: +44 (0) 24 7630 5050 Fax: +44 (0) 24 7630 2437 > > > Web: www.serck-controls.com Admin: p...@serck-controls.co.uk A > > > subsidiary of Schneider Electric. > > > > > > ** > > > This email and files transmitted with it are confidential > > and intended > > > solely for the use of the individual or entity to whom they are > > > addressed. If you have received this email in error please > > notify the > > > above. Any views or opinions presented are those of the > > author and do > > > not necessarily represent those of Serck Controls Ltd. > > > > > > This message has been scanned for malware by Mailcontrol. > > > www.Mailcontrol.com > > > > >
RE: Slow initial repo access (https method)
Hi, Poor description on my part sorry, in my httpd.conf i explicitly turned of hostname lookups. That made no noticeable difference. I think its mod_ssl/openssl (am using 0.9.8r) just being slow, unfortunately even with apache logs set to "debug" verbosity there is no output on this, so will try and turn on logging within mod_ssl. regards Matthew J Fletcher > -Original Message- > From: Daniel Shahaf [mailto:d...@daniel.shahaf.name] > Sent: 21 April 2011 10:25 > To: Matthew Fletcher > Cc: users@subversion.apache.org > Subject: Re: Slow initial repo access (https method) > > What do you mean by "accessing the repo via IP address"? > Without having read your link, I expect it does a reverse DNS > lookup on the client's address (i.e., the IP address of the > box where the working copy resides), and just accessing the > repository as http://192.87.106.227 instead of > http://svn.apache.org won't affect that. > > The test would be > % ssh svn.apache.org time dig -x '$(echo $SSH_CONNECTION | > cut -d " " -f 1)' > > (this assumes you have a sh-like, not csh-like, login shell) > > Matthew Fletcher wrote on Thu, Apr 21, 2011 at 10:16:29 +0100: > > Hi, > > > > Our IT guys have been quite helpful and checked the DNS > setup (and showed it to me), looks fine. A test accessing the > repo via IP address gives the same delay. > > > > > > regards > > > > Matthew J Fletcher | Serck Controls | United Kingdom > | Lead Software Engineer > > Phone: +44 2476 305050 | Direct Dial: +44 2476 515089 > > Email: mfletc...@serck-controls.co.uk | Site: > www.serck-controls.co.uk | Address: Rowley Drive, > Coventry, CV3 4FH, United Kingdom > > > > > > > > > > > > > -Original Message- > > > From: Matthew Fletcher > > > Sent: 21 April 2011 09:55 > > > To: 'Daniel Shahaf'; users@subversion.apache.org > > > Subject: RE: Slow initial repo access (https method) > > > > > > > > > Hi, > > > > > > Nothing as fancy as LDAP, just a basic file. > > > > > > AuthType Basic > > > AuthBasicProvider file > > > > > > > > > I wonder if it could be due to DNS name lookup issues, > > > > > > http://old.nabble.com/Using-SSL-between-Apache-proxy-and-Synap > > > > se-causes-consistent-10-second-delay-in-SSL-handshake-td16014406.htm > > > l > > > > > > "Check if reverse DNS lookups are the same for both > production and > > > stating/integration. The 10 second delays are usually caused by > > > reverse lookups when accepting connections, since we try > to get the > > > hostname for logging." > > > > > > I will look into that. > > > > > > > > > regards > > > > > > Matthew > > > > > > > > > > > > > -Original Message- > > > > From: Daniel Shahaf [mailto:d...@daniel.shahaf.name] > > > > Sent: 21 April 2011 09:49 > > > > To: Matthew Fletcher; users@subversion.apache.org > > > > Subject: RE: Slow initial repo access (https method) > > > > > > > > The problem may be not openssl but rather your > > > authentication backend > > > > (as configured in httpd.conf). For example, if you use LDAP > > > > authentication and your LDAP server is slow to respond, > that could > > > > account for some seconds' difference. > > > > > > > > On Thu, 21 Apr 2011 09:44 +0100, "Matthew Fletcher" > > > > wrote: > > > > > > > > > > Thanks for the info, enabling apache logging i can see > > > > where the pause is comming from,. its between the inital client > > > > connection and granting the access rights to my user. > > > > A full 15 seconds ! This is a fast quad core xeon > server as well. > > > > > > > > > > [Thu Apr 21 09:32:30 2011] [info] Connection: Client IP: > > > > > 10.141.81.134, Protocol: TLSv1, Cipher: > > > DHE-RSA-AES256-SHA (256/256 > > > > > bits) [Thu Apr 21 09:32:45 2011] [info] [client > > > > 10.141.81.134] Access > > > > > granted: 'MFletcher' OPTIONS Play:/ > > > > > > > > > > How do i go about finding out why its taking so long to do > > > > the inital https / SSL stuff before granti
RE: Slow initial repo access (https method)
Hi, Thanks for the ideas, but we just use a self signed certificate. I am not really an expert with certificates, would there be any kind of lookup done to a certificate authority or other internet site with a self signed certificate ? If so is there any way to disable it ? we are just using svn internaly in an small company and dont have a big IT department. regards Matthew J Fletcher > -Original Message- > From: Brian Brophy [mailto:brianmbro...@gmail.com] > Sent: 22 April 2011 16:39 > To: Matthew Fletcher; users@subversion.apache.org > Subject: Re: Slow initial repo access (https method) > > Is your trace below only showing traffic from client to SVN > server? If so, try grabbing all traffic from the client to > see what else may be going on. There are a couple features > of SSL which may explain this. > > First, does your certificate signing chain include > intermediary signers? An example would be something like > VeriSign Public Primary G3 > -> VeriSign International Root G5 -> Your Certificate. If so, the > server should have all signers in the chain available to it, > so that if the client needs them to verify the handshake, the > server can hand them down. If the client determines it needs > the intermediaries and the server does not have them, the > client may go out to the provider (ie, > VeriSign) and attempt to download the missing certificates in > order to perform the verification. > > Second, some SSL certificates include a configuration for a > revocation link. This is a location (typically a publicly > accessible URL) that the client can view on the certificate's > metadata, and if implemented, call out to the revocation > location in order to determine if the certificate the client > is attempting to handshake with (ie, your server cert) has > since been revoked by the signer. It is a mechanism for the > signer to "de-certify" a certificate that otherwise would be > ok (ie, start/end dates are within window, cn matches hostname, etc). > > Some thoughts, > Brian > > Matthew Fletcher wrote: > > Hi, > > > > We are using svn 1.6.16 on the server and have noticed that > there is a large pause in the inital https requests, (snip of > wireshark shown bellow). Basically it looks like this is a > server side issue but i am not sure where to look for logs (apache ?). > > > > 10.141.80.130 (server) > > 10.141.81.134 (client) > > > > No. TimeSourceDestination > Protocol Info > > 1 0.0010.141.81.134 10.141.80.130 > TCP 50186 > pcsync-https [SYN] Seq=0 Win=8192 Len=0 > MSS=1460 SACK_PERM=1 > > No. TimeSourceDestination > Protocol Info > > 2 0.00012510.141.80.130 10.141.81.134 > TCP pcsync-https > 50186 [SYN, ACK] Seq=0 Ack=1 > Win=16384 Len=0 MSS=1460 SACK_PERM=1 > > No. TimeSourceDestination > Protocol Info > > 3 0.00016610.141.81.134 10.141.80.130 > TCP 50186 > pcsync-https [ACK] Seq=1 Ack=1 Win=64240 Len=0 > > No. TimeSourceDestination > Protocol Info > > 4 0.00030710.141.81.134 10.141.80.130 > TCP 50186 > pcsync-https [PSH, ACK] Seq=1 Ack=1 > Win=64240 Len=227 > > No. TimeSourceDestination > Protocol Info > > 5 0.01304110.141.80.130 10.141.81.134 > TCP pcsync-https > 50186 [ACK] Seq=1 Ack=228 > Win=65308 Len=1460 > > No. TimeSourceDestination > Protocol Info > > 6 0.01306810.141.80.130 10.141.81.134 > TCP pcsync-https > 50186 [PSH, ACK] Seq=1461 Ack=228 > Win=65308 Len=13 > > No. TimeSourceDestination > Protocol Info > > 7 0.01308410.141.81.134 10.141.80.130 > TCP 50186 > pcsync-https [ACK] Seq=228 Ack=1474 > Win=64240 Len=0 > > No. TimeSourceDestination > Protocol Info > > 8 0.02923410.141.81.134 10.141.80.130 > TCP 50186 > pcsync-https [PSH, ACK] Seq=228 Ack=1474 > Win=64240 Len=198 > > No. TimeSourceDestination > Protocol Info > > 9 0.03349910.141.80.130 10.141.81.134 > TCP pcsync-https > 50186 [PSH, ACK] Seq=1474 Ack=426 > Win=65110 Len=266 > > No. TimeSourceDestination
Re: Slow initial repo access (https method)
Hi, Just to let everyone know that the problem turned out to be that SSL applications on Windows (the TortoiseSVN client in our case) lookup www.download.windowsupdate.com to get updates to the certificate revocation list. See http://support.microsoft.com/kb/317541 We operate in an environment with no direct internet access (proxy only) so this request failed and made a 15 second pause on every connection. regards Matthew J Fletcher ** Serck Controls Ltd, Rowley Drive, Coventry, CV3 4FH, UK A company registered in England Reg. No. 4353634 Tel: +44 (0) 24 7630 5050 Fax: +44 (0) 24 7630 2437 Web: www.serck-controls.com Admin: p...@serck-controls.co.uk A subsidiary of Schneider Electric. ** This email and files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the above. Any views or opinions presented are those of the author and do not necessarily represent those of Serck Controls Ltd. This message has been scanned for malware by Mailcontrol. www.Mailcontrol.com
