Re: [Python-Dev] how to debug httplib slowness

2009-09-17 Thread Bill Janssen
Oleg Broytmann wrote: > On Fri, Sep 04, 2009 at 04:02:39PM +0100, Chris Withers wrote: > > So, httplib does this: > > > > GET / HTTP/1.1 > [skip] > > While wget does this: > > > > GET / HTTP/1.0 > [skip] > > - Apache responds with a chunked response only to httplib. Why is that? > >Probably

Re: [Python-Dev] how to debug httplib slowness

2009-09-04 Thread Antoine Pitrou
Guido van Rossum python.org> writes: > > +1 on trying this. Constructing a 116MB string by concatenating 1KB > buffers surely must take forever. (116MB divided by 85125 recv() calls > give 1365 byte per chunk, which is awful.) It certainly is quite small but perhaps the server tries to stay belo

Re: [Python-Dev] how to debug httplib slowness

2009-09-04 Thread Chris Withers
Guido van Rossum wrote: +1 on trying this. Constructing a 116MB string by concatenating 1KB buffers surely must take forever. (116MB divided by 85125 recv() calls give 1365 byte per chunk, which is awful.) The HTTP/1.0 business looks like a red herring. Also agreed that this is an embarrassment.

Re: [Python-Dev] how to debug httplib slowness

2009-09-04 Thread Guido van Rossum
On Fri, Sep 4, 2009 at 4:28 AM, Simon Cross wrote: > On Fri, Sep 4, 2009 at 1:11 PM, Chris Withers wrote: >> Am I right in reading this as most of the time is being spent in httplib's >> HTTPResponse._read_chunked and none of the methods it calls? >> >> If so, is there a better way that a bunch of

Re: [Python-Dev] how to debug httplib slowness

2009-09-04 Thread Scott Dial
Chris Withers wrote: > - Apache in this instance responds with HTTP 1.1, even though the wget > request was 1.0, is that allowed? > > - Apache responds with a chunked response only to httplib. Why is that? > I find it very confusing that you say "Apache" since your really want to say "Coyote" wh

Re: [Python-Dev] how to debug httplib slowness

2009-09-04 Thread Oleg Broytmann
On Fri, Sep 04, 2009 at 04:02:39PM +0100, Chris Withers wrote: > So, httplib does this: > > GET / HTTP/1.1 [skip] > While wget does this: > > GET / HTTP/1.0 [skip] > - Apache responds with a chunked response only to httplib. Why is that? Probably because wget uses HTTP/1.0? Oleg. -- Oleg

Re: [Python-Dev] how to debug httplib slowness

2009-09-04 Thread Chris Withers
Simon Cross wrote: Well, since the source for _read_chunked includes the comment # XXX This accumulates chunks by repeated string concatenation, # which is not efficient as the number or size of chunks gets big. you might gain some speed improvement with minimal effort by gather

Re: [Python-Dev] how to debug httplib slowness

2009-09-04 Thread Antoine Pitrou
Simon Cross gmail.com> writes: > > Well, since the source for _read_chunked includes the comment > > # XXX This accumulates chunks by repeated string concatenation, > # which is not efficient as the number or size of chunks gets big. > > you might gain some speed improvement wit

Re: [Python-Dev] how to debug httplib slowness

2009-09-04 Thread Simon Cross
On Fri, Sep 4, 2009 at 1:11 PM, Chris Withers wrote: > Am I right in reading this as most of the time is being spent in httplib's > HTTPResponse._read_chunked and none of the methods it calls? > > If so, is there a better way that a bunch of print statements to find where > in that method the time

Re: [Python-Dev] how to debug httplib slowness

2009-09-04 Thread Chris Withers
Guido van Rossum wrote: You might see a pattern. Is this on Windows? Well, yes, but I'm not 100%. The problematic machine is a Windows box, but there are no non-windows boxes on that network and vpn'ing from one of my non-windows boxes slows things down enough that I'm not confident what I'd be

Re: [Python-Dev] how to debug httplib slowness

2009-08-12 Thread Antoine Pitrou
Chris Withers simplistix.co.uk> writes: > > Well, it's locked at 25% on a quad core box, so yeah, I'd say something > is wrong > > I guess I could try profile it and finding out where most of the time is > being spent? I guess you could indeed :-) Antoine.

Re: [Python-Dev] how to debug httplib slowness

2009-08-12 Thread Guido van Rossum
On Wed, Aug 12, 2009 at 8:34 AM, Chris Withers wrote: > Guido van Rossum wrote: >> >> Try instrumenting the actual calls to the lowest-level socket methods >> (recv() and send()) and log for each one the arguments, return time, >> and how long it took. > > Can I do that in python code? Probably if

Re: [Python-Dev] how to debug httplib slowness

2009-08-12 Thread Chris Withers
Guido van Rossum wrote: Try instrumenting the actual calls to the lowest-level socket methods (recv() and send()) and log for each one the arguments, return time, and how long it took. Can I do that in python code? You might see a pattern. Is this on Windows? Well, yes, but I'm not 100%. Th

Re: [Python-Dev] how to debug httplib slowness

2009-08-12 Thread Guido van Rossum
s/return time/return size/ On Wed, Aug 12, 2009 at 8:07 AM, Guido van Rossum wrote: > Try instrumenting the actual calls to the lowest-level socket methods > (recv() and send()) and log for each one the arguments, return time, > and how long it took. You might see a pattern. Is this on Windows? >

Re: [Python-Dev] how to debug httplib slowness

2009-08-12 Thread Guido van Rossum
Try instrumenting the actual calls to the lowest-level socket methods (recv() and send()) and log for each one the arguments, return time, and how long it took. You might see a pattern. Is this on Windows? It's embarrassing, we've had problems with socket speed on Windows since 1999 and they're sti

Re: [Python-Dev] how to debug httplib slowness

2009-08-12 Thread Chris Withers
Antoine Pitrou wrote: Chris Withers simplistix.co.uk> writes: However, I'm kinda stumped on where to start with debugging the difference. I've tried upping the buffer size as suggested in the issue, but it's had no effect... Then perhaps it's not the same bug. Please take a look at CPU utili

Re: [Python-Dev] how to debug httplib slowness

2009-08-12 Thread Antoine Pitrou
Chris Withers simplistix.co.uk> writes: > > However, I'm kinda stumped on where to start with debugging the > difference. I've tried upping the buffer size as suggested in the issue, > but it's had no effect... Then perhaps it's not the same bug. Please take a look at CPU utilization during th