Re: [Python-Dev] standard library mimetypes module pathologically broken?
Benjamin Peterson wrote: Then, you might garner some more reviews by putting your patch up on Rietveld; it makes reviewing much painful. "... much _less_ painful", I hope! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Study on communication and collaboration i n software development teams
Dear Python developer, within the scope of my diploma thesis at the University of Paderborn, Germany, with the title "Study about communication and collaboration in software development in teams" I am conducting a survey of members of software development teams. I would be very grateful if you help me in my studies and answer the survey at http://thales.cs.upb.de/limesurvey185/index.php?lang=en&sid=91192&token=kkzwtzjpy5yyhxz This is the official description of the survey: In the last years many means of communication and collaboration were introduced in software projects to assist the development teams with their daily work. With this study we want to identify requirements for a communication- and collaboration-supporting platform for software development. For this purpose we will evaluate the utilization and effectiveness of different means of communication and collaboration in solving software and managerial problems in software development teams. The survey will take about 10-15 minutes and contains 55 questions that cover various topics. Many thanks for your support of my research. If there are any further questions, don't hesitate to contact me. Best regards from Paderborn, Germany Martin Gelhaus (gelh...@uni-paderborn.de) -- Click here to do the survey: http://thales.cs.upb.de/limesurvey185/index.php?lang=en&sid=91192&token=kkzwtzjpy5yyhxz Martin Gelhaus Graduand at Didactics of Informatics chair at University of Paderborn Fürstenallee 11 Room F2.416 D-33102 Paderborn ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] how to debug httplib slowness
Hi All, I'd like to work on this issue: http://bugs.python.org/issue2576 Specifically, in my case, while IE can download a 150Mb file from a local server in about 3 seconds, httplib takes over 20 minutes! However, I'm kinda stumped on where to start with debugging the difference. I've tried upping the buffer size as suggested in the issue, but it's had no effect... Any ideas? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how to debug httplib slowness
Chris Withers simplistix.co.uk> writes: > > However, I'm kinda stumped on where to start with debugging the > difference. I've tried upping the buffer size as suggested in the issue, > but it's had no effect... Then perhaps it's not the same bug. Please take a look at CPU utilization during the download. If Python takes close to 100% CPU, it might be due to the lack of buffering or any other suboptimal situation in the implementation. If Python takes close to 0%, then it's just waiting on data to arrive from the network... ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [issue6673] Py3.1 hangs in coroutine and eats up all memory
[moving this from the bug tracker] Alexandre Vassalotti wrote: > Alexandre Vassalotti added the comment: > > Not a bug. > > The list comprehension in your chunker: > > while True: > target.send([ (yield) for i in range(chunk_size) ]) > > is equivalent to the following generator in Python 3: > > while True: > def g(): > for i in range(chunk_size): > yield (yield) > target.send(list(g())) > > This clearly needs not what you want. Does this do anything meaningful, or would it make sense to output a compiler warning (or better: an error) here? Using yield in a comprehension (as opposed to a generator expression, which I intuitively expected not to work) doesn't look any dangerous at first glance, so it was quite surprising to see it fail that drastically. This is also an important issue for other Python implementations. Cython simply transforms comprehensions into the equivalent for-loop, so when we implement PEP 342 in Cython, we will have to find a way to emulate CPython's behaviour here (unless we decide to stick with Py2.x sematics, which would not be my preferred solution). Stefan > So, just rewrite your code using for-loop: > > while True: > result = [] > for i in range(chunk_size): > result.append((yield)) > target.send(result) > > -- > nosy: +alexandre.vassalotti > resolution: -> invalid > status: open -> closed ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] standard library mimetypes module pathologically broken?
Benjamin Peterson wrote: > > If python-dev was more interested, we would have a policy for this. *cough* > PEP 5 isn't enough? (I'll grant that PEP could probably do with mentioning the use of warnings.warn(DeprecationWarning) explicitly, but the policy itself seems fine) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] standard library mimetypes module pathologically broken?
Nick Coghlan wrote: > Benjamin Peterson wrote: >> >> If python-dev was more interested, we would have a policy for this. *cough* >> > > PEP 5 isn't enough? (I'll grant that PEP could probably do with > mentioning the use of warnings.warn(DeprecationWarning) explicitly, but > the policy itself seems fine) Oops, I get it now :) Cheers, Nick. P.S. For anyone else that is slow like me, take a close look at PEP 387... -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [issue6673] Py3.1 hangs in coroutine and eats up all memory
Stefan Behnel wrote: > This is also an important issue for other Python implementations. Cython > simply transforms comprehensions into the equivalent for-loop, so when we > implement PEP 342 in Cython, we will have to find a way to emulate > CPython's behaviour here (unless we decide to stick with Py2.x sematics, > which would not be my preferred solution). How do you do that without leaking the iteration variable into the current namespace? Avoiding that leakage is where the semantic change between 2.x and 3.x came from here: 2.x just creates the for loop inline (thus leaking the iteration variable into the current scope), while 3.x creates an inner function that does the iteration so that the iteration variables exist in their own scope without polluting the namespace of the containing function. The translation of your example isn't quite as Alexandre describes it - we do at least avoid the overhead of creating a generator function in the list comprehension case. It's more like: while True: def f(): result = [] for i in range(chunk_size): result.append((yield)) return result target.send(f()) So what you end up with is a generator that has managed to bypass the syntactic restriction that disallows returning non-None values from generators. In CPython it appears that happens to end up being executed as if the return was just another yield expression (most likely due to a quirk in the implementation of RETURN_VALUE inside generators): while True: def f(): result = [] for i in range(chunk_size): result.append((yield)) yield result target.send(f()) It seems to me that CPython should be raising a SyntaxError for yield expressions inside comprehensions (in line with the "no returning values other than None from generator functions" rule), and probably for generator expressions as well. Cheers, Nick. P.S. Experimentation at a 3.x interpreter prompt: >>> def f(): ... return [(yield) for i in range(10)] ... >>> x = f() >>> next(x) >>> for i in range(8): ... x.send(i) ... >>> x.send(8) >>> next(x) [0, 1, 2, 3, 4, 5, 6, 7, 8, None] >>> x = f() >>> next(x) >>> for i in range(10): # A statement with a return value! ... x.send(i) ... [0, 1, 2, 3, 4, 5, 6, 7, 8, None] >>> dis(f) 2 0 LOAD_CONST 1 ( at 0xb7c53bf0, file "", line 2>) 3 MAKE_FUNCTION0 6 LOAD_GLOBAL 0 (range) 9 LOAD_CONST 2 (10) 12 CALL_FUNCTION1 15 GET_ITER 16 CALL_FUNCTION1 19 RETURN_VALUE >>> dis(f.__code__.co_consts[1]) 2 0 BUILD_LIST 0 3 LOAD_FAST0 (.0) >>6 FOR_ITER13 (to 22) 9 STORE_FAST 1 (i) 12 LOAD_CONST 0 (None) 15 YIELD_VALUE 16 LIST_APPEND 2 19 JUMP_ABSOLUTE6 >> 22 RETURN_VALUE -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how to debug httplib slowness
Antoine Pitrou wrote: Chris Withers simplistix.co.uk> writes: However, I'm kinda stumped on where to start with debugging the difference. I've tried upping the buffer size as suggested in the issue, but it's had no effect... Then perhaps it's not the same bug. Please take a look at CPU utilization during the download. If Python takes close to 100% CPU, it might be due to the lack of buffering or any other suboptimal situation in the implementation. Well, it's locked at 25% on a quad core box, so yeah, I'd say something is wrong ;-) I guess I could try profile it and finding out where most of the time is being spent? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [issue6673] Py3.1 hangs in coroutine and eats up all memory
Nick Coghlan wrote: > Stefan Behnel wrote: >> This is also an important issue for other Python implementations. Cython >> simply transforms comprehensions into the equivalent for-loop, so when we >> implement PEP 342 in Cython, we will have to find a way to emulate >> CPython's behaviour here (unless we decide to stick with Py2.x sematics, >> which would not be my preferred solution). > > How do you do that without leaking the iteration variable into the > current namespace? We currently have 2.x sematics for comprehensions anyway, but the (long-standing) idea is to move comprehensions into their own scope (not a function, just a new type of scope), so that all names defined inside the expressions end up inside of the inner scope. This is completely orthogonal to the loop transformation itself, though, which would simply happen inside of the inner scope. However, having to emulate the other Py3 semantics for comprehensions that this thread is about, would pretty much kill such a simple solution. > The translation of your example isn't quite as Alexandre describes it - > we do at least avoid the overhead of creating a generator function in > the list comprehension case. It's more like: > > while True: > def f(): > result = [] > for i in range(chunk_size): > result.append((yield)) > return result > target.send(f()) So the problem is that f(), i.e. the function-wrapped comprehension itself, swallows the "(yield)" expression (which redundantly makes it a generator). That means that the outer function in my example, which was def chunker(chunk_size, target): while True: target.send([ (yield) for i in range(chunk_size) ]) doesn't become a generator itself, so the above simply ends up as an infinite loop. IMHO, that's pretty far from obvious when you look at the code. Also, the target receives a "generator object " instead of a list. That sounds weird. > It seems to me that CPython should be raising a SyntaxError for yield > expressions inside comprehensions (in line with the "no returning values > other than None from generator functions" rule), and probably for > generator expressions as well. Yes, that's what I was suggesting. Disallowing it in genexps is a more open question, though. I wouldn't mind being able to send() values into a generator expression, or to throw() exceptions during their execution. Anyway, I have no idea about a use case, so it might just as well be disallowed for symmetry reasons. Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how to debug httplib slowness
Try instrumenting the actual calls to the lowest-level socket methods (recv() and send()) and log for each one the arguments, return time, and how long it took. You might see a pattern. Is this on Windows? It's embarrassing, we've had problems with socket speed on Windows since 1999 and they're still not gone... :-( On Wed, Aug 12, 2009 at 4:05 AM, Chris Withers wrote: > Hi All, > > I'd like to work on this issue: > > http://bugs.python.org/issue2576 > > Specifically, in my case, while IE can download a 150Mb file from a local > server in about 3 seconds, httplib takes over 20 minutes! > > However, I'm kinda stumped on where to start with debugging the difference. > I've tried upping the buffer size as suggested in the issue, but it's had no > effect... > > Any ideas? > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how to debug httplib slowness
s/return time/return size/ On Wed, Aug 12, 2009 at 8:07 AM, Guido van Rossum wrote: > Try instrumenting the actual calls to the lowest-level socket methods > (recv() and send()) and log for each one the arguments, return time, > and how long it took. You might see a pattern. Is this on Windows? > It's embarrassing, we've had problems with socket speed on Windows > since 1999 and they're still not gone... :-( > > On Wed, Aug 12, 2009 at 4:05 AM, Chris Withers wrote: >> Hi All, >> >> I'd like to work on this issue: >> >> http://bugs.python.org/issue2576 >> >> Specifically, in my case, while IE can download a 150Mb file from a local >> server in about 3 seconds, httplib takes over 20 minutes! >> >> However, I'm kinda stumped on where to start with debugging the difference. >> I've tried upping the buffer size as suggested in the issue, but it's had no >> effect... >> >> Any ideas? >> >> Chris >> >> -- >> Simplistix - Content Management, Batch Processing & Python Consulting >> - http://www.simplistix.co.uk >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [issue6673] Py3.1 hangs in coroutine and eats up all memory
Stefan Behnel behnel.de> writes: > > IMHO, that's pretty far from obvious when you look at the code. A "yield" wrapped in a list comprehension looks far from obvious IMO anyway, whether in 2.x or 3.x. It's this kind of "smart" writing tricks people find that only makes code more difficult to read for others (à la Perl). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how to debug httplib slowness
Guido van Rossum wrote: Try instrumenting the actual calls to the lowest-level socket methods (recv() and send()) and log for each one the arguments, return time, and how long it took. Can I do that in python code? You might see a pattern. Is this on Windows? Well, yes, but I'm not 100%. The problematic machine is a Windows box, but there are no non-windows boxes on that network and vpn'ing from one of my non-windows boxes slows things down enough that I'm not confident what I'd be seeing was indicative of the same problem... It's embarrassing, we've had problems with socket speed on Windows since 1999 and they're still not gone... :-( Oh dear :-( Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how to debug httplib slowness
On Wed, Aug 12, 2009 at 8:34 AM, Chris Withers wrote: > Guido van Rossum wrote: >> >> Try instrumenting the actual calls to the lowest-level socket methods >> (recv() and send()) and log for each one the arguments, return time, >> and how long it took. > > Can I do that in python code? Probably if you hack on the socket.py file long enough. >> You might see a pattern. Is this on Windows? > > Well, yes, but I'm not 100%. The problematic machine is a Windows box, but > there are no non-windows boxes on that network and vpn'ing from one of my > non-windows boxes slows things down enough that I'm not confident what I'd > be seeing was indicative of the same problem... Time to set up a more conclusive test. Do you have something like curl or wget available on the same box? >> It's embarrassing, we've had problems with socket speed on Windows >> since 1999 and they're still not gone... :-( > > Oh dear :-( Well it may be that it's really just your box. Or proxy settings. Look into proxy settings. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] www/svn python.org status update
I replaced the RAID controller, the old data was still intact, so I brought the temporary machine down and the new machine up. Everything seems to work just fine, so happy svn-up'ing. (I will reboot mail.python.org for a few minutes, to check its serial console configuration, but that shouldn't affect anyone.) On Tue, Aug 11, 2009 at 07:26, Thomas Wouters wrote: > > > On Mon, Aug 10, 2009 at 21:12, Thomas Wouters wrote: > >> >> >> On Sat, Aug 8, 2009 at 22:22, A.M. Kuchling wrote: >> >>> The following sites are up again on a new machine, but cannot be >>> updated through SVN hooks or whatever mechanism: >>> >>> www.python.org >>> docs.python.org >>> www.jython.org >>> planet.python.org >>> planet.jython.org >>> >>> svn.python.org was deliberately not brought up again. The backups >>> were a few hours behind and missing the ~10 most recent commits. Not >>> disastrous, but it could probably mess up people's SVN trees, so after >>> some IRC discussion, the decision was to wait until the original disks >>> are available again. That will probably not occur until Monday, maybe >>> Tuesday. >> >> >> I'm still waiting on a replacement controller, so it wasn't to be today. >> Hopefully tomorrow, if the hardware supplier has one in stock. Still no >> news on whether we have any chance at all on getting the old data back. >> >> > > The new card had to be ordered (and I couldn't find any other place that > had them in stock) bit it should arrive tomorrow or thursday. On the plus > side, Martin found out there should be no problem with just inserting the > card and having it detect the RAID, so as long as the dying card didn't > write garbage to the disks we should be back up and running quite fast. > > -- > Thomas Wouters > > Hi! I'm a .signature virus! copy me into your .signature file to help me > spread! > -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] www/svn python.org status update
On Wed, Aug 12, 2009 at 9:33 AM, Thomas Wouters wrote: > > I replaced the RAID controller, the old data was still intact, so I brought > the temporary machine down and the new machine up. Everything seems to work > just fine, so happy svn-up'ing. > (I will reboot mail.python.org for a few minutes, to check its serial > console configuration, but that shouldn't affect anyone.) Yay! Thanks for your dedicated work! -gps > > On Tue, Aug 11, 2009 at 07:26, Thomas Wouters wrote: >> >> >> On Mon, Aug 10, 2009 at 21:12, Thomas Wouters wrote: >>> >>> >>> On Sat, Aug 8, 2009 at 22:22, A.M. Kuchling wrote: The following sites are up again on a new machine, but cannot be updated through SVN hooks or whatever mechanism: www.python.org docs.python.org www.jython.org planet.python.org planet.jython.org svn.python.org was deliberately not brought up again. The backups were a few hours behind and missing the ~10 most recent commits. Not disastrous, but it could probably mess up people's SVN trees, so after some IRC discussion, the decision was to wait until the original disks are available again. That will probably not occur until Monday, maybe Tuesday. >>> >>> I'm still waiting on a replacement controller, so it wasn't to be today. >>> Hopefully tomorrow, if the hardware supplier has one in stock. Still no >>> news on whether we have any chance at all on getting the old data back. >>> >> >> The new card had to be ordered (and I couldn't find any other place that >> had them in stock) bit it should arrive tomorrow or thursday. On the plus >> side, Martin found out there should be no problem with just inserting the >> card and having it detect the RAID, so as long as the dying card didn't >> write garbage to the disks we should be back up and running quite fast. >> -- >> Thomas Wouters >> >> Hi! I'm a .signature virus! copy me into your .signature file to help me >> spread! > > > > -- > Thomas Wouters > > Hi! I'm a .signature virus! copy me into your .signature file to help me > spread! > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] standard library mimetypes module pathologically broken?
Benjamin Peterson wrote: > It looks like you need to add some tests for the bugs you fixed to > test_mimetypes. While you're at it, you could improve that test > generally, since it's not exactly extensive. Okay, I'll try to do this sometime in the next few days, if I get the chance. > Then, you might garner some more reviews by putting your patch up on > Rietveld; it makes reviewing much painful. Okay, now that svn.python.org is back up, here's a Rietveld link: http://codereview.appspot.com/104091/show Cheers, Jacob Rus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how to debug httplib slowness
Chris Withers simplistix.co.uk> writes: > > Well, it's locked at 25% on a quad core box, so yeah, I'd say something > is wrong > > I guess I could try profile it and finding out where most of the time is > being spent? I guess you could indeed :-) Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
