[Python-Dev] Issues with multiprocessing
Hi, I have an application that uses multiprocessing pools (multiprocessing.Pool(processes=.)). There are multiple such pools and each pool has configurable number of processes. Once the process is spawned, it keeps on executing and does the needed processing. If there is nothing to process (like ftp'ing files from some source, if the files are not there, the process would sleep for some time, and then again check for files, that way, it is a infinite loop with some sleep), the process 'sleeps' for some time and continues. I am using a T5220, Solaris box with Solaris 10. Problem -there are multiple pools and multiple processes, i am seeing that not all the processes get spawned. They get spawned when the sleep time is increased (say from 0.1 sec to 1 sec). If I further increase the number of processes, again some process do not get spawned. For that, I further need to increase the sleep time (say to 2 sec), then the processes get spawned. Typically, in a multiprocessing I would expect that if a process sleeps for even a small time, other processes should get their chance to execute, but this does not seem to be happening here. Can you please throw some light on it? Thanks and Regards, Yateen V. Joshi This email and any attachments may contain legally privileged and/or confidential information of Starent Networks, Corp. and is intended only for the individual or entity named in the message. The information transmitted may not be used to create or change any contractual obligations of Starent Networks, Corp. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this e-mail and its attachments by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify the sender immediately -- by replying to this message or by sending an email to postmas...@starentnetworks.com -- and destroy all copies of this message and any attachments without reading or disclosing their contents. Thank you. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issues with multiprocessing
On 09/12/2009 08:51, Joshi, Yateen wrote: Hi, I have an application that uses multiprocessing pools (multiprocessing.Pool(processes=.)). There are multiple such pools and each pool has configurable number of processes. Once the process is spawned, it keeps on executing and does the needed processing. If there is nothing to process (like ftp'ing files from some source, if the files are not there, the process would sleep for some time, and then again check for files, that way, it is a infinite loop with some sleep), the process 'sleeps' for some time and continues. I am using a T5220, Solaris box with Solaris 10. Problem --there are multiple pools and multiple processes, i am seeing that not all the processes get spawned. They get spawned when the sleep time is increased (say from 0.1 sec to 1 sec). If I further increase the number of processes, again some process do not get spawned. For that, I further need to increase the sleep time (say to 2 sec), then the processes get spawned. Typically, in a multiprocessing I would expect that if a process sleeps for even a small time, other processes should get their chance to execute, but this does not seem to be happening here. Can you please throw some light on it? Hello Joshi, This email list is for the development of Python, not for developing with Python. A better place to try would be the python-list (comp.lang.python - which has both google groups and gmane gateways). All the best, Michael Foord Thanks and Regards, Yateen V. Joshi This email and any attachments may contain legally privileged and/or confidential information of Starent Networks, Corp. and is intended only for the individual or entity named in the message. The information transmitted may not be used to create or change any contractual obligations of Starent Networks, Corp. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this e-mail and its attachments by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify the sender immediately -- by replying to this message or by sending an email to postmas...@starentnetworks.com -- and destroy all copies of this message and any attachments without reading or disclosing their contents. Thank you. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposing PEP 386 for addition
More English language fixes:
-In Python there are no real restriction yet on how a project should
+In Python there are no real restrictions yet on how a project should
-Furthermore, this will make OS packagers work easier when repackaging standards
-compliant distributions, as of now it can be difficult to decide how two
+Furthermore, this will make OS packagers' work easier when repackaging
standards
+compliant distributions, because as of now it can be difficult to decide how
two
-to support many or all existing versioning schemas.
+to support all or even most of existing versioning schemas.
-reasons and we cannot expect that to change. +reasons
that we cannot expect to change.
-version schemas and is a preferable alternative than supporting
+version schemas and is a preferable alternative to supporting
-there should be possible to express more than one versioning level
+it should be possible to express more than one versioning level
-The problem with this is that while it allows expressing requisite any
-nesting level it doesn't allow to express special meaning versions
-(pre and post-releases as well as development versions), as expressed in +The
problem with this is that while it allows expressing any
+nesting level it doesn't allow giving special meaning to versions
+(pre- and post-releases as well as development versions) as expressed in
-Also, notice that Distutils version classes are not really used in the
+Also, note that Distutils version classes are not really used in the
-which does not enforce any rule for the version, but
+which does not enforce any rules for the version, but
-Setuptools function is quite spread because it's used by tools
+Setuptools function is quite widespread because it's used by tools
-post-releases -- which apparently is used by a number of projects
+post-releases -- which apparently are used by a number of projects
This one is particularly critical to your intended meaning as I read it:
-before a .post345
marker.
+before a .post456
marker.
Technical question:
I don't know what notation this versioning schema was trying for, especially in
regards to what the +'s mean:
N.N[.N]+[abc]N[.N]+[.postN+][.devN+]
Am I missing something here? You could maybe explain what the pluses mean in
the PEP, and why some are inside the [] and others are outside.
Or a regular expression version like this might be more specific.
N.N(.N)?([abc]N(.N)?)?(.postN)?(.devN)?
Or an linux help version like this may be more readible.
N.N[.N][{abc}N[.N]][.postN][.devN]
Cheers,
Michael Mysinger
(longtime python-dev lurker)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] recursive closures - reference cycle
> -Original Message- > From: python-dev-bounces+kristjan=ccpgames@python.org > [mailto:python-dev-bounces+kristjan=ccpgames@python.org] On Behalf > Of Greg Ewing > Sent: 8. desember 2009 22:09 > To: python-dev@python.org > Subject: Re: [Python-Dev] recursive closures - reference cycle > > You could use a class: Yes, and a number of different workarounds. That's not really the issue. The issue is that what looks like a perfectly safe idiom (calling a function recursively) happens to create a reference cycle if that function is a closure. This is a non-obvious "gotcha" that I must now educate my team about. Note that at least parts of the library strive to avoid reference cycles even though "gc" exists. An example being xlm.minidom.Document.unlink() Cheers, Kristján ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposing PEP 386 for addition
On Tue, Dec 08, 2009 at 08:53:18PM -0800, Michael Mysinger wrote:
> Technical question:
>
> I don't know what notation this versioning schema was trying for, especially
> in regards to what the +'s mean:
> N.N[.N]+[abc]N[.N]+[.postN+][.devN+]
> Am I missing something here? You could maybe explain what the pluses mean in
> the PEP, and why some are inside the [] and others are outside.
>
> Or a regular expression version like this might be more specific.
> N.N(.N)?([abc]N(.N)?)?(.postN)?(.devN)?
The full regex (stripped from named groups) is the rather unreadable:
\d+\.\d+(\.\d+)*([abc]?\d+(\.\d+)*)?((\.post\d+)?(\.dev\d+)?)?
(And hopfully I didn't make a mistake here)
So the '+' in the pseudo notation above roughly means "one or more"
with the brackets meaning "zero or one" so plus and brackets combined
result into "zero or more". But even then it's might be missing
square brackets around the whole of "[abc]N[.N]+".
Note that the meaning of the contents of the brackets changes too
("abc" is a choice, .postN+ is the recursive notation) so it'll
probably never work exactly. So maybe the PEP should also include the
full regex for exactness.
Regards
Floris
--
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] recursive closures - reference leak
Kristján Valur Jónsson ccpgames.com> writes: > > a gc.collect() cycle visits a large amount of objects that it > won‘t release causing cache thrashing. > > There is a reason we disabled ‚gc‘, and it is simply because we > get lower cpu and smoother execution. Could you try to enable the gc with Python trunk and report whether it does fewer and/or shorter gc collections? The gc has been optimized a bit compared to 2.6 and earlier. Thank you Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issues with multiprocessing
On Wed, Dec 9, 2009 at 6:22 AM, Michael Foord wrote: > On 09/12/2009 08:51, Joshi, Yateen wrote: > > Hi, > > > > I have an application that uses multiprocessing pools > (multiprocessing.Pool(processes=…..)). There are multiple such pools and > each pool has configurable number of processes. Once the process is spawned, > it keeps on executing and does the needed processing. If there is nothing to > process (like ftp’ing files from some source, if the files are not there, > the process would sleep for some time, and then again check for files, that > way, it is a infinite loop with some sleep), the process ‘sleeps’ for some > time and continues. > > > > I am using a T5220, Solaris box with Solaris 10. > > Problem –there are multiple pools and multiple processes, i am seeing that > not all the processes get spawned. They get spawned when the sleep time is > increased (say from 0.1 sec to 1 sec). If I further increase the number of > processes, again some process do not get spawned. For that, I further need > to increase the sleep time (say to 2 sec), then the processes get spawned. > > > > Typically, in a multiprocessing I would expect that if a process sleeps for > even a small time, other processes should get their chance to execute, but > this does not seem to be happening here. Can you please throw some light on > it? > > Hello Joshi, > > This email list is for the development of Python, not for developing with > Python. A better place to try would be the python-list (comp.lang.python - > which has both google groups and gmane gateways). > > All the best, > > Michael Foord > Additionally, if you suspect there is a reproducible box (preferably not just on solaris, which I do not have access to) please file it in the tracker. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Unittest/doctest formatting differences in 2.7a1?
I just ran the tests for zope.testing on Python 2.7, and the results are not good. It seems that there are multiple minor difference in the output formatting of the testresults between 2.7 and previous 2.x versions. The result is that all the tests that test testing (zope.testing includes a testrunner) fails. Is these changes necessary? It's going to be hell to test any form of testrunner under both 2.6 and 2.7 if the formatting of test results have changed. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
2009/12/9 Lennart Regebro : > Is these changes necessary? It's going to be hell to test any form of > testrunner under both 2.6 and 2.7 if the formatting of test results have > changed. Could you mention what specific changes are causing problems? -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
On 09/12/2009 16:27, Lennart Regebro wrote: I just ran the tests for zope.testing on Python 2.7, and the results are not good. It seems that there are multiple minor difference in the output formatting of the testresults between 2.7 and previous 2.x versions. The result is that all the tests that test testing (zope.testing includes a testrunner) fails. Is these changes necessary? It's going to be hell to test any form of testrunner under both 2.6 and 2.7 if the formatting of test results have changed. Can you be more specific? How has formatting changed and how has that broken your tests? Relying on specific formatting for test results *sounds* like you are relying on an implementation detail (although this is one of the reasons that I rarely use doctest exception in documentation - because it depends on implementation details like the repr of objects...) Some of the failure reporting in unittest has *improved* in Python 2.7 - are you feeding the output of unittest back into doctest... ? All the best, Michael Foord -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
On Wed, Dec 9, 2009 at 11:29 AM, Benjamin Peterson wrote: > Could you mention what specific changes are causing problems? I'd be glad to head Lennart confirm this, but I suspect this is primarily changes in exception messages; 2.7 in many places provides better messages. I found some of this with a few of the Zope-related packages that I tried testing under pre-2.7 versions of Python. If this is what Lennart's referring to, I consider this to be a problem with the tests. Unfortunately, doctest lends itself to having tests that check these sorts of implementation details, and the Zope community has certainly succumbed to that temptation. Evolving the tests to avoid depending on these sorts of implementation details is reasonable, IMO, and cuold even be considered a bugfix by the Zope community. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
On Wed, Dec 9, 2009 at 17:34, Michael Foord wrote: > Can you be more specific? Only with an insane amount of work. I'll hold that off for a while. > Relying on specific formatting for test results *sounds* like you are relying > on an implementation detail Yeah, but that's doctests for you. And it's a mode of testing that makes sense when you test testrunners, as the result of a testrunner is the output. It's extremely hard to *not* test a testrunner without looking at the output. > Some of the failure reporting in unittest has *improved* in Python 2.7 - are > you feeding the output of unittest back into doctest... ? Yes, as this is a testrunner being tested. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
On Wed, Dec 9, 2009 at 11:34 AM, Michael Foord wrote: > On 09/12/2009 16:27, Lennart Regebro wrote: > > I just ran the tests for zope.testing on Python 2.7, and the results are not > good. It seems that there are multiple minor difference in the output > formatting of the testresults between 2.7 and previous 2.x versions. The > result is that all the tests that test testing (zope.testing includes a > testrunner) fails. > > Is these changes necessary? It's going to be hell to test any form of > testrunner under both 2.6 and 2.7 if the formatting of test results have > changed. > This is fuzzy . It is necessary to be more precise and describe what happens in detail. Besides, I think that it's also necessary to mention (briefly) how tests look like . Perhaps this is a situation where many fragile tests [1]_ have been written and something > > Relying on specific formatting for test results *sounds* like you are > relying on an implementation detail This makes sense (~65% , and not because it's partially wrong but because of subjective interpretations and criteria) . Perhaps the best way would be to analyze contents of test results . The nature of test runners implies that many features (e.g. formatting used to display test results) are not standard . OTOH , the contents in instances of TestResult are (should be more) standard, thereby it's (probably) better to analyze the contents recorded in there (using doctest or unittest, or ... ;o) PS: I'm talking loud with my eyes closed , just a few preliminary thoughts while waiting for the details ;o) > > Some of the failure reporting in unittest has *improved* in Python 2.7 - are > you feeding the output of unittest back into doctest... ? > ... and I'm assuming that this is what's been done, isn't it ? .. [1] Fragile tests (http://xunitpatterns.com/Fragile%20Test.html) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
On Wed, Dec 9, 2009 at 17:43, Fred Drake wrote: > On Wed, Dec 9, 2009 at 11:29 AM, Benjamin Peterson > wrote: >> Could you mention what specific changes are causing problems? > > I'd be glad to head Lennart confirm this, but I suspect this is > primarily changes in exception messages; 2.7 in many places provides > better messages. Ouch, that is *not* good. That's going to break tests for all doctests, not just zope.testing (which obviously is a special case). I know PythonZope 3 has changes there do, and that's annoying in itself but acceptable since Python 3 doesn't promise to be backwards compatible, but a similar change in 2.7 is something I would consider a bug. > I found some of this with a few of the Zope-related packages that I > tried testing under pre-2.7 versions of Python. You remember which one? > If this is what Lennart's referring to, No, the only modules I've run under 2.7 so far is zope.interface which passes the tests, zope.exception which has no doctests and zope.event, which has 4 lines of code. :) And zope.testing, then, which is massive and had loads and loads of tests, mostly doctests. > I consider this to be a problem with the tests. If the exception format has changed, I consider it a bug. Possibly a bug in doctest, as the only way to test for exceptions in that case is like this: >>> try: ... throw_an_exception() ... print "Did not throw the exception" ... except DesiredException: ... print "passed" passed And that frankly is rather ugly. That's how I have done with doctests when porting to Python 3 so far. I don't think it's a good idea to require it for Python 2.7 as well. > Evolving the tests to avoid depending on these sorts of implementation > details is reasonable, IMO, and cuold even be considered a bugfix by > the Zope community. Evolving doctest.py so it can handle this by itself would be considered a bugfix by me. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
On Wed, Dec 9, 2009 at 12:14 PM, Olemis Lang wrote:
> On Wed, Dec 9, 2009 at 11:34 AM, Michael Foord
> wrote:
>> On 09/12/2009 16:27, Lennart Regebro wrote:
>>
>> I just ran the tests for zope.testing on Python 2.7, and the results are not
>> good. It seems that there are multiple minor difference in the output
>> formatting of the testresults between 2.7 and previous 2.x versions. The
>> result is that all the tests that test testing (zope.testing includes a
>> testrunner) fails.
>>
>> Is these changes necessary?
Well all Pythons need to change their (skin?) from time to time , isn't it ?
:P
>> It's going to be hell to test any form of
>> testrunner under both 2.6 and 2.7 if the formatting of test results have
>> changed.
>>
>
> This is fuzzy . It is necessary to be more precise and describe what
> happens in detail.
>
Sorry I didn't see your confirmation in the last msg you posted to the list .
@Fred Drake
{{{
Evolving the tests to avoid depending on these sorts of implementation
details is reasonable, IMO, and cuold even be considered a bugfix by
the Zope community.
}}}
Salomon says :
- Problem could impact beyond Zope itself
- unittest has been very stable (until now) so many solutions might be
tightly coupled to its implementation details (e.g. formatting and reports)
- so why not to include an option in order to keep runners compatible with
< 2.7 ?
A dinosaur will not become a Python in just a few days
;o)
> Besides, I think that it's also necessary to mention (briefly) how
> tests look like . Perhaps this is a situation where many fragile tests
> [1]_ have been written and something
>
... changed somewhere and your tests suddenly don't pass anymore
:(
>
> PS: I'm talking loud with my eyes closed , just a few preliminary
> thoughts while waiting for the details ;o)
>
... opening my eyes
:o)
> .. [1] Fragile tests
> (http://xunitpatterns.com/Fragile%20Test.html)
>
--
Regards,
Olemis.
Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/
Featured article:
Depurando errores en aplicaciones web CGI con Python -
http://feedproxy.google.com/~r/simelo-es/~3/xxyDHHH1YZ8/depurando-errores-en-aplicaciones-web.html
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
On Wed, Dec 9, 2009 at 12:23 PM, Lennart Regebro wrote: > On Wed, Dec 9, 2009 at 17:43, Fred Drake wrote: >> On Wed, Dec 9, 2009 at 11:29 AM, Benjamin Peterson >> wrote: > >> Evolving the tests to avoid depending on these sorts of implementation >> details is reasonable, IMO, and cuold even be considered a bugfix by >> the Zope community. > > Evolving doctest.py so it can handle this by itself would be > considered a bugfix by me. :) +1 This seems to be a better solution -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Depurando errores en aplicaciones web CGI con Python - http://feedproxy.google.com/~r/simelo-es/~3/xxyDHHH1YZ8/depurando-errores-en-aplicaciones-web.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
On Wed, Dec 9, 2009 at 11:23 AM, Lennart Regebro wrote: > > Evolving the tests to avoid depending on these sorts of implementation > > details is reasonable, IMO, and cuold even be considered a bugfix by > > the Zope community. > > Evolving doctest.py so it can handle this by itself would be > considered a bugfix by me. :) It's about time doctest got another run of development anyway. I can imagine a couple features that might help: * Already in there, but sometimes hard to enable, is ellipsis. Can you already do this? >>> throw_an_exception() Traceback (most recent call last): ... DesiredException: ... I'd like to see doctests be able to enable the ELLIPSIS option internally and globally (currently it can only be enabled outside the doctest, or for a single line). * Another option might be something version-specific, like: >>> throw_an_exception() # +python<2.7 ... old exception ... >>> throw_an_exception() # +python>=2.7 ... new exception ... * Maybe slightly more general, would be the ability to extend OutputCheckers more easily than currently. Maybe for instance "# py_version(less=2.7)" would enable the "py_version" output checker, which would always succeed if the version was greater than or equal to 2.7 (effectively ignoring the output). Or, maybe checkers could be extended so they could actually suppress the execution of code (avoiding throw_an_exception() from being called twice). * Or, something more explicit than ELLIPSIS but able also be more flexible than currently possible, like: >>> throw_an_exception() Traceback (most recent call last): ... DesiredException: [[2.6 error message | 2.7 error message]] -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
On Wed, 09 Dec 2009 18:23:11 +0100, Lennart Regebro wrote: > If the exception format has changed, I consider it a bug. Possibly a > bug in doctest, as the only way to test for exceptions in that case is > like this: > > >>> try: > ... throw_an_exception() > ... print "Did not throw the exception" > ... except DesiredException: > ... print "passed" > passed > > And that frankly is rather ugly. That's how I have done with doctests > when porting to Python 3 so far. I don't think it's a good idea to > require it for Python 2.7 as well. If it's a bug it would have to be in doctest, since the format of traceback messages is explicitly *not* part of the Python API. In what way is the doctest ellipsis support not sufficient for this case? --David (RDM) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
On 09/12/2009 17:23, Lennart Regebro wrote: [snip...] Evolving the tests to avoid depending on these sorts of implementation details is reasonable, IMO, and cuold even be considered a bugfix by the Zope community. Evolving doctest.py so it can handle this by itself would be considered a bugfix by me. :) That may or may not be possible (it certainly sounds like a good change). You still haven't told us what change it is that breaks things, so it is a little hard to know what to fix. Some of the unittest failure messages have changed as well, and it is just as likely to be those. Some of them were truly awful (default error message for assertTrue was "AssertionError: None") so I would be very reluctant to revert those. Even if doctest is changed (more templating so that you can specify exception messages more easily) it will probably still require some changes to the tests. Finding an alternative testing technique for your test runners may or may not be more work... Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
On Wed, Dec 9, 2009 at 18:45, Ian Bicking wrote: > It's about time doctest got another run of development anyway. I can > imagine a couple features that might help: > * Already in there, but sometimes hard to enable, is ellipsis. Can you > already do this? > > >>> throw_an_exception() > Traceback (most recent call last): > ... > DesiredException: ... I think so, but what you need is: > >>> throw_an_exception() > Traceback (most recent call last): > ... >...DesiredException: ... Because one of the changes in Python 3 is that you now get the full class path to the exception. > * Maybe slightly more general, would be the ability to extend OutputCheckers > more easily than currently. Maybe for instance "# py_version(less=2.7)" > would enable the "py_version" output checker, which would always succeed if > the version was greater than or equal to 2.7 (effectively ignoring the > output). Or, maybe checkers could be extended so they could actually > suppress the execution of code (avoiding throw_an_exception() from being > called twice). zope.testing has an output checker to which you can add transformers, that's run on both the expected and actual output. Something like that might work. We could then have a transformer that handles differences in exception formats by default. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
On Wed, Dec 9, 2009 at 18:55, Michael Foord wrote: > That may or may not be possible (it certainly sounds like a good change). > You still haven't told us what change it is that breaks things There are many changes, and it's already clear from the responses here that people consider most of them improvements, and the case of testing testrunners is special enough that I already can feel that nobody is going to care, so I'm not going to spend the hours I'm going to need to document the changes, it seems a waste of time. I suspect zope.testing just won't be tested on Python 2.7. Let's hope it continues to work on it. :) But changes in the exception format will affect everyone using doctests, so that's more serious, and will need to be handled. If we can get a similar change into 3.2 so we can handle the exception format changes between 2.6 and 3.x that would be an added bonus. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
On 05:11 pm, lrege...@jarn.com wrote: On Wed, Dec 9, 2009 at 17:34, Michael Foord wrote: Can you be more specific? Only with an insane amount of work. I'll hold that off for a while. I don't know if this is related at all (and I guess we won't until Lennart can be more specific :), but here are some Twisted unit test failures which are probably due to unittest changes in 2.7: === [FAIL]: twisted.trial.test.test_loader.LoaderTest.test_sortCases Traceback (most recent call last): File "/home/buildslave/pybot/trunk.glass-x86/build/Twisted/twisted/trial/test/test_loader.py", line 167, in test_sortCases [test._testMethodName for test in suite._tests]) twisted.trial.unittest.FailTest: not equal: a = ['test_b', 'test_c', 'test_a'] b = ['test_c', 'test_b', 'test_a'] === [FAIL]: twisted.trial.test.test_loader.ZipLoadingTest.test_sortCases Traceback (most recent call last): File "/home/buildslave/pybot/trunk.glass-x86/build/Twisted/twisted/trial/test/test_loader.py", line 167, in test_sortCases [test._testMethodName for test in suite._tests]) twisted.trial.unittest.FailTest: not equal: a = ['test_b', 'test_c', 'test_a'] b = ['test_c', 'test_b', 'test_a'] === [FAIL]: twisted.trial.test.test_tests.TestSkipMethods.test_reasons Traceback (most recent call last): File "/home/buildslave/pybot/trunk.glass-x86/build/Twisted/twisted/trial/test/test_tests.py", line 143, in test_reasons str(reason)) twisted.trial.unittest.FailTest: not equal: a = 'skip1 (twisted.trial.test.test_tests.SkippingTests)' b = 'skip1' === [FAIL]: twisted.trial.test.test_class.AttributeSharing.test_shared Traceback (most recent call last): File "/home/buildslave/pybot/trunk.glass-x86/build/Twisted/twisted/trial/test/test_class.py", line 131, in test_shared 'test_2') twisted.trial.unittest.FailTest: not equal: a = 'test_2 (twisted.trial.test.test_class.ClassAttributeSharer)' b = 'test_2' === I'm not opposed to the improvement of unittest (or any part of Python). Perhaps more of the improvements can be provided in new APIs rather than by changing the behavior of existing APIs, though. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
On Wed, Dec 9, 2009 at 18:52, R. David Murray wrote: > In what way is the doctest ellipsis support not sufficient for this > case? For 2.7 it's probably going to work, as the changes there (from what I can see amongst the massive failures I get in zope.testing) is that there is a couple of lines extra. In 3.x though the format changes from AnException: to themodule.AnException: and it seems ellipsis can't handle that case. At least it doesn't work for me. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
On 09/12/2009 18:02, exar...@twistedmatrix.com wrote: On 05:11 pm, lrege...@jarn.com wrote: On Wed, Dec 9, 2009 at 17:34, Michael Foord wrote: Can you be more specific? Only with an insane amount of work. I'll hold that off for a while. I don't know if this is related at all (and I guess we won't until Lennart can be more specific :), but here are some Twisted unit test failures which are probably due to unittest changes in 2.7: === [FAIL]: twisted.trial.test.test_loader.LoaderTest.test_sortCases Traceback (most recent call last): File "/home/buildslave/pybot/trunk.glass-x86/build/Twisted/twisted/trial/test/test_loader.py", line 167, in test_sortCases [test._testMethodName for test in suite._tests]) twisted.trial.unittest.FailTest: not equal: a = ['test_b', 'test_c', 'test_a'] b = ['test_c', 'test_b', 'test_a'] Is the order significant? Can you just compare sorted versions of the lists instead? If the order *is* significant I would be interested to know which change caused this. === [FAIL]: twisted.trial.test.test_loader.ZipLoadingTest.test_sortCases Traceback (most recent call last): File "/home/buildslave/pybot/trunk.glass-x86/build/Twisted/twisted/trial/test/test_loader.py", line 167, in test_sortCases [test._testMethodName for test in suite._tests]) twisted.trial.unittest.FailTest: not equal: a = ['test_b', 'test_c', 'test_a'] b = ['test_c', 'test_b', 'test_a'] Ditto. === [FAIL]: twisted.trial.test.test_tests.TestSkipMethods.test_reasons Traceback (most recent call last): File "/home/buildslave/pybot/trunk.glass-x86/build/Twisted/twisted/trial/test/test_tests.py", line 143, in test_reasons str(reason)) twisted.trial.unittest.FailTest: not equal: a = 'skip1 (twisted.trial.test.test_tests.SkippingTests)' b = 'skip1' === [FAIL]: twisted.trial.test.test_class.AttributeSharing.test_shared Traceback (most recent call last): File "/home/buildslave/pybot/trunk.glass-x86/build/Twisted/twisted/trial/test/test_class.py", line 131, in test_shared 'test_2') twisted.trial.unittest.FailTest: not equal: a = 'test_2 (twisted.trial.test.test_class.ClassAttributeSharer)' b = 'test_2' These two test failures are due to the change in repr of something I guess? Is a or b the original output? === I'm not opposed to the improvement of unittest (or any part of Python). Perhaps more of the improvements can be provided in new APIs rather than by changing the behavior of existing APIs, though. Well, introducing lots of new APIs has its own problems of course. Hearing about difficulties changes cause is good though (the reason for alphas I guess) and if it is possible to work out why things are broken then maybe we can still have the new functionality without the breakage. The problem with Lennart's report is that it is just "things are broken" without much clue. I am sympathetic to this (working out *exactly* what is broken in someone else's code can be a painful chore) but I'm not sure what can be done about it without more detail. All the best, Michael Jean-Paul -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
On Wed, Dec 9, 2009 at 12:23 PM, Lennart Regebro wrote: > You remember which one? Sorry, that was several months back. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
twistedmatrix.com> writes: > > I don't know if this is related at all (and I guess we won't until > Lennart can be more specific :), but here are some Twisted unit test > failures which are probably due to unittest changes in 2.7: You should do a specific diagnosis for each of these failures, and perhaps post issues on the tracker. It's not obvious to me that they are due to significant unittest changes rather than, say, text formatting details or even a misunderstanding about how the APIs work. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
On Wed, Dec 9, 2009 at 12:45 PM, Ian Bicking wrote:
> On Wed, Dec 9, 2009 at 11:23 AM, Lennart Regebro wrote:
>>
>> > Evolving the tests to avoid depending on these sorts of implementation
>> > details is reasonable, IMO, and cuold even be considered a bugfix by
>> > the Zope community.
>>
>> Evolving doctest.py so it can handle this by itself would be
>> considered a bugfix by me. :)
>
> It's about time doctest got another run of development anyway.
+1 ... that's why I implemented `dutest` BTW (I wanted more out of doctest ;o)
> I can
> imagine a couple features that might help:
>
> * Already in there, but sometimes hard to enable, is ellipsis. Can you
> already do this?
>
> >>> throw_an_exception()
> Traceback (most recent call last):
> ...
> DesiredException: ...
>
Probably useful ... unless /me want to match something inside the
error message (which seems very hard to specify if error messages
change from time to time )
> I'd like to see doctests be able to enable the ELLIPSIS option internally
> and globally (currently it can only be enabled outside the doctest, or for a
> single line).
>
Depending on what you mean when you mention «internally and globally»
, you could do such kinds of things with `dutest` too
> * Another option might be something version-specific, like:
> >>> throw_an_exception() # +python<2.7
> ... old exception ...
> >>> throw_an_exception() # +python>=2.7
> ... new exception ...
You mean skip LOCs if python version is not «compatible» with the one
specified in the doctests (e.g. # +python>=2.7) ?
> Maybe for instance "# py_version(less=2.7)"
>
[...]
>
> Or, maybe checkers could be extended so they could actually
> suppress the execution of code (avoiding throw_an_exception() from being
> called twice).
Ah ! Just what I thought !
IMO that choice should be made explicit in the test code but once
again doctest does not support something like
{{{
>>> if sys.version_info[:3] <= (2, 7) :
... throw_an_exception()
... else :
... throw_an_exception() # +python>=2.7
...
}}}
I'd definitely prefer something like that or like what you've
mentioned before (i.e. # +python<2.7)
but the idea needs to be refined ;o)
To be more explicit, the main limitation is that you cannot make
assertions for individual statements inside blocks (e.g. if, for ,
...) mainly because that's not how interactive sessions look like
;o)
> * Maybe slightly more general, would be the ability to extend OutputCheckers
> more easily than currently.
You can definitely play with these kind of things (e.g. change
OutputCheckers at runtime ;o) by using `dutest` ;o)
> * Or, something more explicit than ELLIPSIS but able also be more flexible
> than currently possible, like:
> >>> throw_an_exception()
> Traceback (most recent call last):
> ...
> DesiredException: [[2.6 error message | 2.7 error message]]
IMO, this would be limited to be used only with tracebacks, and
ellipsis are more general than that (match against anything) My
preferences above.
PS: Sorry for the commercials . I'm just mentioning that there's
something out there that enhances doctest to support some features you
mentioned , that was proposed some time ago to be included in stdlib
[3]_ and (at least in theory) is waiting for assessment &| approval .
That time the main stopper was its recent release [1]_ , now it has
about (184 + 60 + 117 = 361 downloads from PyPI ;o) [2]_ ...
.. [3] [Python-Dev] An OO API for doctest / unittest integration...
(http://mail.python.org/pipermail/python-dev/2008-August/081761.html)
.. [1] [Python-Dev] An OO API for doctest / unittest integration...
(http://mail.python.org/pipermail/python-dev/2008-August/081762.html)
.. [2] Download dutest @ PyPI
(http://pypi.python.org/pypi/dutest)
--
Regards,
Olemis.
Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/
Featured article:
Depurando errores en aplicaciones web CGI con Python -
http://feedproxy.google.com/~r/simelo-es/~3/xxyDHHH1YZ8/depurando-errores-en-aplicaciones-web.html
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] recursive closures - reference cycle
Kristján Valur Jónsson wrote: Yes, and a number of different workarounds. That's not really the issue. The issue is that what looks like a perfectly safe idiom (calling a function recursively) happens to create a reference cycle if that function is a closure. This is a non-obvious "gotcha" that I must now educate my team about. Documentation seems to be about the only thing that can be done about this. Recursion inherently involve self-reference, and that's going to create reference cycles somewhere, somehow. You get a reference cycle with top-level recursion too, the only difference is that just one cycle is created when the module is imported, rather than one every time you call the function. This is only a problem if you care about avoiding cycles, and if you do, this is just one of many subtle ways of creating them. It's hard to tell where abouts in the docs to put information about this, however. It really relates to a certain pattern of using language features rather than to any particular feature. Maybe there should be a section devoted to avoidance of reference cycles where all of these known pitfalls can be pointed out. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] recursive closures - reference cycle
> Yes, and a number of different workarounds. That's not really the > issue. The issue is that what looks like a perfectly safe idiom > (calling a function recursively) happens to create a reference cycle > if that function is a closure. This is a non-obvious "gotcha" that I > must now educate my team about. I'm not sure why you call that non-obvious. As Antoine(?) pointed out, *any* recursive function calls a reference cycle, and that is not surprising, because the function refers to itself, so the function definition is *obviously* cyclic (i.e. from within the function body, you can get back to the function object). I know you dismissed the global function example as non-relevant, because the cycle is through the globals dictionary (i.e. any global function is in a cycle, whether it is recursive or not), and because it is possible to break the cycle (by deleting the function from the globals). However, that's beside the point, because a) it is an optimization that closures are only cyclic when they need to be (i.e. a nested function only keeps references to those name bindings that are mentioned in its body, rather than referencing the entire closure), and b) it is also possible to break the cycle in the closure case, by setting the cell target to None (say), and c) the actual problem that you are having is not that the nested functions create cycles, but that you get a fresh one every time you call the outer function (so that you get the impression of a memory leak). > Note that at least parts of the library strive to avoid reference > cycles even though "gc" exists. An example being > xlm.minidom.Document.unlink() That doesn't avoid reference cycles. Instead, it helps breaking them (as a minidom tree *is* cyclic). minidom has this explicit API for breaking cycles because it predated the introduction of cyclic GC in Python. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
2009/12/9 Lennart Regebro : > On Wed, Dec 9, 2009 at 18:45, Ian Bicking wrote: >> It's about time doctest got another run of development anyway. I can >> imagine a couple features that might help: >> * Already in there, but sometimes hard to enable, is ellipsis. Can you >> already do this? >> >> >>> throw_an_exception() >> Traceback (most recent call last): >> ... >> DesiredException: ... > > I think so, but what you need is: > >> >>> throw_an_exception() >> Traceback (most recent call last): >> ... >> ...DesiredException: ... No you don't. From the manual: """ When the IGNORE_EXCEPTION_DETAIL doctest option is is specified, everything following the leftmost colon is ignored. """ So just use #doctest: +IGNORE_EXCEPTION_DETAIL Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest/doctest formatting differences in 2.7a1?
On Wed, Dec 9, 2009 at 5:47 PM, Paul Moore wrote: > 2009/12/9 Lennart Regebro : >> On Wed, Dec 9, 2009 at 18:45, Ian Bicking wrote: >>> It's about time doctest got another run of development anyway. I can >>> imagine a couple features that might help: >>> * Already in there, but sometimes hard to enable, is ellipsis. Can you >>> already do this? >>> >>> >>> throw_an_exception() >>> Traceback (most recent call last): >>> ... >>> DesiredException: ... >> >> I think so, but what you need is: >> >>> >>> throw_an_exception() >>> Traceback (most recent call last): >>> ... >>> ...DesiredException: ... > > No you don't. From the manual: > > """ > When the IGNORE_EXCEPTION_DETAIL doctest option is is specified, > everything following the leftmost colon is ignored. > """ > > So just use #doctest: +IGNORE_EXCEPTION_DETAIL Maybe that could be extended to also ignore everything up to a period (i.e., ignore the module name that seems to show up in 2.7 exception names, but not in previous versions). -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposing PEP 386 for addition
Floris Bruynooghe gmail.com> writes:
> On Tue, Dec 08, 2009 at 08:53:18PM -0800, Michael Mysinger wrote:
> > I don't know what notation this versioning schema was trying for, especially
in regards to what the +'s mean:
> > N.N[.N]+[abc]N[.N]+[.postN+][.devN+]
> >
> The full regex (stripped from named groups) is the rather unreadable:
> \d+\.\d+(\.\d+)*([abc]?\d+(\.\d+)*)?((\.post\d+)?(\.dev\d+)?)?
The ()? around the combination of post and dev is not needed. I also think
[abc]? should just be [abc], as one letter is required to proceed the digit in
that case, and the full regular expression does help to distinguish exactly
which of those two is required by the PEP.
> So the '+' in the pseudo notation above roughly means "one or more"
> with the brackets meaning "zero or one" so plus and brackets combined
> result into "zero or more". But even then it's might be missing
> square brackets around the whole of "[abc]N[.N]+".
What is confusing about the +'s is that they are not consistent. If your regular
expression with my modifications above is right, then using the substitions 'N
for \d+', '{} for []', '[] for ()?' and '+ for *' leaves:
N.N[.N]+[{abc}N[.N]+][.postN][.devN]
Notice that the last two +'s are gone, and overall I think this is more
consistent psuedo-code.
> Note that the meaning of the contents of the brackets changes too
> ("abc" is a choice, .postN+ is the recursive notation) so it'll
> probably never work exactly. So maybe the PEP should also include the
> full regex for exactness.
>
> Regards
> Floris
Yes, it probably should have the full regex for absolute clarity, and it can
still have some type of psuedo-code for easier reading, but inconsistent
psuedo-code just adds confusion instead of simplification.
Cheers,
Michael
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
