Re: [Python-Dev] Can LOAD_GLOBAL be optimized to a simple array lookup?
Alex Martelli wrote: > > On Aug 23, 2006, at 2:22 PM, K.S.Sreeram wrote: > >> Hi all, >> >> I noticed in Python/ceval.c that LOAD_GLOBAL uses a dictionary lookup, >> and was wondering if that can be optimized to a simple array lookup. >> >> If i'm right there are 3 kinds of name lookups: locals, outer >> scopes(closures), and globals. (not counting attribute lookup). Locals >> are identified by, either the presence of assignments, or their presence >> in the arg list. So all name lookups can be classified into the 3 types >> at compile/load time. >> >> Since we know, at load time, which names are global.. Can't we simply >> build a global name table and replace LOAD_GLOBALs with a lookup at the >> corresponding index into the global name table? > > At the time the function's body gets compiled, the global (or builtin) > it's trying to access might or might not be there -- as long as it gets > added afterwards, before the function's body gets _executed_, no problem > (in today's scheme). It's not obvious to me how you could compile a > ``corresponding index'' into the LOAD_GLOBAL opcode, since that index is > in general unknown at compile time. How about this approach: (disclaimer: this is just a rough sketch!) At compile time: --- - When compiling the module, a list of global names is built. This list is populated in the order in which the names appear in the module. References to the global will be replaced by references to the index in this list. - Instead of generating LOAD_GLOBAL and STORE_GLOBAL opcodes, we generate LOAD_GLOBAL_BY_INDEX and STORE_GLOBAL_BY_INDEX, which are new opcodes. The indexes used with the opcode will be the index of the name from the list. - This list of names should be stored in the pyc file so that the ordering is remembered. At load time: - - At module load time, we build an array of PyObject*, which is used to store the values for the globals. The values will be NULL, until the actual binding is created. - For each name in the global_name_list in the pyc file, we pass the pointer to the corresponding entry in the global table, to the dict. PyDict_SetLocationForName( module_dict, name, PyObject **ptr_to_entry_table ) This is done so that 'get' and 'set' of these names in the dict will update/fetch from the global table directly. This approach should result in faster access to globals defined within the module. It will still work if we try to shadow any builtins. As far as i see, this doesn't seem to change any existing semantics either. This does not result in faster access to existing builtins. But a similar scheme can be worked out for that too. > >> The module's dict object will need to be special so that whenever a name >> gets 'set', the global name table should get updated. > > It seems that you'd need to chase down and modify all of the LOAD_GLOBAL > opcodes too, at every such modification. (the concept of modifying > builtins becomes extremely scary...). Considering the amortized speed > of a dict lookup for an interned string (hash value cached and thus > immediately available, equality comparison with other interned string a > single machine-level operation), it's not clear to me that the huge > complexity (and potential performance impact) of all this could ever > possibly be justified. > > A change in Python semantics allowing some level of "nailing down" of > builtins (and perhaps globals too) *COULD* easily yield large > performance benefits, but that's a subject for the Python-3000 mailing > list; as long as builtins and globals stay as fluid as today, I'm > skeptical on the optimization opportunities here. [sreeram;] signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 362 open issues
Brett Cannon wrote: > Another question: could it be helpful to make Parameter.default_value a > weak reference? > > > Perhaps, but I don't think it is necessarily required. I can change it > if others want to go that way, but for now I am happy with the way it is. Leave it as a normal reference - if I have a reference to a Signature object, I'd like all it's attributes to stick around, thank you very much :) Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can LOAD_GLOBAL be optimized to a simple arraylookup?
Zitat von "K.S.Sreeram" <[EMAIL PROTECTED]>: > How about this approach: > (disclaimer: this is just a rough sketch!) This is actually the problem. There are many fine details which can affect performance and correctness. So about there are only two sensible ideas to treat such ideas: either implement them, or ignore them. If you think your approach could work, please try to implement it. It's not that anybody is objecting the goal; there is just debate about the feasibility of the implementation. So if you can come up with an implementation, we are in a much better position to praise or criticise the approach: it then becomes possible to study whether the implementation is really compatible with the current Python, and whether it really does improve performance. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] xrange accepting non-ints
I've been working on enhancing xrange and there are a bunch of issues to consider. I've got pretty much complete implementations in both C and Python. Currently xrange is 2 objects: range and the iter. These only work on C longs. Here's what I propose: 2.6: * Add deprecation warning if a float is passed to xrange (currently silently truncates) * Allow int, long, float, or obj.__index__ * Implement xrange in python * Implement iterator over C longs (or Py_ssize_t) in C * Implement iterator over Python longs in Python (* may lose __length_hint__) * Capture the values on construction, so mutating objects wouldn't affect xrange The idea is to expand xrange's capabilities so that it can replace range in 3k. I've profiled various combinations. Here are the various results normalized doing xrange(0, 1e6, 1): Run on all integer (32-bit) values for start, step, end: C xrange and iter: 1 Py xrange w/C iter: 1 Py xrange w/Py iter (gen): 5-8 Py xrange w/Py iter (class): ~30 So having xrange in python is the same speed as if xrange is written in C. The important part is that the iterator needs to be written in C for speed. If we use a generator, something like: while value < end: yield value value += step The result is ~5 times slower in a release build and 8 times slower in a debug build than with an iterator implemented in C. Using a generator means that there is no __length_hint__. If we go with a full class that has a __length_hint__ the result was ~32 times slower in a debug build. The Python impl is about 1/10th the size of the C impl, though is lacking some comments. Run on Python longs the result is somewhat interesting. The Python based iterator is faster. There's probably a bug in the C version, but given that there is a lot of object allocation, I wouldn't expect the C version to ever be much faster than a similar Python version. Plus the Python version is trivial (same as above) for ints or longs. The C version for longs is quite a bit of code. Run on all Python longs (still 0..1e6, but sys.maxint..(sys.maxint + 1e6) is the same): C xrange and iter: 1.4 Py xrange w/C iter: not tested Py xrange w/Py iter (gen): 1 Py xrange w/Py iter (class): 4 Caveats: * The generator version above doesn't support floats. We could easily support floats with a different calculation that would be slightly more expensive, but not have accumulated error. * By using the generator version, __length_hint__ gets dropped. This means that converting the iterator into a sequence could be slightly more costly as we have to increase the allocation. This would only happen if any of start, step, end weren't an int. * With a python implementation there is a little bit of bootstraping that is necessary to get the iter implemented in C into the xrange object implemented in Python * Since xrange is implemented in Python, it can be changed. * The Python code is much easier to understand than the C code (there is at least one bug in the current C version where -sys.maxint -1 isn't always displayed properly). Hopefully this is all understandable. If I left anything out, Thomas will remind me. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] xrange accepting non-ints
On 8/24/06, Neal Norwitz <[EMAIL PROTECTED]> wrote: > I've been working on enhancing xrange and there are a bunch of issues > to consider. I've got pretty much complete implementations in both C > and Python. Currently xrange is 2 objects: range and the iter. > These only work on C longs. Here's what I propose: > > 2.6: > * Add deprecation warning if a float is passed to xrange (currently > silently truncates) > * Allow int, long, float, or obj.__index__ float? I thought the first bullet says no float? > * Implement xrange in python Since xrange is used in performance critical apps that may be a bad idea. Or maybe only if the args aren't all ints? > * Implement iterator over C longs (or Py_ssize_t) in C > * Implement iterator over Python longs in Python (* may lose __length_hint__) > * Capture the values on construction, so mutating objects wouldn't > affect xrange Right. So capture them as Python int or long only. > The idea is to expand xrange's capabilities so that it can replace range in > 3k. > > I've profiled various combinations. Here are the various results > normalized doing xrange(0, 1e6, 1): > > Run on all integer (32-bit) values for start, step, end: > C xrange and iter: 1 > Py xrange w/C iter: 1 > Py xrange w/Py iter (gen): 5-8 > Py xrange w/Py iter (class): ~30 > > So having xrange in python is the same speed as if xrange is written > in C. I'm not sure I believe this benchmark -- to measure the cost of xrange itself (as opposed to the cost of iterating over the iterator) you should test xrange(0) or xrange(1). > The important part is that the iterator needs to be written in > C for speed. If we use a generator, something like: > > while value < end: > yield value > value += step > > The result is ~5 times slower in a release build and 8 times slower in > a debug build than with an iterator implemented in C. Nobody cares about the speed of the debug build. :-) > Using a > generator means that there is no __length_hint__. If we go with a > full class that has a __length_hint__ the result was ~32 times slower > in a debug build. I don't mind not having the length hint for longs. Doesn't current xrange also support a faster version of reversed? > The Python impl is about 1/10th the size of the C impl, though is > lacking some comments. > > Run on Python longs the result is somewhat interesting. The Python > based iterator is faster. There's probably a bug in the C version, > but given that there is a lot of object allocation, I wouldn't expect > the C version to ever be much faster than a similar Python version. > Plus the Python version is trivial (same as above) for ints or longs. > The C version for longs is quite a bit of code. If the longs are large enough, the addition is going to dominate the cost, yes. > Run on all Python longs (still 0..1e6, but sys.maxint..(sys.maxint + > 1e6) is the same): > C xrange and iter: 1.4 > Py xrange w/C iter: not tested > Py xrange w/Py iter (gen): 1 > Py xrange w/Py iter (class): 4 > > Caveats: > * The generator version above doesn't support floats. We could > easily support floats with a different calculation that would be > slightly more expensive, but not have accumulated error. Is there a good use case for supporting float? The problem with floats is that even apart from accumulated error, it's still ambiguous. E.g. will xrange(1.1, 1.9, 0.1) include the end point or not? That would depend on the details of decimal-to-binary conversion. > * By using the generator version, __length_hint__ gets dropped. This > means that converting the iterator into a sequence could be slightly > more costly as we have to increase the allocation. This would only > happen if any of start, step, end weren't an int. Fine with me -- you can't do that at all at the moment. :-) > * With a python implementation there is a little bit of bootstraping > that is necessary to get the iter implemented in C into the xrange > object implemented in Python Long-term, I'd rather see it implemented all in C. Short term, the Python implementation is great to experiment. > * Since xrange is implemented in Python, it can be changed. > * The Python code is much easier to understand than the C code (there > is at least one bug in the current C version where -sys.maxint -1 > isn't always displayed properly). > > Hopefully this is all understandable. If I left anything out, Thomas > will remind me. I'm about to head out to Google now... -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] xrange accepting non-ints
On 8/24/06, Guido van Rossum <[EMAIL PROTECTED]> wrote: On 8/24/06, Neal Norwitz <[EMAIL PROTECTED]> wrote:> I've been working on enhancing xrange and there are a bunch of issues> to consider. I've got pretty much complete implementations in both C > and Python. Currently xrange is 2 objects: range and the iter.> These only work on C longs. Here's what I propose:>> 2.6:> * Add deprecation warning if a float is passed to xrange (currently > silently truncates)> * Allow int, long, float, or obj.__index__float? I thought the first bullet says no float?No, the bullet says 'add warning' :) xrange() currently accepts floats, because it uses one of the integer getargs formats: >>> xrange(1.2, 2.5, 1.)xrange(1, 2)> * Implement xrange in python Since xrange is used in performance critical apps that may be a badidea. Or maybe only if the args aren't all ints?Is the cost of *calling* xrange() really a big issue? I don't think Neal measured this, but he could. I'd imagine performance-critical apps call xrange() once, then use that to iterate. > Caveats:> * The generator version above doesn't support floats. We could > easily support floats with a different calculation that would be> slightly more expensive, but not have accumulated error.Is there a good use case for supporting float? The problem with floats is that even apart from accumulated error, it's still ambiguous. E.g.will xrange(1.1, 1.9, 0.1) include the end point or not? That woulddepend on the details of decimal-to-binary conversion.Supporting floats is definately problematic. It would be nice if xrange() supported arbitrary numeric types, though, like decimals. That would quench the thirst people seem to have for float-ish xranges. > * With a python implementation there is a little bit of bootstraping> that is necessary to get the iter implemented in C into the xrange > object implemented in PythonLong-term, I'd rather see it implemented all in C. Short term, thePython implementation is great to experiment.Why, other than performance? It's a lot simpler and much easier to get right in Python, which is quite good for maintenance, too. -- Thomas Wouters <[EMAIL PROTECTED]>Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] xrange accepting non-ints
Neal Norwitz wrote: > I've profiled various combinations. Here are the various results > normalized doing xrange(0, 1e6, 1): > > Run on all integer (32-bit) values for start, step, end: > C xrange and iter: 1 > Py xrange w/C iter: 1 in real life, loops are a lot shorter than that. if you take that into account, you don't even have to run the benchmark to realize that calling a Python function and checking the arguments before calling a C function takes more time than calling a C function. even if you skip the "check the arguments" part, you take a 5% hit: > timeit -s"def myxrange(a,xrange=xrange): return xrange (a)" "for i in myxrange(100): pass" 10 loops, best of 3: 5.28 usec per loop > timeit "for i in xrange(100): pass" 10 loops, best of 3: 4.98 usec per loop > timeit -s"def myxrange(a,b=None,c=None,xrange=xrange): return xrange(a,b,c)" "for i in myxrange(0,100,1): pass" 10 loops, best of 3: 5.58 usec per loop > timeit "for i in xrange(0,100,1): pass" 10 loops, best of 3: 5.27 usec per loop I doubt adding more code to the myxrange function will speed it up... ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can LOAD_GLOBAL be optimized to a simple array lookup?
Note that there are already three PEPs related to speeding dict-based namespace access; start with: http://www.python.org/dev/peps/pep-0280/ which references the other two. The "normal path" through dict lookups got faster since there was a rash of those, to the extent that more complication elsewhere got much less attractive. It's possible that dict lookups got slower again since /then/, though. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can LOAD_GLOBAL be optimized to a simple array lookup?
Tim Peters wrote: > Note that there are already three PEPs related to speeding dict-based > namespace access; start with: > > http://www.python.org/dev/peps/pep-0280/ > > which references the other two. > > The "normal path" through dict lookups got faster since there was a > rash of those, to the extent that more complication elsewhere got much > less attractive. It's possible that dict lookups got slower again > since /then/, though. Thanks! Looks like i'm only proposing what is already there in pep-267 by Jeremy Hylton. http://www.python.org/dev/peps/pep-0267/ cheers [sreeram;] signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] for 2.5 issues
I don't want to make any more changes to 2.5 unless we absolutely have to. I also don't want to lose fixes. How about for anything that should be resolved in 2.5, but wait for 2.5.1 we set the tracker item to: Group 2.5, Resolution: Later, Priority 7. Then it should be easy to find these things. Thanks, n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] draft for bug guidelines
Below is the current draft of a set of bug guidelines for people to follow when they want to file a bug. The hope is that when we get an email asking "how do I file a bug?" we can point them towards these sets of guidelines for most issues. Let me know about any errors and such. This will (hopefully) end up going up on python.org/dev/ eventually.-About/// These sets of guidelines are to help you file a bug report for thePython programming language on SourceForge_. If your bug is not forthe language but for a third-party application, please report the bugto that third-party. *Please make sure to follow every step as it will make the livesof the Python developers much easier!!!*.. contents::Steps to File a Bug Report/// Get a SourceForge account===In order to file a bug report, you must have an account_ onSourceForge_. We realize some people would like to have anonymousbug reporting for various reasons (anonymity, ease or reporting, etc.). But the Python developers made the decision to turn offanonymous posting so as to make sure you are notified by email whenany action is taken on your report. This can be very important if aPython developer needs more information from you about the bug. Start a new bug===You must be logged into SourceForge to file a bug! See `Get aSourceForge account`_ if you do not have one.Go to the `SourceForge bug page`_ to start a new bug report. There you will find a link called `Submit New`_. Click on it and it willallow you to fill out a new bug report.Once you click on the link, you are presented with a page that hasseveral fields for you to fill in. Here is what to do for each field: * Private Leave this unchecked.* Category Set this to the area that the bug is related to (e.g., documentation, build, etc.).* Group Usually this is set to the major.minor version of Python that you found the bug in.* Summary A one-liner describing the problem so as to make it easy for developers to spot whether they have the expertise needed to work on the bug; it also ends up being displayed as the title on index and search pages.* Detailed Description Following sections of this document discuss what should go in here.* Upload and Attach a File If you are going to upload a file, either type in the path to the file or click the ``Browse...`` button to find the file.* File Description A one-liner describing the file (e.g., how it is different from previously uploaded files); no date info is needed since the upload is timestamped.Specify the Python version===It is important that we have the most accurate version number of theinterpreter you are using in order to best diagnose the issue. There are two ways to get us the version information.If you can run your Python interpreter, execute the following lines atat a shell prompt and paste the result into the``Detailed Description`` field of the bug report:: python -c "import sys; print sys.version"If you are running a version of Python older than 2.5 and are workingfrom a source checkout of Python, the please also report theSubversion revision number for the root of your checkout:: svnversion .If your bug is preventing you from running the interpreter, executePython with the ``-V`` command-line flag and paste the output:: python -VSpecial settings for your Python interpreter Sometimes your environment influences a bug and thus needs to bereported to help find the problem. This means we need to havereported:* Operating System * Environment Variables + ``PYTHONSTARTUP`` If this is set and might be causing the issue, please either upload the file or state what it does. + ``PYTHONCASEOK`` If your bug is on Windows and involves importing, please report if this environment variable is set or not.* ``site-packages`` If you have any third-party packages installed that may be contributing to the bug, please report those.* Custom Patches Any differences between your code and the code the Python developers work off of needs to be reported.Sample code to reproduce bugIf you can, please upload a file the demonstrates the bug. The more succinct the better!Submit!At this point you should have a detailed bug report for developers towork off of. Click the ``Submit`` button and read on to see what you should do after the bug is reported.Respond to requests from developersNo matter how detailed the bug report, there is always the chancethat a developer will need more information to fix a bug. Please be prompt in replying to requests for information by
[Python-Dev] draft of patch guidelines
Below is a draft for a set of patch guidelines. This is meant for that times where people ask "how do a create a patch for a change I made" or where to point people if they created a patch but it comes up short (no tests, etc.). Hopefully this will go up on python.org/dev/ .Let me know of any errors and such.--About/This set of guidelines is meant for people who have written code that they wish to have committed to the Python programming language. Thismeans that this set of guidelines assumes you have already writtencode and thus just want to know the steps required to generate thepatch necessary to submit your change for addition in Python. The assumption you are using a UNIX-style shell is also made.If you need help with any of these steps, please ask for it! Usuallyyou can create a new patch missing a few steps and the Pythondevelopers will try to help you complete it so that your patch can eventuallly be accepted.Guidelines for committing code to Python will be contained in aseparate document... contents::Steps to Creating a Patch/Read the Developer Intro to understand the scope of your proposed change Changes to Python come in various levels of scope. You could befixing a simple bug in the standard library. You might be trying to add a new feature to a module. You might even be suggesting a newfeature to the language. All three have a different amount of scopeand thus require varying amounts of work to get approval.Please read the `Developer Intro`_ doc to make sure you realize the proper scope of what you are doing. It also tells you the rough stepsneeded for varying types of changes to be accepted.Add the appropriate unit tests==Python prides itself on being a very stable piece of software. We maintain this by having a regression test suite that contains a hugeamount of unit tests that exercise both the language and the standardlibrary.Any change you make to Python requires a unit test. If your patch is to fix a bug, then a test exercising that bug and your fix is needed.If it is a new feature then an entirely new regression test might beneeded. The important thing is that your new code is exercised within Python's test suite.Add the proper document changes===If your patch will change or add new semantics, you must update thedocumentation. If your code clarifies some ambiguous behaviour, you must update the docs. The only time you do not need to touch thedocumentation is if your patch fixes a bug or adds new unit tests.Make your code follow the style guidelines== `PEP 7`_ and `PEP 8`_ are the style guidelines for C and Python code,respectively. Your code must conform to the style guidelines beforeit is accepted.Generate a patch In general, people get their copy of the Python source from twolocations: a svn checkout or from downloaded source. Both haveslightly different steps in terms of generating patches.Working from a svn checkout is the best way to create a patch. It makes it easier on the Python developers since there is less chance ofany merge problems. It also raises the chances that a developer willbe willing to put the time and effort into working with you and your patch. If you need help with working with svn and Python'srepository, see the `Developer FAQ`_. Working from downloaded sourceis so inferior to working with a svn checkout that it will not bediscussed here (although patches *will* be accepted against downloaded code).First step is to update your svn checkout with ``svn update``. Onceyou have done that and resolved the merge conflicts, generate a patchcontaining the files you have changed. You can see what files you have touched with ``svn status``. If it lists only modified files youwant to include in your patch, you can then generate your patch:: svn diff > PATCH_FILEMake sure to name ``PATCH_FILE`` to a reasonable name and be prepared to upload it in a later step. Also, please note what revision numberyou are working against. This can be found by executing:: python -c "import sys; print sys.version"Create a tracker item on SourceForge First make sure you have a `SourceForge account`_. Once you have oneand havve logged in, go to the `SF patch tracker`_ for Python andcreate a new tracker item. In the new tracker item, set the item's fields as so:* Private Leave unchecked.* Category Set to what area of Python your patch affects.* Group Set to the version of Python that the patch was generated against. * Assigned To Leave as-is.* Priority: Leave as-is.* Summary Briefly describe what the patch does.* Detailed Description Describe in more detail what the patch does. Make sure to mention the revision number the patch is against here. Also mention any bugs
[Python-Dev] Need help with test_mutants.py
There's a unit test "test_mutants" which I don't understand. If anyone remembers what it's doing, please contact me -- after ripping out dictionary ordering in Py3k, it stops working. In particular, the code in test_one() requires changes, but I don't know how... Please help! -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] draft for bug guidelines
On Thu, Aug 24, 2006 at 12:21:06PM -0700, Brett Cannon wrote: > Start a new bug "Before starting a new bug please try to search if the bug has already been reported. It it has - do not start a new report, add your comments to the existing report." Oleg. -- Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] draft of patch guidelines
On Thu, Aug 24, 2006 at 12:22:42PM -0700, Brett Cannon wrote: > Read the Developer Intro to understand the scope of your proposed change "Search through the PEPs, developer mailing lists and patches. Has a similar patch already been proposed? Has it been accepted, postponed or rejected? Learn the reasons why it's been postponed or rejected." Oleg. -- Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Need help with test_mutants.py
[Guido] > There's a unit test "test_mutants" which I don't understand. If anyone > remembers what it's doing, please contact me -- after ripping out > dictionary ordering in Py3k, Is any form of dictionary comparison still supported, and, if so, what does "dict1 cmp_op dict2" mean now? > it stops working. Traceback? > In particular, the code in test_one() requires changes, but I don't > know how... Please help! The keys and values of dict1 and dict2 are filled with objects of a user-defined class whose __cmp__ method randomly mutates dict1 and dict2. dict1 and dict2 are initially forced to have the same number of elements, so in current Python: c = cmp(dict1, dict2) triggers a world of pain, with the internal dict code doing fancy stuff comparing keys and values. However, every key and value comparison /may/ mutate the dicts in arbitrary ways, so this is testing whether the dict comparison implementation blows up (segfaults, etc) when the dicts it's comparing mutate during comparison. If it's only ordering comparisons that have gone away for dicts, then, e.g., replacing c = cmp(dict1, dict2) with c = dict1 == dict2 instead will still meet the test's intent. No particular /result/ is expected. The test passes if and only if Python doesn't crash. When the test was introduced, it uncovered at least six distinct failure (crashing) modes across the first 20 times it was run, so it's well worth keeping around in some form. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Need help with test_mutants.py
On 8/24/06, Guido van Rossum <[EMAIL PROTECTED]> wrote: > I made that change, and changed class Horrid to define __eq__ instead > of __cmp__. Since dict_equal() only invokes PyObject_RichCompareBool() > with op==Py_EQ that should be all that's needed. > > Now when I run it, it spits out an apaprently infinite number of dots. > Putting a print in that __eq__ method suggests it is never called. Do > you understand this? > > If I change Horrid.__hash__ to always return 42, I get output like this: > > trying w/ lengths 12 14 > trying w/ lengths 48 52 > trying w/ lengths 19 18 > trying w/ lengths 10 9 > trying w/ lengths 48 46 > trying w/ lengths 58 55 > trying w/ lengths 50 48 > trying w/ lengths 45 50 > trying w/ lengths 19 19 . > Traceback (most recent call last): > File "../Lib/test/test_mutants.py", line 158, in > test(100) > File "../Lib/test/test_mutants.py", line 155, in test > test_one(random.randrange(1, 100)) > File "../Lib/test/test_mutants.py", line 141, in test_one > c = dict1 == dict2 > File "../Lib/test/test_mutants.py", line 99, in __eq__ > return self.i == other.i > AttributeError: 'Horrid' object has no attribute 'i' > Segmentation fault > > But it doesn't always end with a segfault -- most of the time, the > AttributeError is the last thing printed. As a follow up to this story line, this appeared to be a refcount bug in dict_equal(). I believe the same bug is probably present in 2.5; it isn't triggered by test_mutants.py because that only exercises dict_compare, not dict_richcompare, and only the latter can call dict_equal (when op==Py_EQ or op==Py_NE). The bug is here: PyObject *key = a->ma_table[i].me_key; /* temporarily bump aval's refcount to ensure it stays alive until we're done with it */ Py_INCREF(aval); bval = PyDict_GetItem((PyObject *)b, key); The fix is to put Py_INCREF(key) // Py_DECREF(key) around the call to PyDict_GetItem(). Apparently what can happen is that the only reference to the key is in the dict, and the evil mutation from the comparison delete the object before the comparison is completely done with it. Should I attempt to reproduce this bug in 2.5 and fix it? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] draft for bug guidelines
Made it the first step. =)-BrettOn 8/24/06, Oleg Broytmann <[EMAIL PROTECTED]> wrote: On Thu, Aug 24, 2006 at 12:21:06PM -0700, Brett Cannon wrote:> Start a new bug "Before starting a new bug please try to search if the bug has alreadybeen reported. It it has - do not start a new report, add your comments to the existing report."Oleg.-- Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] draft of patch guidelines
On 8/24/06, Oleg Broytmann <[EMAIL PROTECTED]> wrote: On Thu, Aug 24, 2006 at 12:22:42PM -0700, Brett Cannon wrote:> Read the Developer Intro to understand the scope of your proposed change "Search through the PEPs, developer mailing lists and patches. Has a similar patch already been proposed? Has it been accepted, postponed orrejected? Learn the reasons why it's been postponed or rejected."Oleg.-- Oleg Broytmann http://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Need help with test_mutants.py
On 8/24/06, Tim Peters <[EMAIL PROTECTED]> wrote: > [Guido] > > There's a unit test "test_mutants" which I don't understand. If anyone > > remembers what it's doing, please contact me -- after ripping out > > dictionary ordering in Py3k, > > Is any form of dictionary comparison still supported, and, if so, what > does "dict1 cmp_op dict2" mean now? Only == and != are supported between dicts. All the work is done by dict_equal(). > > it stops working. > > Traceback? Not particularly interesting: without changes, the code immediately bombs like this: trying w/ lengths 90 90 . Traceback (most recent call last): File "../Lib/test/test_mutants.py", line 152, in test(100) File "../Lib/test/test_mutants.py", line 149, in test test_one(random.randrange(1, 100)) File "../Lib/test/test_mutants.py", line 135, in test_one c = cmp(dict1, dict2) TypeError: unorderable types: dict() > dict() > > In particular, the code in test_one() requires changes, but I don't > > know how... Please help! > > The keys and values of dict1 and dict2 are filled with objects of a > user-defined class whose __cmp__ method randomly mutates dict1 and > dict2. dict1 and dict2 are initially forced to have the same number > of elements, so in current Python: > > c = cmp(dict1, dict2) > > triggers a world of pain, with the internal dict code doing fancy > stuff comparing keys and values. However, every key and value > comparison /may/ mutate the dicts in arbitrary ways, so this is > testing whether the dict comparison implementation blows up > (segfaults, etc) when the dicts it's comparing mutate during > comparison. > > If it's only ordering comparisons that have gone away for dicts, then, > e.g., replacing > > c = cmp(dict1, dict2) > > with > > c = dict1 == dict2 > > instead will still meet the test's intent. I made that change, and changed class Horrid to define __eq__ instead of __cmp__. Since dict_equal() only invokes PyObject_RichCompareBool() with op==Py_EQ that should be all that's needed. Now when I run it, it spits out an apaprently infinite number of dots. Putting a print in that __eq__ method suggests it is never called. Do you understand this? If I change Horrid.__hash__ to always return 42, I get output like this: trying w/ lengths 12 14 trying w/ lengths 48 52 trying w/ lengths 19 18 trying w/ lengths 10 9 trying w/ lengths 48 46 trying w/ lengths 58 55 trying w/ lengths 50 48 trying w/ lengths 45 50 trying w/ lengths 19 19 . Traceback (most recent call last): File "../Lib/test/test_mutants.py", line 158, in test(100) File "../Lib/test/test_mutants.py", line 155, in test test_one(random.randrange(1, 100)) File "../Lib/test/test_mutants.py", line 141, in test_one c = dict1 == dict2 File "../Lib/test/test_mutants.py", line 99, in __eq__ return self.i == other.i AttributeError: 'Horrid' object has no attribute 'i' Segmentation fault But it doesn't always end with a segfault -- most of the time, the AttributeError is the last thing printed. > No particular /result/ is expected. The test passes if and only if > Python doesn't crash. When the test was introduced, it uncovered at > least six distinct failure (crashing) modes across the first 20 times > it was run, so it's well worth keeping around in some form. Well, it looks like it did provoke another crash, so I'll play with it some more. Thanks! -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Need help with test_mutants.py
On 8/24/06, Guido van Rossum <[EMAIL PROTECTED]> wrote: > Should I attempt to reproduce this bug in 2.5 and fix it? Couldn't help myself. The fix is python.org/sf/1546288 . I set the priority to 8 which means Neal and Anthony will look at it. It's probably okay to reduce the priority to 7 and fix it in 2.5.1. I suspect pre-2.5 versions may have the same bug, as long as they have dict_equal(). -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] zip -> izip; is __length_hint__ required?
At today's sprint, Brian Holmes contributed a patch that implements zip as an interator, a la izip. When reviewing Brian's code, I noticed that he added an implementation of __length_hint__. My gut feeling is that this isn't particularly useful given that zip() is almost exclusively used iteratively, and rarely if ever converted to a list or a tuple. (The one common exception is in the test suite, but there it's almost always a short list, and 3 out of 5 were actually tests for zip or izip.) Should we rip it out or keep it? Also, the existing convention for calling __length_hint__ (e.g. in _PyObject_LengthHint() in abstract.c) seems to be to use PyObject_CallMethod() and suppress TypeError and AttributeError coming out of the call. It would seem to make much more sense to check whether the attribute exists without calling it, and once it exists, just call it and not suppress any exceptions that come out of it. Is there any reason why this shouldn't work? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] draft of patch guidelines
Brett, > Below is a draft for a set of patch guidelines. Thanks for getting around to this! > Wait for a developer to contact you > === > > At this point you need to wait for a Python developer to come along > and look at your patch. This might be a while as all Python > developers are volunteers and thus have a limited amount of time. Do > not be surprised if it takes weeks or more before your patch is > examined. > > There is a standing offer that if you review five other patches (by > commenting on its correctness, helping it meet these guidelines, > etc.), then a developer will definitely have a look at any patch you > specify. Just email python-dev with what five patches you reviewed > (and what you did for the review) and the patch you want examined. May I suggest the following wording for this section? This is basically the feedback I found most helpful a few weeks ago; it's more specific about how notification and the 5-for-1 rule work. = When you submit your patch, the tracker notifies a mailing list that most core Python developers subscribe to. But remember that Open Source runs on volunteerism and meritocracy. If you are a new Python contributor, then your patch may languish for weeks or more before anyone volunteers a response. Python addresses this barrier to entry with a "5-for-1 rule." Pick five other patches or bugs, try to apply or reproduce them, and then report your results back to the tracker. Then, send a note to python-dev listing the five you reviewed and the patch id of your own patch, and one of Python's "patch angels" will get the ball rolling on your patch. = chad ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] draft of patch guidelines
On 8/24/06, Chad Whitacre <[EMAIL PROTECTED]> wrote: Brett, > Below is a draft for a set of patch guidelines.Thanks for getting around to this!Welcome. > Wait for a developer to contact you > === > > At this point you need to wait for a Python developer to come along > and look at your patch. This might be a while as all Python > developers are volunteers and thus have a limited amount of time. Do > not be surprised if it takes weeks or more before your patch is > examined. > > There is a standing offer that if you review five other patches (by > commenting on its correctness, helping it meet these guidelines, > etc.), then a developer will definitely have a look at any patch you > specify. Just email python-dev with what five patches you reviewed > (and what you did for the review) and the patch you want examined.May I suggest the following wording for this section? This is basicallythe feedback I found most helpful a few weeks ago; it's more specific about how notification and the 5-for-1 rule work.=When you submit your patch, the tracker notifies a mailing list thatmost core Python developers subscribe to. But remember that Open Source runs on volunteerism and meritocracy. If you are a new Pythoncontributor, then your patch may languish for weeks or more beforeanyone volunteers a response.Python addresses this barrier to entry with a "5-for-1 rule." Pick five other patches or bugs, try to apply or reproduce them, and then reportyour results back to the tracker. Then, send a note to python-devlisting the five you reviewed and the patch id of your own patch, andone of Python's "patch angels" will get the ball rolling on your patch. =Changed the wording slightly, but I mostly ended up using your text:When you submit your patch, the tracker notifies a mailing list thatmost core Python developers subscribe to of the creation of your new patch. But remember that Open Source runs on volunteerism and meritocracy. Your patch may languish for weeks or more before anyone volunteers a response because of limited time on the part of Python developers. Python addresses this barrier to entry with a "5-for-1 rule." Pickfive other patches or bugs, review them, and then report your results back to the tracker. Then, send a note to python-dev listing the five you reviewed and the patch id of your own patch, and a Python developer will take a look at your patch.-Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
