[Python-Dev] PEP: __source__ proposal
Hi all. Now that 2.4 is out and everything maybe it's about time to start discussing the "use the __source__ Luke" feature which IMO will really boost python into a new domain of exciting possibilities. I've prepared a pre-PEP which is not very good but it is a base. In short, the feature is good and it enables editing of python code at runtime instead of the runfile-exit-edit-run-exit-edit-run cycle. We have the following possibilities as to whether __source__ data is marshalled and the feature is always enabled. [1] Command line switch and not marshalled [2] Always on and not marshalled [3] Always on and marshalled There is also [4] which doesn't make much sense. If I was BDFL I'd go for [1] so whoever wants it can enable it and whoever doesn't can't complain, and they'll all leave me alone. Phillip J. Eby expressed some concerns that the modules that depend on __source__ will eventually take over and it will become a standard. Anyway, the PEP is attached. You can mail me with votes on the feature and if you want on your preferred option from 1,2,3. If I get votes I'll post the results later. If this is accepted I'll try to come up with a good patch vs 2.4 Thanks, St. ---ATTACHED PYTHON ENHANCEMENT PROPOSAL--- PEP: XXX Title: The __source__ attribute Version: $Revision: 1.10 $ Last-Modified: $Date: 2003/09/22 04:51:49 $ Author: Stelios Xanthakis Status: Draft Type: Standards Track Content-Type: text/plain Created: 19-Nov-2004 Python-Version: 2.4.1 Post-History: Abstract This PEP suggests the implementation of __source__ attribute for functions and classes. The attribute is a read-only string which is generated by the parser and is a copy of the original source code of the function/class (including comments, indentation and whitespace). Motivation It is generally a tempting idea to use python as an interface to a program. The developers can implement all the functionality and instead of designing a user interface, provide a python interpreter to their users. Take for example one of the existing web browsers: they have everything that would be needed to write a script which downloads pages automatically or premutates the letters of web pages before they are displayed, but it is not possible for the user to do these things because the interface of these applications is static. A much more powerful approach would be an interface which is dynamically constructed by the user to meet the user's needs. The most common development cycle of python programs is: write .py file - execute .py file - exit - enhance .py file - execute .py file - etc. With the implementation of the __source__ attribute though the development/modification of python code can happen at run-time. Functions and classes can be defined, modified or enhanced while the python shell is running and all the changes can be saved by saving the __source__ attribute of globals before termination. Moreover, in such a system it is possible to modify the "code modification routines" and eventually we have a self-modifying interface. Using a program also means improving its usability. The current solution of using 'inspect' to get the source code of functions is not adequate because it doesn't work for code defined with "exec" and it doesn't have the source of functions/classes defined in the interactive mode. Generally, a "file" is something too abstract. What is more real is the data received by the python parser and that is what is stored in __source__. Specification The __source__ attribute is a read-only attribute of functions and classes. Its type is string or None. In the case of None it means that the source was not available. The indentation of the code block is the original identation obeying nested definitions. For example: >>> class A: ... def foo (self): ... print """Santa-Clauss ... is coming to town""" >>> def spam (): ... def closure (): ... pass ... return closure >>> print A.foo.__source__ def foo (self): print """Santa-Clauss is coming to town""" >>> print spam().__source__ def closure (): pass The attribute is not marshaled and therefore not stored in ".pyc" files. As a consequence, functions and classes of imported modules have __source__==None. We propose that the generation of __source__ will be controlled by a command line option. In the case this feature is not activated by the command line option, the attribute is absent. Rationale Generally, "import" refers to modules that either have a file in a standard location or they are distributed in ".pyc" form only. Therefore in the case of modules, getting the source with "inspect" is adequate
Re: [Python-Dev] PEP: __source__ proposal
Hi Stelios, [Stelios Xanthakis Fri, Dec 03, 2004 at 11:54:25AM +0200] > Abstract > > This PEP suggests the implementation of __source__ attribute for > functions and classes. The attribute is a read-only string which > is generated by the parser and is a copy of the original source > code of the function/class (including comments, indentation and > whitespace). I've had similar ideas in the past as we are doing dynamic code generation in PyPy, as well as in other projects. After some discussion with Armin i think there is another possibility for storing "source" or any other such meta information with code/module objects: make __file__ and co_filename instances of a subclass of 'str' providing an extra attribute. For a simple example, they could have a 'source' attribute, which could be tried first by appropriate inspect functions and traceback related functionality. We are about to test out this approach with the py lib (http://codespeak.net/py) and want to have it work for for Python 2.2, 2.3. and 2.4. I suspect there may be some issues lurking (also in your proposed PEP) especially with respect to encodings. Also we have some use cases where we want to retrieve source code from non-local locations and want to integrate this seemlessly with the introspection facilities of Python which obviously is an important part of the equation. I can report back if there is interest. cheers, holger ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: Unicode in doctests
Bjorn Tillenius wrote: > There are some issues regarding the use of unicode in doctests. Consider > the following three tests. > >>>> foo = u'föö' >>>> foo >u'f\xf6\xf6' > >>>> foo >u'f\u00f6\u00f6' > >>>> foo >u'föö' > > To me, those are identical. really? if a function is expected to print "öl", is it alright to print "\u00f6l" instead? wouldn't you complain if your newspaper used Unicode escapes in headlines instead of Swedish characters? > Is it supposed to be like this, or have I missed something? If I could > specify the encoding for DocFileSuite to use, I would at least be > partially happy. repr() always generates the same output, no matter what encoding you use. just use repr, and you're done. ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Removing --with-wctype-functions support
I would like to remove the support for using libc wctype functions (e.g. towupper(), towlower(), etc.) from the code base. The reason is that compiling Python using this switch not only breaks the test suite, it also causes the functions .lower() and .upper() to become locale aware and creates wrong results as a result (which are hard to track down if you don't know whether Python was compiled with the wctype switch or not). The argument for having this switch at the time was to reduce the interpreter size. This was true for Python 1.6 since the Unicode type database was conditionally removed. Starting with Python 2.0 a different approach was taken which resulted in having to keep the type database in one piece - even with the switch enabled. As a result, performance became the only argument. However, because the libc functions are locale aware, the added overhead causes the libc functions not to perform better than the builtin Python versions. In the end, we loose complexity and reduce the Python configuration space by one dimension. Here's the bug that revealed the problem: https://sourceforge.net/tracker/index.php?func=detail&aid=1076790&group_id=5470&atid=105470 Question is: Should this be done in 2.4.1 or will it have to wait until 2.5 ? We might want to remove the support for 2.4.1 and simply ignore the compile time switch (or print a warning). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 03 2004) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] MS VC compiler versions
Here's an update of the list I currently have: * MS VS6 SP5 (International version, optimizing VC6 compiler with SP5): Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8168 for 80x86 Copyright (C) Microsoft Corp 1984-1998. All rights reserved. * MS VC6 (German version; optimizing VC6 compiler with SP6): Optimierender Microsoft (R) 32-Bit C/C++-Compiler, Version 12.00.8804, fuer x86 Copyright (C) Microsoft Corp 1984-1998. Alle Rechte vorbehalten. * MS VC6 (US version; optimizing VC6 compiler with SP6): Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8804 for 80x86 Copyright (C) Microsoft Corp 1984-1998. All rights reserved. * MS .NET SDK 1.1 Compiler (German version; non-optimizing VC7.1 compiler): Microsoft (R) 32-Bit C/C++-Standardcompiler Version 13.10.3077 für 80x86 Copyright (C) Microsoft Corporation 1984-2002. Alle Rechte vorbehalten. * MS VC7.1 (aka .NET 2003, German version; optimizing VC7.1 compiler) Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 13.10.3077 for 80x86 Copyright (C) Microsoft Corporation 1984-2002. All rights reserved. * MS Windows XP DDK (International version, optimizing VC 7.0): Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 13.00.9176 for 80x86 Copyright (C) Microsoft Corporation 1984-2001. All rights reserved. Does someone have the details for the MS Toolkit compiler ? Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 03 2004) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: MS VC compiler versions
M.-A. Lemburg wrote: Here's an update of the list I currently have: * MS VS6 SP5 (International version, optimizing VC6 compiler with SP5): Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8168 for 80x86 Copyright (C) Microsoft Corp 1984-1998. All rights reserved. * MS VC6 (German version; optimizing VC6 compiler with SP6): Optimierender Microsoft (R) 32-Bit C/C++-Compiler, Version 12.00.8804, fuer x86 Copyright (C) Microsoft Corp 1984-1998. Alle Rechte vorbehalten. * MS VC6 (US version; optimizing VC6 compiler with SP6): Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8804 for 80x86 Copyright (C) Microsoft Corp 1984-1998. All rights reserved. * MS .NET SDK 1.1 Compiler (German version; non-optimizing VC7.1 compiler): Microsoft (R) 32-Bit C/C++-Standardcompiler Version 13.10.3077 für 80x86 Copyright (C) Microsoft Corporation 1984-2002. Alle Rechte vorbehalten. * MS VC7.1 (aka .NET 2003, German version; optimizing VC7.1 compiler) Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 13.10.3077 for 80x86 Copyright (C) Microsoft Corporation 1984-2002. All rights reserved. * MS Windows XP DDK (International version, optimizing VC 7.0): Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 13.00.9176 for 80x86 Copyright (C) Microsoft Corporation 1984-2001. All rights reserved. Does someone have the details for the MS Toolkit compiler ? Thanks, \Program Files\Microsoft Visual C++ Toolkit 2003\bin\cl.exe: Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 13.10.3077 for 80x86 Copyright (C) Microsoft Corporation 1984-2002. All rights reserved. -- -- Scott David Daniels [EMAIL PROTECTED] ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: Unicode in doctests
Fredrik Lundh pythonware.com> writes: > Bjorn Tillenius wrote: > > There are some issues regarding the use of unicode in doctests. Consider > > the following three tests. > > > >>>> foo = u'fÃÃ' > >>>> foo > >u'f\xf6\xf6' > > > >>>> foo > >u'f\u00f6\u00f6' > > > >>>> foo > >u'fÃÃ' > > > > To me, those are identical. > > really? if a function is expected to print "Ãl", is it alright to print > "\u00f6l" instead? wouldn't you complain if your newspaper used > Unicode escapes in headlines instead of Swedish characters? No, I wouldn't like the newspaper to use Unicode escapes. For the same reason, I don't want my documentation to contain Unicode escapes. That's why I would like the latter test to pass. But I understand, it tries to match the output of repr(foo), I guess I can live with that. I can always do: >>> foo == u'fÃÃ' True On the other hand, since there already are some flags to modify the matching algorithm, one could argue for adding another flag... or at least provide the possibility for the user to alter the matching himself. Although it's not that important for me. > > Is it supposed to be like this, or have I missed something? If I could > > specify the encoding for DocFileSuite to use, I would at least be > > partially happy. > > repr() always generates the same output, no matter what encoding > you use. just use repr, and you're done. What is important for me, though, is to be able to specify an encoding to DocFileSuite. As you said, one doesn't want to read Unicode escapes. At the moment none of the tests I've given as example will pass in a DocFileSuite (given that the text file is encoded using UTF-8). I do find it a bit strange that I can't just copy a doctest within a docstring to a text file. I have to Unicode escape all non-ASCII characters, which produces ugly documentation. Regards, Bjorn ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SRE bug and notifications
"A.M. Kuchling" <[EMAIL PROTECTED]> writes: > Perhaps the summaries should include an "unassigned bugs" list to nag > us to look at bugs and assign them to the right person. The bug report is derived from the bug and patch email lists, so that information isn't available without scraping it off SF. However, the SF trackers can be set to query for Unassigned / Open. It looks like about 2/3 of the open bugs and 3/4 of the open patches are unassigned. I often unassign IDLE bugs that I'm not planning to fix right now, hoping that may encourage someone to step forward by removing the illusion that something is happening. Conversely, if I make progress on an unassigned bug/patch, I'll assign it to myself as an indication that it's being worked on, avoiding duplication of effort. I track IDLE bugs via the IDLE category. That's very useful, maybe we need more categories? -- KBK ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: __source__ proposal
On Fri, 3 Dec 2004, holger krekel wrote: ... there is another possibility for storing "source" or any other such meta information with code/module objects: make __file__ and co_filename instances of a subclass of 'str' providing an extra attribute. For a simple example, they could have a 'source' attribute, which could be tried first by appropriate inspect functions and traceback related functionality. Attaching such info on 'code objects' is indeed a more general case. But, OTOH, AFAIK, a class is not a code object. At least by what I was able to figure out from python sources. It seems reasonable to make 'source' a dynamic object which will get its info from file/line if available. Now the thing is that if we had __source__ from the start, 'inspect' would have been much different. So the fact that we have some functionality with inspect does not mean that it's good enough. Probably inspect will be rewritten/improved if __source__ is implemented. We are about to test out this approach with the py lib (http://codespeak.net/py) and want to have it work for for Python 2.2, 2.3. and 2.4. Do you plan hacking python ? It appears that tok_nextc() is the best place to catch all the source passed to the interpreter. A patch would be interesting. Stelios ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: __source__ proposal
Stelios Xanthakis wrote: Now that 2.4 is out and everything maybe it's about time to start discussing the "use the __source__ Luke" feature which IMO will really boost python into a new domain of exciting possibilities. I'm opposed to this idea. It creates overhead in the size of .pyc files, for no additional value that couldn't be obtained otherwise. As the rationale, the PEP lists: 1. It is generally a tempting idea to use python as an interface to a program. I cannot see how this rationale is related to the PEP. You can use Python as interface to a program with or without __source__. 2. The developers can implement all the functionality and instead of designing a user interface, provide a python interpreter to their users. This does not require __source, either. 3. A much more powerful approach would be an interface which is dynamically constructed by the user to meet the user's needs. Dynamic code generation doesn't require __source__, either. 4. The most common development cycle of python programs is: write .py file - execute .py file - exit - enhance .py file - execute .py file - etc. With the implementation of the __source__ attribute though the development/modification of python code can happen at run-time. This works just fine as well at the moment; see IDLE for an example. Functions and classes can be defined, modified or enhanced while the python shell is running and all the changes can be saved by saving the __source__ attribute of globals before termination. Currently, you can define classes dynamically, and you can also save the source code of the class to a file in case you need it later. > Moreover, in such a system it is possible to modify the "code modification routines" and eventually we have a self-modifying interface. Using a program also means improving its usability. Self-modifying source code is currently also possible. Just read the old source code from a .py file, modify it, and recompile it. The current solution of using 'inspect' to get the source code of functions is not adequate because it doesn't work for code defined with "exec" and it doesn't have the source of functions/classes defined in the interactive mode. I fail to see why it isn't adequate. Anybody who wants to modify source code that was originally passed to exec just needs to preserve a copy of the source code, separately. > Generally, a "file" is something too abstract. What is more real is the data received by the python parser and that is what is stored in __source__. Not at all. A file is precisely the level of granularity that is burnt into the Python language. A module is *always* a file, executed from top to bottom. It is not possible to recreate the source code of a module if you have only the source code of all functions, and all classes. Regards, Martin ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] test_shutils.py fails for 2.4 install
I have a PC with an AMD cpu and Mandrake 10.1. While installing Python 2.4 "make test" failed in "test_shutils.py". Here is the output of "./python ./Lib/test/test_shutil.py": test_dont_copy_file_onto_link_to_itself (__main__.TestShutil) ... ok test_dont_move_dir_in_itself (__main__.TestShutil) ... ok test_on_error (__main__.TestShutil) ... FAIL test_rmtree_dont_delete_file (__main__.TestShutil) ... ok test_rmtree_errors (__main__.TestShutil) ... ok == FAIL: test_on_error (__main__.TestShutil) -- Traceback (most recent call last): File "./Lib/test/test_shutil.py", line 34, in test_on_error self.assertEqual(self.errorState, 2) AssertionError: 0 != 2 -- Ran 5 tests in 0.005s FAILED (failures=1) Traceback (most recent call last): File "./Lib/test/test_shutil.py", line 106, in ? test_main() File "./Lib/test/test_shutil.py", line 103, in test_main test_support.run_unittest(TestShutil) File "/usr/local/src/Python-2.4/Lib/test/test_support.py", line 290, in run_u run_suite(suite, testclass) File "/usr/local/src/Python-2.4/Lib/test/test_support.py", line 275, in run_s raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "./Lib/test/test_shutil.py", line 34, in test_on_error self.assertEqual(self.errorState, 2) AssertionError: 0 != 2 ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP: __source__ proposal
Hi! I think you were the omitting the more interesting closure examples (namely with free variables inside the closure's source): def foo(): pass def bar(x): def fiep(): return x() return fiep what's bar(foo).__source__? Generally, I'm opposed to the feature -- I don't think it provides a real advantage giving it's limitations (doesn't work for import'ed modules, classes + methods != module, etc.). Cheers, Michael On Fri, 3 Dec 2004 11:54:25 +0200 (EET), Stelios Xanthakis <[EMAIL PROTECTED]> wrote: > > Hi all. > > Now that 2.4 is out and everything maybe it's > about time to start discussing the "use the > __source__ Luke" feature which IMO will really > boost python into a new domain of exciting > possibilities. > > I've prepared a pre-PEP which is not very good > but it is a base. > > In short, the feature is good and it enables > editing of python code at runtime instead of > the runfile-exit-edit-run-exit-edit-run cycle. > > We have the following possibilities as to whether > __source__ data is marshalled and the feature is > always enabled. > > [1] Command line switch and not marshalled > [2] Always on and not marshalled > [3] Always on and marshalled > > There is also [4] which doesn't make much sense. > > If I was BDFL I'd go for [1] so whoever wants it > can enable it and whoever doesn't can't complain, > and they'll all leave me alone. > Phillip J. Eby expressed some concerns that the > modules that depend on __source__ will eventually > take over and it will become a standard. > > Anyway, the PEP is attached. > You can mail me with votes on the feature and if you > want on your preferred option from 1,2,3. > If I get votes I'll post the results later. > > If this is accepted I'll try to come up with a good > patch vs 2.4 > > Thanks, > > St. > > ---ATTACHED PYTHON ENHANCEMENT PROPOSAL--- > PEP: XXX > Title: The __source__ attribute > Version: $Revision: 1.10 $ > Last-Modified: $Date: 2003/09/22 04:51:49 $ > Author: Stelios Xanthakis > Status: Draft > Type: Standards Track > Content-Type: text/plain > Created: 19-Nov-2004 > Python-Version: 2.4.1 > Post-History: > > Abstract > > This PEP suggests the implementation of __source__ attribute for > functions and classes. The attribute is a read-only string which > is generated by the parser and is a copy of the original source > code of the function/class (including comments, indentation and > whitespace). > > Motivation > > It is generally a tempting idea to use python as an interface to > a program. The developers can implement all the functionality > and instead of designing a user interface, provide a python > interpreter to their users. Take for example one of the existing > web browsers: they have everything that would be needed to write > a script which downloads pages automatically or premutates the > letters of web pages before they are displayed, but it is not > possible for the user to do these things because the interface > of these applications is static. > > A much more powerful approach would be an interface which is > dynamically constructed by the user to meet the user's needs. > The most common development cycle of python programs is: > write .py file - execute .py file - exit - enhance .py file - > execute .py file - etc. With the implementation of the __source__ > attribute though the development/modification of python code > can happen at run-time. Functions and classes can be defined, > modified or enhanced while the python shell is running and > all the changes can be saved by saving the __source__ attribute > of globals before termination. Moreover, in such a system > it is possible to modify the "code modification routines" and > eventually we have a self-modifying interface. Using a > program also means improving its usability. > > The current solution of using 'inspect' to get the source > code of functions is not adequate because it doesn't work > for code defined with "exec" and it doesn't have the source > of functions/classes defined in the interactive mode. Generally, > a "file" is something too abstract. What is more real is the > data received by the python parser and that is what is stored > in __source__. > > Specification > > The __source__ attribute is a read-only attribute of functions > and classes. Its type is string or None. In the case of None > it means that the source was not available. > > The indentation of the code block is the original identation > obeying nested definitions. For example: > > >>> class A: > ... def foo (self): > ... print """Santa-Clauss > ... is coming to town""" > >>> def spam (): > ... def closure (): > ... pass > ... return closure > >>> print A.
