Hi all.
Now that 2.4 is out and everything maybe it's
about time to start discussing the "use the
__source__ Luke" feature which IMO will really
boost python into a new domain of exciting
possibilities.
I've prepared a pre-PEP which is not very good
but it is a base.
In short, the feature is good and it enables
editing of python code at runtime instead of
the runfile-exit-edit-run-exit-edit-run cycle.
We have the following possibilities as to whether
__source__ data is marshalled and the feature is
always enabled.
[1] Command line switch and not marshalled
[2] Always on and not marshalled
[3] Always on and marshalled
There is also [4] which doesn't make much sense.
If I was BDFL I'd go for [1] so whoever wants it
can enable it and whoever doesn't can't complain,
and they'll all leave me alone.
Phillip J. Eby expressed some concerns that the
modules that depend on __source__ will eventually
take over and it will become a standard.
Anyway, the PEP is attached.
You can mail me with votes on the feature and if you
want on your preferred option from 1,2,3.
If I get votes I'll post the results later.
If this is accepted I'll try to come up with a good
patch vs 2.4
Thanks,
St.
-------------------ATTACHED PYTHON ENHANCEMENT PROPOSAL---
PEP: XXX
Title: The __source__ attribute
Version: $Revision: 1.10 $
Last-Modified: $Date: 2003/09/22 04:51:49 $
Author: Stelios Xanthakis
Status: Draft
Type: Standards Track
Content-Type: text/plain
Created: 19-Nov-2004
Python-Version: 2.4.1
Post-History:
Abstract
This PEP suggests the implementation of __source__ attribute for
functions and classes. The attribute is a read-only string which
is generated by the parser and is a copy of the original source
code of the function/class (including comments, indentation and
whitespace).
Motivation
It is generally a tempting idea to use python as an interface to
a program. The developers can implement all the functionality
and instead of designing a user interface, provide a python
interpreter to their users. Take for example one of the existing
web browsers: they have everything that would be needed to write
a script which downloads pages automatically or premutates the
letters of web pages before they are displayed, but it is not
possible for the user to do these things because the interface
of these applications is static.
A much more powerful approach would be an interface which is
dynamically constructed by the user to meet the user's needs.
The most common development cycle of python programs is:
write .py file - execute .py file - exit - enhance .py file -
execute .py file - etc. With the implementation of the __source__
attribute though the development/modification of python code
can happen at run-time. Functions and classes can be defined,
modified or enhanced while the python shell is running and
all the changes can be saved by saving the __source__ attribute
of globals before termination. Moreover, in such a system
it is possible to modify the "code modification routines" and
eventually we have a self-modifying interface. Using a
program also means improving its usability.
The current solution of using 'inspect' to get the source
code of functions is not adequate because it doesn't work
for code defined with "exec" and it doesn't have the source
of functions/classes defined in the interactive mode. Generally,
a "file" is something too abstract. What is more real is the
data received by the python parser and that is what is stored
in __source__.
Specification
The __source__ attribute is a read-only attribute of functions
and classes. Its type is string or None. In the case of None
it means that the source was not available.
The indentation of the code block is the original identation
obeying nested definitions. For example:
>>> class A:
... def foo (self):
... print """Santa-Clauss
... is coming to town"""
>>> def spam ():
... def closure ():
... pass
... return closure
>>> print A.foo.__source__
def foo (self):
print """Santa-Clauss
is coming to town"""
>>> print spam().__source__
def closure ():
pass
The attribute is not marshaled and therefore not stored in
".pyc" files. As a consequence, functions and classes of
imported modules have __source__==None.
We propose that the generation of __source__ will be
controlled by a command line option. In the case this
feature is not activated by the command line option, the
attribute is absent.
Rationale
Generally, "import" refers to modules that either have a file in
a standard location or they are distributed in ".pyc" form only.
Therefore in the case of modules, getting the source with
"inspect" is adequate. Moreover, it does not make sense saving
__source__ in ".pyc" because the point would be to save
modifications in the original ".py" file (if available).
On the issue of the command-line option controlling the generation
of __source__, please refer to the section about the overhead
of this feature. The rationale is that those applications that
do not wish to use this feature can avoid it (cgi scripts in
python benchmarked against another language).
Overhead
The python's parser is not exactly well-suited for such a feature.
Execution of python code goes through the stages of lexical
analysis, tokenization, generation of AST and execution of
bytecode. In order to implement __source__, the tokenizer has
to be modified to store the lines of the current translation
unit. Those lines are then attached the root node of the
AST. While the AST is compiled we have to keep a reference
of the current node in order to be able to find the next node
after the node for which we wish to generate __source__, get
the first and the last line of our block and then refer to
the root node to extract these lines and make a string. All
these actions add a minor overhead to some heavily optimized
parts of python. However, once compilation to bytecode is
done, this feature no longer affects the performance of the
execution of the bytecode.
There is also the issue of the memory spent to store __source__.
In our opinion, this is worth the tradeoff for those who
are willing to take advantage of it.
Implementation
There is a sample implementation at [2] which consists of a
patch against python 2.3.4. The patch has to be improved
to avoid generating __source__ for the case we are importing
modules for the first time (not from .pyc). In the sample
implementation there is also included a sample shell that
takes advantage of __source__ and demonstrates some aspects
that motivated us towards patching python and submitting this
PEP.
References
[1] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton
http://www.python.org/peps/pep-0001.html
[2] Sample implementation
http://students.ceid.upatras.gr/~sxanth/ISYSTEM/python-PIESS.tar.gz
Copyright
This document has been placed in the public domain.
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
End:
_______________________________________________
Python-Dev mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
