Re: [Python-Dev] Improving the Python Memory Allocator

["Martin v. Löwis"]

Evan Jones wrote:
Sure. This should be done even for patches which should absolutely not 
be committed?
Difficult question. I think the answer goes like this: "Patches that
should absolutely not be committed should not be published at all".
There are different shades of gray, of course - but people typically
dislike receiving patches through a mailing list.
OTOH, I'm guilty of committing a patch myself which was explicitly
marked as not-to-be-committed on SF, so I cannot really advise to
use SF in this case. Putting it on your own web server would be
best.
Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Speed up function calls

[Neal Norwitz]

On Wed, 26 Jan 2005 09:47:41 -0500, Raymond Hettinger <[EMAIL PROTECTED]> wrote:
> > I agree that METH_O and METH_NOARGS are near
> > optimal wrt to performance.  But if we could have one METH_UNPACKED
> > instead of 3 METH_*, I think that would be a win.
>  . . .
> > Sorry, I meant eliminated w/3.0.
> 
> So, leave METH_O and METH_NOARGS alone.  They can't be dropped until 3.0
> and they can't be improved speedwise.

I was just trying to point out possible directions.  I wasn't trying
to suggest that the patch as a whole should be integrated now.

> > Ultimately, I think we can speed things up more by having 9 different
> > op codes, ie, one for each # of arguments.  CALL_FUNCTION_0,
> > CALL_FUNCTION_1, ...
> > (9 is still arbitrary and subject to change)
> 
> How is the compiler to know the arity of the target function?  If I call
> pow(3,5), how would the compiler know that pow() can take an optional
> third argument which would be need to be initialized to NULL?

The compiler wouldn't know anything about pow().  It would only know
that 2 arguments are passed.  That would help get rid of the first
switch statement.
I need to think more about the NULL initialization.  I may have mixed
2 separate issues.

> > Then we would have N little functions, each with the exact # of
> > parameters.  Each would still need a switch to call the C function
> > because there may be optional parameters.  Ultimately, it's possible
> > the code would be small enough to stick it into the eval_frame loop.
> > Each of these steps would need to be tested, but that's a possible
> > longer term direction.
>  . . .
> > There would only be an if to check if it was a C function or not.
> > Maybe we could even get rid of this by more fixup at import time.
> 
> This is what I mean about the patch taking on a life of its own.  It's
> an optimization patch that slows down METH_O and METH_NOARGS.  It's a
> incremental change that throws away backwards compatibility.  It's a
> simplification that introduces a bazillion new code paths.  It's a
> simplification that can't be realized until 3.0.  It's a minor change
> that entails new opcodes, compiler changes, and changes in all
> extensions that have ever been written.

I really didn't want to do this now (or necessarily in 2.5).  I was
just trying to provide insight into future direction.  This brings up
another discussion about working towards 3.0.  But I'll make a new
thread for that.

At this point, it seems there aren't many disagreements about the
general idea.  There is primarily a question about what is acceptable
now.  I will rework the patch based on Raymond's feedback and continue
update the tracker.  Unless if anyone disagrees, I don't see a reason
to continue the remainder of this discussion on py-dev.

Neal
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Moving towards Python 3.0 (was Re: [Python-Dev] Speed up function calls)

[Neal Norwitz]

On Wed, 26 Jan 2005 09:47:41 -0500, Raymond Hettinger <[EMAIL PROTECTED]> wrote:
>
> This is what I mean about the patch taking on a life of its own.  It's
> an optimization patch that slows down METH_O and METH_NOARGS.  It's a
> incremental change that throws away backwards compatibility.  It's a
> simplification that introduces a bazillion new code paths.  It's a
> simplification that can't be realized until 3.0.

I've been thinking about how to move towards 3.0.  There are many
changes that are desirable and unlikely to occur prior to 3.0.  But if
we defer so many enhancments, the changes will be voluminous,
potentially difficult to manage, and possibly error prone.  There is a
risk that many small warts will not be fixed, only because they fell
through the cracks.

I thought about making a p3k branch in CVS.  It could be worked on
slowly and would be the implementation of PEP 3000.  However, if a
branch was created all changes would need to be forward ported to it
and it would need to be kept up to date.  I know I wouldn't have
enough time to maintain this.

The benefit is that people could test the portability of their
applications with 3.0 sooner rather than later.  They could see if the
switch to iterators created problems, or integer division, or
new-style exceptions, etc.  We could try to improve performance by
simplifying architecture.  We could see how much a problem it would be
to (re)move some builtins.

Any ideas how we could start to realize some benefits of Py3.0 before
it arrives?  I'm not sure if this is worth it, if it's premature, or
if there are other ways to acheive the goal of easing transition for
users and simplifying developers tasks (by spreading over a longer
period of time) and reducing the possibility of not fixing warts.

Neal
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Should Python's library modules be written to help the freeze tools?

[Tony Meyer]

The Python 2.4 Lib/bsddb/__init__.py contains this:

"""
# for backwards compatibility with python versions older than 2.3, the
# iterator interface is dynamically defined and added using a mixin
# class.  old python can't tokenize it due to the yield keyword.
if sys.version >= '2.3':
exec """
import UserDict
from weakref import ref
class _iter_mixin(UserDict.DictMixin):
...
"""

Because the imports are inside an exec, modulefinder (e.g. when using bsddb
with a py2exe built application) does not realise that the imports are
required.  (The requirement can be manually specified, of course, if you
know that you need to do so).

I believe that changing the above code to:

"""
if sys.version >= '2.3':
import UserDict
from weakref import ref
exec """
class _iter_mixin(UserDict.DictMixin):
"""

Would still have the intended effect and would let modulefinder do its work.

The main question (to steal Thomas's words) is whether the library modules
should be written to help the freeze tools - if the answer is 'yes', then
I'll submit the above as a patch for 2.5.

Thanks!

=Tony.Meyer

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Should Python's library modules be written to help the freeze tools?

["Martin v. Löwis"]

Tony Meyer wrote:
The main question (to steal Thomas's words) is whether the library modules
should be written to help the freeze tools - if the answer is 'yes', then
I'll submit the above as a patch for 2.5.
The answer to this question certainly is "yes, if possible". In this
specific case, I wonder whether the backwards compatibility is still
required in the first place. According to PEP 291, Greg Smith and
Barry Warsaw decide on this, so I think they would need to comment
first because any patch can be integrated. If they comment that 2.1
compatibility is still desirable, your patch would be fine (I guess);
if they say that the compatibility requirement can be dropped for 2.5,
I suggest that the entire exec statement is removed, along with the
conditional clause.
Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

RE: [Python-Dev] Should Python's library modules be written to help the freeze tools?

[Tony Meyer]

[Tony Meyer]
> The main question (to steal Thomas's words) is whether the 
> library modules should be written to help the freeze tools
> - if the answer is 'yes', then I'll submit the above as a
> patch for 2.5.

[Martin v. Löwis]
> The answer to this question certainly is "yes, if possible". In this
> specific case, I wonder whether the backwards compatibility is still
> required in the first place. According to PEP 291, Greg Smith and
> Barry Warsaw decide on this, so I think they would need to comment
> first because any patch can be integrated.
[...]

Thanks!  I've gone ahead and submitted a patch, in that case:

[ 1112812 ] Patch for Lib/bsddb/__init__.py to work with modulefinder


I realise that neither of the people that need to look at this are part of
the '5 for 1' deal, so I need to wait for one of them to have time to look
at it (plenty of time left before 2.5 anyway) but I'll do 5 reviews for the
karma anyway, today or tomorrow.

=Tony.Meyer

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Should Python's library modules be written to help the freeze tools?

[Bob Ippolito]

On Jan 30, 2005, at 5:50 PM, Martin v. Löwis wrote:

Tony Meyer wrote:
The main question (to steal Thomas's words) is whether the library  
modules
should be written to help the freeze tools - if the answer is 'yes',  
then
I'll submit the above as a patch for 2.5.
The answer to this question certainly is "yes, if possible". In this
specific case, I wonder whether the backwards compatibility is still
required in the first place. According to PEP 291, Greg Smith and
Barry Warsaw decide on this, so I think they would need to comment
first because any patch can be integrated. If they comment that 2.1
compatibility is still desirable, your patch would be fine (I guess);
if they say that the compatibility requirement can be dropped for 2.5,
I suggest that the entire exec statement is removed, along with the
conditional clause.
py2app  handles this situation by  
using a much richer way to analyze module dependencies, in that it can  
use hooks (called "recipes") to trigger arbitrary behavior when the  
responsible recipe sees that a certain module is in the dependency  
graph.  This is actually necessary to do the right thing in the case of  
extensions and modules that are not friendly with bytecode analysis.   
Though there are not many of these in the standard library, a few  
common packages such as PIL have a real need for this.  Also, since  
modulegraph uses a graph data structure it is much better suited to  
pruning the dependency graph.  For example, pydoc imports Tkinter to  
support an obscure feature, but this is almost never desired in the  
context of an application freeze tool.  py2app ships with a recipe that  
automatically breaks the edge between pydoc and Tkinter  
, so if Tkinter is not explicitly included or used by  
anything else in the dependency graph, it is correctly excluded from  
the resultant application bundle.

In order to correctly cover the Python API, I needed to ALWAYS include:  
unicodedata, warnings, encodings, and weakref because they can be used  
by the implementation of Python itself without any "import" hints  
(which, if py2exe also did this, would've probably solved Tony's issue  
with bsddb).  Also, I did an analysis of the Python standard library  
and I discovered that the following (hopefully rather complete) list of  
implicit dependencies (from  
):

{
# imports done from builtin modules in C code (untrackable by  
modulegraph)
"time": ["_strptime"],
"datetime": ["time"],
"MacOS":["macresource"],
"cPickle":  ["copy_reg", "cStringIO"],
"parser":   ["copy_reg"],
"codecs":   ["encodings"],
"cStringIO":["copy_reg"],
"_sre": ["copy", "string", "sre"],
"zipimport":["zlib"],
# mactoolboxglue can do a bunch more of these
# that are far harder to predict, these should be tracked
# manually for now.

# this isn't C, but it uses __import__
"anydbm":   ["dbhash", "gdbm", "dbm", "dumbdbm", "whichdb"],
}
I would like to write a PEP for modulegraph as a replacement for  
modulefinder at some point, but I can't budget the time for it right  
now.  The current implementation is also largely untested on other  
platforms.  I hear it has been used by a Twisted developer to create  
Windows NT services using py2exe (augmenting py2exe's rather simple  
dependency resolution mechanism), however I'm not sure if that is in  
public svn or not.  If the authors of the other freeze tools are  
interested, they can feel free to use modulegraph from py2app -- it is  
cross-platform code under MIT license, but I can dual-license if  
necessary (however I think it should be compatible with cx_freeze,  
py2exe, and Python itself).  The API is purposefully different than  
modulefinder, but it is close enough such that most of the work  
involved is just removing unnecessary kludges.

-bob
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: Moving towards Python 3.0 (was Re: [Python-Dev] Speed up function calls)

[Brett C.]

Neal Norwitz wrote:
On Wed, 26 Jan 2005 09:47:41 -0500, Raymond Hettinger <[EMAIL PROTECTED]> wrote:
[SNIP]
Any ideas how we could start to realize some benefits of Py3.0 before
it arrives?  I'm not sure if this is worth it, if it's premature, or
if there are other ways to acheive the goal of easing transition for
users and simplifying developers tasks (by spreading over a longer
period of time) and reducing the possibility of not fixing warts.
The way I always imagined Python 3.0 would come about would be through preview 
releases.  Once the final 2.x version was released and went into maintennance 
we would start developing Python 3.0 .  During that development, when a major 
semantic change was checked in and seemed to work we could do a quick preview 
release for people to use to see if the new features up to that preview release 
would break their code.

Any other way, though, through concurrent development, seems painful.  As you 
mentioned, Neal, branches require merges eventually and that can be painful.  I 
suspect people will just have to put up with a longer dev time for Python 3.0 . 
 That longer dev time might actually be a good thing in the end.  It would 
enable us to really develop a very stable 2.x version of Python that we all 
know will be in use for quite some time by old code.

-Brett
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

RE: Moving towards Python 3.0 (was Re: [Python-Dev] Speed up function calls)

[Raymond Hettinger]

Neal Norwitz 
> I thought about making a p3k branch in CVS

I had hoped for the core of p3k to be built for scratch so that even the
most pervasive and fundamental implementation choices would be open for
discussion:

* Possibly write in C++.
* Possibly replace bytecode with Forth style threaded code.
* Possibly toss ref counting in favor of some kind of GC.
* Consider ways to leverage multiple processor environments.
* Consider alternative ways to implement exception handling (long jumps,
etc, signals, etc.)
* Look at alternate ways of building, passing, and parsing function
arguments.
* Use b-trees instead of dictionaries (just kidding).


Raymond


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

RE: Moving towards Python 3.0 (was Re: [Python-Dev] Speed up function calls)

[Skip Montanaro]

Raymond> I had hoped for the core of p3k to be built for scratch ...

Then we should just create a new CVS module for it (or go whole hog and try
a new revision control system altogether - svn, darcs, arch, whatever).

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: Moving towards Python 3.0 (was Re: [Python-Dev] Speed up function calls)

[Guido van Rossum]

> I had hoped for the core of p3k to be built for scratch [...]

Stop right there.

I used to think that was a good idea too, and was hoping to do exactly
that (after retirement :). However, the more I think about it, the
more I believe it would be throwing away too much valuable work.

Please read this article by Joel Spolsky (if you're not yet in the
habit of reading "Joel on Software", you're missing something):
http://joelonsoftware.com/articles/fog69.html

Then tell me if you still want to start over. I expect that if we do
piecemeal replacement of modules rather than starting from scratch
we'll be more productive sooner with less effort. After all, the
Python 3000 effort shouldn't be as pervasive as the Perl 6 design --
we're not redesigning the language from scratch, we're just tweaking
(albeit allowing backwards incompatibilities).

> * Possibly write in C++.
> * Possibly replace bytecode with Forth style threaded code.
> * Possibly toss ref counting in favor of some kind of GC.
> * Consider ways to leverage multiple processor environments.
> * Consider alternative ways to implement exception handling (long jumps,
> etc, signals, etc.)
> * Look at alternate ways of building, passing, and parsing function
> arguments.
> * Use b-trees instead of dictionaries (just kidding).

The "just kidding" applies to the whole list, right? None of these
strike me as good ideas, except for improvements to function argument
passing.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Bug tracker reviews

[Tony Meyer]

As promised, here are five bug reviews with recommendations.  If they help 

[ 1112812 ] Patch for Lib/bsddb/__init__.py to work with modulefinder


get reviewed, then that'd be great.  Otherwise I'll just take the good karma
and run :)

-

[ 531205 ] Bugs in rfc822.parseaddr()


What to do when an email address contains spaces, when RFC2822 says it
can't.  At the moment the spaces are stripped.  Recommend closing "Won't
Fix", for reasons outlined in the tracker by Tim Roberts.

[ 768419 ] Subtle bug in os.path.realpath on Cygwin


Agree with Sjoerd that this is a Cygwin bug rather than a Python one (and no
response from OP for a very long time).  Recommend closing "Won't Fix".

[ 803413 ] uu.decode prints to stderr


The question is whether it is ok for library modules to print to stderr if a
recoverable error occurs.  Looking at other modules, it seems uncommon, but
ok, so recommend closing "Won't fix", but making the suggested documentation
change.
(Alternatively, change from printing to stderr to using warnings.warn, which
would be a simple change and possibly more correct, although giving the same
result).

[ 989333 ] Empty curses module is loaded in win32


Importing curses loads an empty module instead of raising ImportError on
win32.  I cannot duplicate this: recommend closing as "Invalid".

[ 1090076 ] Defaults in ConfigParser.get overrides section values


Behaviour of ConfigParser doesn't match the documentation.  The included
patch for ConfigParser does fix the problem, but might break existing code.
A decision needs to be made which is the desired behaviour, and the tracker
can be closed either "Won't Fix" or "Fixed" (and the fix applied for 2.5 and
2.4.1).

=Tony.Meyer

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com