[Python-Dev] Weak references: dereference notification
Hello, I have come across a situation where I find the current weak references interface for extension types insufficient. Currently you only have a tp_weaklistoffset slot, pointing to a PyObject with weak references. However, in my case[1] I _really_ need to be notified when a weak reference is dereferenced. What happens now is that, when you call a weakref object, a simple Py_INCREF is done on the referenced object. It would be easy to implement a new slot to contain a function that should be called when a weak reference is dereferenced. Or, alternatively, a slot or class attribute that indicates an alternative type that should be used to create weak references: instead of the builtin weakref object, a subtype of it, so you can override tp_call. Does this sounds acceptable? Regards. [1] http://bugzilla.gnome.org/show_bug.cgi?id=320428 -- Gustavo J. A. M. Carneiro <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> The universe is always one step beyond logic. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
On 11/9/05, Guido van Rossum <[EMAIL PROTECTED]> wrote: > You didn't show us what's in the zip file. Can you show a zipinfo output? $ zipinfo modules.zip Archive: modules.zip 426 bytes 2 files -rw-r--r-- 2.3 unx 109 bx defN 31-Oct-05 14:49 module_o.pyo -rw-r--r-- 2.3 unx 109 bx defN 31-Oct-05 14:48 module_c.pyc 2 files, 218 bytes uncompressed, 136 bytes compressed: 37.6% > My intention with import was always that without -O, *.pyo files are > entirely ignored; and with -O, *.pyc files are entirely ignored. > > It sounds like you're saying that you want to change this so that .pyc > and .pyo are always honored (with .pyc preferred if -O is not present > and .pyo preferred if -O is present). I'm not sure that I like that > better. If that's how zipimport works, I think it's broken! Yes, this is how zipimport works and I think this is good in cases where a third-party binary module/package is available only with .pyo files and others only with .pyc files (without .py source files, of course). I know we can rename the files, but this is a good solution? Well, I don't have a strong opinion about the solution adopted and I really like to see other alternatives and opinions. Thanks, Osvaldo -- Osvaldo Santana Neto (aCiDBaSe) icq, url = (11287184, "http://www.pythonbrasil.com.br";) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
Maybe it makes more sense to deprecate .pyo altogether and instead have a post-load optimizer optimize .pyc files according to the current optimization settings? Unless others are interested in this nothing will happen. I've never heard of a third party making their code available only as .pyo, so the use case for changing things isn't very strong. In fact the only use cases I know for not making .py available are in situations where a proprietary "canned" application is distributed to end users who have no intention or need to ever add to the code. --Guido On 11/9/05, Osvaldo Santana <[EMAIL PROTECTED]> wrote: > On 11/9/05, Guido van Rossum <[EMAIL PROTECTED]> wrote: > > You didn't show us what's in the zip file. Can you show a zipinfo output? > > $ zipinfo modules.zip > Archive: modules.zip 426 bytes 2 files > -rw-r--r-- 2.3 unx 109 bx defN 31-Oct-05 14:49 module_o.pyo > -rw-r--r-- 2.3 unx 109 bx defN 31-Oct-05 14:48 module_c.pyc > 2 files, 218 bytes uncompressed, 136 bytes compressed: 37.6% > > > My intention with import was always that without -O, *.pyo files are > > entirely ignored; and with -O, *.pyc files are entirely ignored. > > > > It sounds like you're saying that you want to change this so that .pyc > > and .pyo are always honored (with .pyc preferred if -O is not present > > and .pyo preferred if -O is present). I'm not sure that I like that > > better. If that's how zipimport works, I think it's broken! > > Yes, this is how zipimport works and I think this is good in cases > where a third-party binary module/package is available only with .pyo > files and others only with .pyc files (without .py source files, of > course). > > I know we can rename the files, but this is a good solution? Well, I > don't have a strong opinion about the solution adopted and I really > like to see other alternatives and opinions. > > Thanks, > Osvaldo > > -- > Osvaldo Santana Neto (aCiDBaSe) > icq, url = (11287184, "http://www.pythonbrasil.com.br";) > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak references: dereference notification
Gustavo J. A. M. Carneiro wrote: > Hello, > > I have come across a situation where I find the current weak > references interface for extension types insufficient. > > Currently you only have a tp_weaklistoffset slot, pointing to a > PyObject with weak references. However, in my case[1] I _really_ need > to be notified when a weak reference is dereferenced. What happens now > is that, when you call a weakref object, a simple Py_INCREF is done on > the referenced object. It would be easy to implement a new slot to > contain a function that should be called when a weak reference is > dereferenced. Or, alternatively, a slot or class attribute that > indicates an alternative type that should be used to create weak > references: instead of the builtin weakref object, a subtype of it, so > you can override tp_call. > > Does this sounds acceptable? Since you can now (as of 2.4) subclass the weakref.ref class, you should be able to do this yourself in Python. See for example, weakref.KeyedRef. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak references: dereference notification
Qua, 2005-11-09 às 11:50 -0500, Jim Fulton escreveu: > Gustavo J. A. M. Carneiro wrote: > > Hello, > > > > I have come across a situation where I find the current weak > > references interface for extension types insufficient. > > > > Currently you only have a tp_weaklistoffset slot, pointing to a > > PyObject with weak references. However, in my case[1] I _really_ need > > to be notified when a weak reference is dereferenced. What happens now > > is that, when you call a weakref object, a simple Py_INCREF is done on > > the referenced object. It would be easy to implement a new slot to > > contain a function that should be called when a weak reference is > > dereferenced. Or, alternatively, a slot or class attribute that > > indicates an alternative type that should be used to create weak > > references: instead of the builtin weakref object, a subtype of it, so > > you can override tp_call. > > > > Does this sounds acceptable? > > Since you can now (as of 2.4) subclass the weakref.ref class, you should be > able to > do this yourself in Python. See for example, weakref.KeyedRef. I know I can subclass it, but it doesn't change anything. If people keep writing code like weakref.ref(myobj) instead of myweakref(myobj), it still won't work. I wouldn't want to have to teach users of the library that they need to use an alternative type; that seldom doesn't work. Now, if there was a place in the type that contained information like "for creating weak references of instances of this type, use this weakref class" and weakref.ref was smart enough to lookup this type and use it, only _then_ it could work. Thanks, -- Gustavo J. A. M. Carneiro <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> The universe is always one step beyond logic. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak references: dereference notification
> > Gustavo J. A. M. Carneiro wrote: > > > I have come across a situation where I find the current weak > > > references interface for extension types insufficient. > > > > > > Currently you only have a tp_weaklistoffset slot, pointing to a > > > PyObject with weak references. However, in my case[1] I _really_ need > > > to be notified when a weak reference is dereferenced. I find reading through the bug discussion a bit difficult to understand your use case. Could you explain it here? If you can't explain it you certainly won't get your problem solved! :-) > > > What happens now > > > is that, when you call a weakref object, a simple Py_INCREF is done on > > > the referenced object. It would be easy to implement a new slot to > > > contain a function that should be called when a weak reference is > > > dereferenced. Or, alternatively, a slot or class attribute that > > > indicates an alternative type that should be used to create weak > > > references: instead of the builtin weakref object, a subtype of it, so > > > you can override tp_call. > > > > > > Does this sounds acceptable? [Jim Fulton] > > Since you can now (as of 2.4) subclass the weakref.ref class, you should be > > able to > > do this yourself in Python. See for example, weakref.KeyedRef. > > I know I can subclass it, but it doesn't change anything. If people > keep writing code like weakref.ref(myobj) instead of myweakref(myobj), > it still won't work. > > I wouldn't want to have to teach users of the library that they need > to use an alternative type; that seldom doesn't work. > > Now, if there was a place in the type that contained information like > > "for creating weak references of instances of this type, use this > weakref class" > > and weakref.ref was smart enough to lookup this type and use it, only > _then_ it could work. Looks what you're looking for is a customizable factory fuction. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
> Maybe it makes more sense to deprecate .pyo altogether and instead > have a post-load optimizer optimize .pyc files according to the > current optimization settings? That would not be enough, because it would leave the docstrings in the .pyc files. > Unless others are interested in this nothing will happen. The status quo is good enough, for "normal" imports. If zipimport works differently, well, that's not nice. > I've never heard of a third party making their code available only as > .pyo, *cough* Ahem, here we are (the firm I work for). > so the use case for changing things isn't very strong. In fact > the only use cases I know for not making .py available are in > situations where a proprietary "canned" application is distributed to > end users who have no intention or need to ever add to the code. Well, exactly. :-) -- Nicola Larosa - [EMAIL PROTECTED] No inventions have really significantly eased the cognitive difficulty of writing scalable concurrent applications and it is unlikely that any will in the near term. [...] Most of all, threads do not help, in fact, they make the problem worse in many cases. -- G. Lefkowitz, August 2005 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak references: dereference notification
Qua, 2005-11-09 às 09:23 -0800, Guido van Rossum escreveu: > > > Gustavo J. A. M. Carneiro wrote: > > > > I have come across a situation where I find the current weak > > > > references interface for extension types insufficient. > > > > > > > > Currently you only have a tp_weaklistoffset slot, pointing to a > > > > PyObject with weak references. However, in my case[1] I _really_ need > > > > to be notified when a weak reference is dereferenced. > > I find reading through the bug discussion a bit difficult to > understand your use case. Could you explain it here? If you can't > explain it you certainly won't get your problem solved! :-) This is a typical PyObject wrapping C object (GObject) problem. Both PyObject and GObject have independent reference counts. For each GObject there is at most one PyObject wrapper. When the refcount on the wrapper drops to zero, tp_dealloc is called. In tp_dealloc, and if the GObject refcount is > 1, I do something slightly evil: I 'resurect' the PyObject (calling PyObject_Init), create a weak reference to the GObject, and drop the "strong" reference. I call this a 'hibernation state'. Now the problem. Suppose the user had a weak ref to the PyObject: 1- At certain point in time, when the wrapper is in hibernation state, the user calls the weak ref 2- It gets a PyObject that contains a weak reference to the GObject; 3- Now suppose whatever was holding the GObject ref drops its reference, which was the last one, and the GObject dies; 4- Now the user does something with the PyObject obtained through the weakref -> invalid memory access. The cause for the problem is that between steps 2 and 3 the wrapper needs to change the weak reference to the GObject to a strong one. Unfortunately, I don't get any notification that 2 happened. BTW, I fixed this problem in the mean time with a bit more of slightly evil code. I override tp_call of the standard weakref type :-P [...] > > and weakref.ref was smart enough to lookup this type and use it, only > > _then_ it could work. > > Looks what you're looking for is a customizable factory fuction. Sure, if weakref.ref could be such a factory, and could take "advice" on what type of weakref to use for each class. Regards. -- Gustavo J. A. M. Carneiro <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> The universe is always one step beyond logic. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
On 11/9/05, Guido van Rossum <[EMAIL PROTECTED]> wrote: > Maybe it makes more sense to deprecate .pyo altogether and instead > have a post-load optimizer optimize .pyc files according to the > current optimization settings? I agree with this idea, but we've to think about docstrings (like Nicola said in his e-mail). Maybe we want to create a different and optimization-independent option to remove docstrings from modules? > Unless others are interested in this nothing will happen. > > I've never heard of a third party making their code available only as > .pyo, so the use case for changing things isn't very strong. In fact > the only use cases I know for not making .py available are in > situations where a proprietary "canned" application is distributed to > end users who have no intention or need to ever add to the code. I've other use case: I'm porting Python to Maemo Platform and I want to reduce the size of modules. The .pyo (-OO) are smaller than .pyc files (mainly because docstring removing) and we start to use this optimization flag to compile our Python distribution. In this case we want to force developers to call Python Interpreter with -O flags, set PYTHONOPTIMIZE, or apply my patch :) to make this more transparent. I've noticed this inconsistency when we stop to use zipimport in our Python For Maemo distribution. We've decided to stop using zipimport because the device (Nokia 770) uses a compressed filesystem. Some friends (mainly Gustavo Barbieri) help me to create the suggested patch after some discussion in our PythonBrasil mailing list. Thanks, Osvaldo -- Osvaldo Santana Neto (aCiDBaSe) icq, url = (11287184, "http://www.pythonbrasil.com.br";) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
On 11/9/05, Osvaldo Santana <[EMAIL PROTECTED]> wrote: > I've noticed this inconsistency when we stop to use zipimport in our > Python For Maemo distribution. We've decided to stop using zipimport > because the device (Nokia 770) uses a compressed filesystem. I won't comment further on the brainstorm that's going on (this is becoming a topic for c.l.py) but I think you are misunderstanding the point of zipimport. It's not done (usually) for the compression but for the index. Finding a name in the zipfile index is much more efficient than doing a directory search; and the zip index can be cached. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak references: dereference notification
On 9-nov-2005, at 18:52, Gustavo J. A. M. Carneiro wrote: > Qua, 2005-11-09 às 09:23 -0800, Guido van Rossum escreveu: Gustavo J. A. M. Carneiro wrote: > I have come across a situation where I find the current weak > references interface for extension types insufficient. > > Currently you only have a tp_weaklistoffset slot, pointing to a > PyObject with weak references. However, in my case[1] I > _really_ need > to be notified when a weak reference is dereferenced. >> >> I find reading through the bug discussion a bit difficult to >> understand your use case. Could you explain it here? If you can't >> explain it you certainly won't get your problem solved! :-) > > This is a typical PyObject wrapping C object (GObject) problem. > Both > PyObject and GObject have independent reference counts. For each > GObject there is at most one PyObject wrapper. > > When the refcount on the wrapper drops to zero, tp_dealloc is > called. > In tp_dealloc, and if the GObject refcount is > 1, I do something > slightly evil: I 'resurect' the PyObject (calling PyObject_Init), > create > a weak reference to the GObject, and drop the "strong" reference. I > call this a 'hibernation state'. Why do you do that? The only reasons I can think of are that you hope to gain some speed from this or that you want to support weak references to the GObject. For what its worth, in PyObjC we don't support weak references to the underlying Objective-C object and delete the proxy object when it is garbage collected. Objective-C also has reference counts, we increase that in the constructor for the proxy object and decrease it again in the destroctor. Ronald ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
At 11:32 AM 11/9/2005 -0800, Guido van Rossum wrote: >On 11/9/05, Osvaldo Santana <[EMAIL PROTECTED]> wrote: > > I've noticed this inconsistency when we stop to use zipimport in our > > Python For Maemo distribution. We've decided to stop using zipimport > > because the device (Nokia 770) uses a compressed filesystem. > >I won't comment further on the brainstorm that's going on (this is >becoming a topic for c.l.py) but I think you are misunderstanding the >point of zipimport. It's not done (usually) for the compression but >for the index. Finding a name in the zipfile index is much more >efficient than doing a directory search; and the zip index can be >cached. zipimport also helps distribution convenience - a large and elaborate package can be distributed in a single zipfile (such as is built by setuptools' "bdist_egg" command) and simply placed on PYTHONPATH or directly on sys.path. And tools like py2exe can also append all an application's modules to an executable file in zipped form. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
On 11/9/05, Guido van Rossum <[EMAIL PROTECTED]> wrote: > On 11/9/05, Osvaldo Santana <[EMAIL PROTECTED]> wrote: > > I've noticed this inconsistency when we stop to use zipimport in our > > Python For Maemo distribution. We've decided to stop using zipimport > > because the device (Nokia 770) uses a compressed filesystem. > > I won't comment further on the brainstorm that's going on (this is > becoming a topic for c.l.py) but I think you are misunderstanding the > point of zipimport. It's not done (usually) for the compression but > for the index. Finding a name in the zipfile index is much more > efficient than doing a directory search; and the zip index can be > cached. Any way, not loading .pyo if no .pyc or .py is available is a drawback, specially on unices that have scripts starting with "#!/usr/bin/python" or "#!/usr/bin/env python" and the system just have .pyo files, due a bunch of reasons, in this case the small disc space. -- Gustavo Sverzut Barbieri -- Computer Engineer 2001 - UNICAMP Mobile: +55 (19) 9165 8010 Phone: +1 (347) 624 6296 @ sip.stanaphone.com Jabber: [EMAIL PROTECTED] ICQ#: 17249123 MSN: [EMAIL PROTECTED] GPG: 0xB640E1A2 @ wwwkeys.pgp.net ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
It's a shame that 1) there's no equivalent of "java -jar", i.e., "python -z FILE.ZIP", and 2) the use of zipfiles is so poorly documented. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
On Nov 9, 2005, at 1:22 PM, Bill Janssen wrote: > It's a shame that > > 1) there's no equivalent of "java -jar", i.e., "python -z > FILE.ZIP", and This should work on a few platforms: env PYTHONPATH=FILE.zip python -m some_module_in_the_zip -bob ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
Bob Ippolito <[EMAIL PROTECTED]> writes: > On Nov 9, 2005, at 1:22 PM, Bill Janssen wrote: > >> It's a shame that >> >> 1) there's no equivalent of "java -jar", i.e., "python -z >> FILE.ZIP", and > > This should work on a few platforms: > env PYTHONPATH=FILE.zip python -m some_module_in_the_zip It should, yes - but it doesn't: -m doesn't work with zipimport. Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
On Nov 9, 2005, at 1:48 PM, Thomas Heller wrote: > Bob Ippolito <[EMAIL PROTECTED]> writes: > >> On Nov 9, 2005, at 1:22 PM, Bill Janssen wrote: >> >>> It's a shame that >>> >>> 1) there's no equivalent of "java -jar", i.e., "python -z >>> FILE.ZIP", and >> >> This should work on a few platforms: >> env PYTHONPATH=FILE.zip python -m some_module_in_the_zip > > It should, yes - but it doesn't: -m doesn't work with zipimport. That's dumb, someone should fix that. Is there a bug filed? -bob ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
Bob Ippolito wrote: > On Nov 9, 2005, at 1:22 PM, Bill Janssen wrote: > >> It's a shame that >> >> 1) there's no equivalent of "java -jar", i.e., "python -z >> FILE.ZIP", and > > This should work on a few platforms: > env PYTHONPATH=FILE.zip python -m some_module_in_the_zip Really? I wrote the '-m' code, and I wouldn't expect that to work anywhere because 'execfile' and the C equivalent that -m relies on expect a real file. PEP 328 goes some way towards fixing that by having a Python fallback to find and execute the module if the current C code fails. If we had execmodule as a Python function, it would make it much easier to add support for compiling and executing the target module directly, rather than indirecting through the file-system-dependent execfile. In theory this could be done in C, but execmodule is fairly long even written in Python. I'm actually fairly sure it *could* be written in C, but I think doing so would be horribly tedious (and not as useful in the long run). Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://boredomandlaziness.blogspot.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak references: dereference notification
On Wed, 2005-11-09 at 20:40 +0100, Ronald Oussoren wrote: > On 9-nov-2005, at 18:52, Gustavo J. A. M. Carneiro wrote: > > > Qua, 2005-11-09 às 09:23 -0800, Guido van Rossum escreveu: > Gustavo J. A. M. Carneiro wrote: > > I have come across a situation where I find the current weak > > references interface for extension types insufficient. > > > > Currently you only have a tp_weaklistoffset slot, pointing to a > > PyObject with weak references. However, in my case[1] I > > _really_ need > > to be notified when a weak reference is dereferenced. > >> > >> I find reading through the bug discussion a bit difficult to > >> understand your use case. Could you explain it here? If you can't > >> explain it you certainly won't get your problem solved! :-) > > > > This is a typical PyObject wrapping C object (GObject) problem. > > Both > > PyObject and GObject have independent reference counts. For each > > GObject there is at most one PyObject wrapper. > > > > When the refcount on the wrapper drops to zero, tp_dealloc is > > called. > > In tp_dealloc, and if the GObject refcount is > 1, I do something > > slightly evil: I 'resurect' the PyObject (calling PyObject_Init), > > create > > a weak reference to the GObject, and drop the "strong" reference. I > > call this a 'hibernation state'. > > Why do you do that? The only reasons I can think of are that you hope > to gain > some speed from this or that you want to support weak references to > the GObject. We want to support weak references to GObjects. Mainly because that support has always been there and we don't want/can't break API. And it does have some uses... > > For what its worth, in PyObjC we don't support weak references to the > underlying > Objective-C object and delete the proxy object when it is garbage > collected. > Objective-C also has reference counts, we increase that in the > constructor for > the proxy object and decrease it again in the destroctor. OK, but what if it is a subclass of a builtin type, with instance variables? What if the PyObject is GC'ed but the ObjC object remains alive, and later you get a new reference to it? Do you create a new PyObject wrapper for it? What happened to the instance variables? Our goal in wrapping GObject is that, once a Python wrapper for a GObject instance is created, it never dies until the GObject dies too. At the same time, once the python wrapper loses all references, it should not stop keeping the GObject alive. What happens currently, which is what I'm trying to change, is that there is a reference loop between PyObject and GObject, so that deallocation only happens with the help of the cyclic GC. But relying on the GC for _everything_ causes annoying problems: 1- The GC runs only once in a while, not soon enough if eg. you have an image object with several megabytes; 2- It makes it hard to debug reference counting bugs, as the symptom only appears when the GC runs, far away from the code that cause the problem in the first place; 3- Generally the GC has a lot more work, since every PyGTK object needs it, and a GUI app can have lots of PyGTK objects. Regards. -- Gustavo J. A. M. Carneiro <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> The universe is always one step beyond logic ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
On 11/9/05, Bob Ippolito <[EMAIL PROTECTED]> wrote: > > On Nov 9, 2005, at 1:48 PM, Thomas Heller wrote: > > > Bob Ippolito <[EMAIL PROTECTED]> writes: > > > >> On Nov 9, 2005, at 1:22 PM, Bill Janssen wrote: > >> > >>> It's a shame that > >>> > >>> 1) there's no equivalent of "java -jar", i.e., "python -z > >>> FILE.ZIP", and > >> > >> This should work on a few platforms: > >> env PYTHONPATH=FILE.zip python -m some_module_in_the_zip > > > > It should, yes - but it doesn't: -m doesn't work with zipimport. > > That's dumb, someone should fix that. Is there a bug filed? I did, a while ago. http://www.python.org/sf/1250389 Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
On 11/9/05, Guido van Rossum <[EMAIL PROTECTED]> wrote: > Maybe it makes more sense to deprecate .pyo altogether and instead > have a post-load optimizer optimize .pyc files according to the > current optimization settings? > But I thought part of the point of .pyo files was that they left out docstrings and thus had a smaller footprint? Plus I wouldn't be surprised if we started to move away from bytecode optimization and instead tried to do more AST transformations which would remove possible post-load optimizations. I would have no issue with removing .pyo files and have .pyc files just be as optimized as they the current settings are and leave it at that. Could have some metadata listing what optimizations occurred, but do we really need to have a specific way to denote if bytecode has been optimized? Binary files compiled from C don't note what -O optimization they were compiled with. If someone distributes optimized .pyc files chances are they are going to have a specific compile step with py_compile and they will know what optimizations they are using. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
On Nov 9, 2005, at 6:05 PM, Brett Cannon wrote: > I would have no issue with removing .pyo files and have .pyc files > just be as optimized as they the current settings are and leave it at > that. Could have some metadata listing what optimizations occurred, > but do we really need to have a specific way to denote if bytecode has > been optimized? Binary files compiled from C don't note what -O > optimization they were compiled with. If someone distributes > optimized .pyc files chances are they are going to have a specific > compile step with py_compile and they will know what optimizations > they are using. > This sounds quite sensible. The only thing I'd add is that iff there is a .py file of the same name, and the current optimization settings are different from those in the .pyc file, python should recompile the .py file. James ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
On 11/9/05, Brett Cannon <[EMAIL PROTECTED]> wrote: > On 11/9/05, Guido van Rossum <[EMAIL PROTECTED]> wrote: > > Maybe it makes more sense to deprecate .pyo altogether and instead > > have a post-load optimizer optimize .pyc files according to the > > current optimization settings? > > But I thought part of the point of .pyo files was that they left out > docstrings and thus had a smaller footprint? Very few people care about the smaller footprint (although one piped up here). > Plus I wouldn't be > surprised if we started to move away from bytecode optimization and > instead tried to do more AST transformations which would remove > possible post-load optimizations. > > I would have no issue with removing .pyo files and have .pyc files > just be as optimized as they the current settings are and leave it at > that. Could have some metadata listing what optimizations occurred, > but do we really need to have a specific way to denote if bytecode has > been optimized? Binary files compiled from C don't note what -O > optimization they were compiled with. If someone distributes > optimized .pyc files chances are they are going to have a specific > compile step with py_compile and they will know what optimizations > they are using. Currently, .pyo files have some important semantic differences with .pyc files; -O doesn't remove docstrings (that's -OO) but it does remove asserts. I wouldn't want to accidentally use a .pyc file without asserts compiled in unless the .py file wasn't around. For application distribution, the following probably would work: - instead of .pyo files, we use .pyc files - the .pyc file records whether optimizations were applied, whether asserts are compiled, and whether docstrings are retained - if the compiler finds a .pyc that is inconsistent with the current command line, it ignores it and rewrites it (if it is writable) just as if the .py file were newer However, this would be a major pain for the standard library and other shared code -- there it's really nice to have a cache for each of the optimization levels since usually regular users can't write the .py[co] files there, meaning very slow always-recompilation if the standard .pyc files aren't of the right level, causing unacceptable start-up times. The only solutions I can think of that use a single file actually *increase* the file size by having unoptimized and optimized code side-by-side, or some way to quickly skip the assertions -- the -OO option is a special case that probably needs to be done differently anyway and only for final distribution. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
On 11/9/05, Guido van Rossum <[EMAIL PROTECTED]> wrote: > On 11/9/05, Brett Cannon <[EMAIL PROTECTED]> wrote: > > Plus I wouldn't be > > surprised if we started to move away from bytecode optimization and > > instead tried to do more AST transformations which would remove > > possible post-load optimizations. > > > > I would have no issue with removing .pyo files and have .pyc files > > just be as optimized as they the current settings are and leave it at > > that. Could have some metadata listing what optimizations occurred, > > but do we really need to have a specific way to denote if bytecode has > > been optimized? Binary files compiled from C don't note what -O > > optimization they were compiled with. If someone distributes > > optimized .pyc files chances are they are going to have a specific > > compile step with py_compile and they will know what optimizations > > they are using. > > Currently, .pyo files have some important semantic differences with > .pyc files; -O doesn't remove docstrings (that's -OO) but it does > remove asserts. I wouldn't want to accidentally use a .pyc file > without asserts compiled in unless the .py file wasn't around. > > For application distribution, the following probably would work: > > - instead of .pyo files, we use .pyc files > - the .pyc file records whether optimizations were applied, whether > asserts are compiled, and whether docstrings are retained > - if the compiler finds a .pyc that is inconsistent with the current > command line, it ignores it and rewrites it (if it is writable) just > as if the .py file were newer > > However, this would be a major pain for the standard library and other > shared code -- there it's really nice to have a cache for each of the > optimization levels since usually regular users can't write the > .py[co] files there, meaning very slow always-recompilation if the > standard .pyc files aren't of the right level, causing unacceptable > start-up times. > What if PEP 304 came into being? Then people would have a place to have the shared code's recompiled version stored and thus avoid the overhead from repeated use. > The only solutions I can think of that use a single file actually > *increase* the file size by having unoptimized and optimized code > side-by-side, or some way to quickly skip the assertions -- the -OO > option is a special case that probably needs to be done differently > anyway and only for final distribution. > One option would be to introduce an ASSERTION bytecode that has an argument specifying the amount of bytecode for the assertion. The eval loop can then just igonore the bytecode if assertions are being evaluated and fall through to the bytecode for the assertions (and thus be the equivalent of NOP) or use the argument to jump forward that number of bytes in the bytecode and completely skip over the assertion (and thus be just like a JUMP_FORWARD). Either way assertions becomes slightly more costly but it should be very minimal. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
At 03:25 PM 11/9/2005 -0800, Guido van Rossum wrote: >The only solutions I can think of that use a single file actually >*increase* the file size by having unoptimized and optimized code >side-by-side, or some way to quickly skip the assertions -- the -OO >option is a special case that probably needs to be done differently >anyway and only for final distribution. We could have a "JUMP_IF_NOT_DEBUG" opcode to skip over asserts and "if __debug__" blocks. Then under -O we could either patch this to a plain jump, or compact the bytecode to remove the jumped-over part(s). By the way, while we're on this subject, can we make the optimization options be part of the compile() interface? Right now the distutils has to actually exec another Python process whenever you want to compile code with a different optimization level than what's currently in effect, whereas if it could pass the desired level to compile(), this wouldn't be necessary. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
On 11/9/05, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > At 03:25 PM 11/9/2005 -0800, Guido van Rossum wrote: > >The only solutions I can think of that use a single file actually > >*increase* the file size by having unoptimized and optimized code > >side-by-side, or some way to quickly skip the assertions -- the -OO > >option is a special case that probably needs to be done differently > >anyway and only for final distribution. > > We could have a "JUMP_IF_NOT_DEBUG" opcode to skip over asserts and "if > __debug__" blocks. Then under -O we could either patch this to a plain > jump, or compact the bytecode to remove the jumped-over part(s). That sounds very reasonable. > By the way, while we're on this subject, can we make the optimization > options be part of the compile() interface? Right now the distutils has to > actually exec another Python process whenever you want to compile > code with > a different optimization level than what's currently in effect, whereas if > it could pass the desired level to compile(), this wouldn't be necessary. Makes sense to me; we need a patch of course. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
[Guido] > > However, this would be a major pain for the standard library and other > > shared code -- there it's really nice to have a cache for each of the > > optimization levels since usually regular users can't write the > > .py[co] files there, meaning very slow always-recompilation if the > > standard .pyc files aren't of the right level, causing unacceptable > > start-up times. [Brett] > What if PEP 304 came into being? Then people would have a place to > have the shared code's recompiled version stored and thus avoid the > overhead from repeated use. Still sounds suboptimal for the standard library; IMO it should "just work". > > The only solutions I can think of that use a single file actually > > *increase* the file size by having unoptimized and optimized code > > side-by-side, or some way to quickly skip the assertions -- the -OO > > option is a special case that probably needs to be done differently > > anyway and only for final distribution. > > One option would be to introduce an ASSERTION bytecode that has an > argument specifying the amount of bytecode for the assertion. The > eval loop can then just igonore the bytecode if assertions are being > evaluated and fall through to the bytecode for the assertions (and > thus be the equivalent of NOP) or use the argument to jump forward > that number of bytes in the bytecode and completely skip over the > assertion (and thus be just like a JUMP_FORWARD). Either way > assertions becomes slightly more costly but it should be very minimal. I like Phillip's suggestion -- no new opcode, just a conditional jump that can be easily optimized out. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
On 11/9/05, Guido van Rossum <[EMAIL PROTECTED]> wrote: > [Guido] > > > However, this would be a major pain for the standard library and other > > > shared code -- there it's really nice to have a cache for each of the > > > optimization levels since usually regular users can't write the > > > .py[co] files there, meaning very slow always-recompilation if the > > > standard .pyc files aren't of the right level, causing unacceptable > > > start-up times. > [Brett] > > What if PEP 304 came into being? Then people would have a place to > > have the shared code's recompiled version stored and thus avoid the > > overhead from repeated use. > > Still sounds suboptimal for the standard library; IMO it should "just work". > Fair enough. > > > The only solutions I can think of that use a single file actually > > > *increase* the file size by having unoptimized and optimized code > > > side-by-side, or some way to quickly skip the assertions -- the -OO > > > option is a special case that probably needs to be done differently > > > anyway and only for final distribution. > > > > One option would be to introduce an ASSERTION bytecode that has an > > argument specifying the amount of bytecode for the assertion. The > > eval loop can then just igonore the bytecode if assertions are being > > evaluated and fall through to the bytecode for the assertions (and > > thus be the equivalent of NOP) or use the argument to jump forward > > that number of bytes in the bytecode and completely skip over the > > assertion (and thus be just like a JUMP_FORWARD). Either way > > assertions becomes slightly more costly but it should be very minimal. > > I like Phillip's suggestion -- no new opcode, just a conditional jump > that can be easily optimized out. Huh? But Phillip is suggesting a new opcode that is essentially the same as my proposal but naming it differently and saying the bytecode should get changed directly instead of having the eval loop handle the semantic differences based on whether -O is being used. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak references: dereference notification
Gustavo J. A. M. Carneiro wrote: > OK, but what if it is a subclass of a builtin type, with instance > variables? What if the PyObject is GC'ed but the ObjC object remains > alive, and later you get a new reference to it? Do you create a new > PyObject wrapper for it? What happened to the instance variables? Your proposed scheme appears to involve destroying and then re-initialising the Python wrapper. Isn't that going to wipe out any instance variables it may have had? Also, it seems to me that as soon as the refcount on the wrapper drops to zero, any weak references to it will be broken. Or does your resurrection code intervene before that happens? -- Greg Ewing, Computer Science Dept, +--+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | [EMAIL PROTECTED] +--+ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
> > I like Phillip's suggestion -- no new opcode, just a conditional jump > > that can be easily optimized out. > > Huh? But Phillip is suggesting a new opcode that is essentially the > same as my proposal but naming it differently and saying the bytecode > should get changed directly instead of having the eval loop handle the > semantic differences based on whether -O is being used. Sorry. Looking back they look pretty much the same to me. Somehow I glanced over Phillip's code and thought he was proposing to use a regular JUMP_IF opcode with the special __debug__ variable (which would be a 3rd possibility, good if we had backwards compatibility requirements for bytecode -- which we don't, fortunately :-). -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter
Martin v. Löwis wrote: > Michiel Jan Laurens de Hoon wrote: > >> 2) Will Tkinter always be the standard toolkit for Python, or are >> there plans to replace it at some point? > > > Python does not evolve along a grand master plan. Instead, individual > contributors propose specific modifications, e.g. through PEPs. At this point, I can't propose a specific modification yet because I don't know the reasoning that went behind the original choice of Tk as the default GUI toolkit for Python (and hence, I don't know if those reasons are still valid today). I can see one disadvantage (using Tk limits our options to run an event loop for other Python extensions), and I am trying to find out why Tk was deemed more appropriate than other GUI toolkits anyway. So let me rephrase the question: What is the advantage of Tk in comparison to other GUI toolkits? Is it Mac availability? More advanced widget set? Installation is easier? Portability? Switching to a different GUI toolkit would break too much existing code? I think that having the answer to this will stimulate further development of alternative GUI toolkits, which may give some future Python version a toolkit at least as good as Tk, and one that doesn't interfere with Python's event loop capabilities. --Michiel. -- Michiel de Hoon Center for Computational Biology and Bioinformatics Columbia University 1150 St Nicholas Avenue New York, NY 10032 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
On 11/9/05, Guido van Rossum <[EMAIL PROTECTED]> wrote: > > > I like Phillip's suggestion -- no new opcode, just a conditional jump > > > that can be easily optimized out. > > > > Huh? But Phillip is suggesting a new opcode that is essentially the > > same as my proposal but naming it differently and saying the bytecode > > should get changed directly instead of having the eval loop handle the > > semantic differences based on whether -O is being used. > > Sorry. No problem. Figured you just misread mine. > Looking back they look pretty much the same to me. Somehow I > glanced over Phillip's code and thought he was proposing to use a > regular JUMP_IF opcode with the special __debug__ variable (which > would be a 3rd possibility, good if we had backwards compatibility > requirements for bytecode -- which we don't, fortunately :-). > Fortunately. =) So does this mean you like the idea? Should this all move forward somehow? -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
On 11/9/05, Brett Cannon <[EMAIL PROTECTED]> wrote: > On 11/9/05, Guido van Rossum <[EMAIL PROTECTED]> wrote: > > > > I like Phillip's suggestion -- no new opcode, just a conditional jump > > > > that can be easily optimized out. > > > > > > Huh? But Phillip is suggesting a new opcode that is essentially the > > > same as my proposal but naming it differently and saying the bytecode > > > should get changed directly instead of having the eval loop handle the > > > semantic differences based on whether -O is being used. > > > > Sorry. > > No problem. Figured you just misread mine. > > > Looking back they look pretty much the same to me. Somehow I > > glanced over Phillip's code and thought he was proposing to use a > > regular JUMP_IF opcode with the special __debug__ variable (which > > would be a 3rd possibility, good if we had backwards compatibility > > requirements for bytecode -- which we don't, fortunately :-). > > > > Fortunately. =) > > So does this mean you like the idea? Should this all move forward somehow? I guess so. :-) It will need someone thinking really hard about all the use cases, edge cases, etc., implementation details, and writing up a PEP. Feel like volunteering? You might squeeze Phillip as a co-author. He's a really good one. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int(string) (was: DRAFT: python-dev Summary for 2005-09-01 through 2005-09-16)
Tim Peters wrote: > ... > Someone want a finite project that would _really_ help their Uncle > Timmy in his slow-motion crusade to get Python on the list of "solved > it!" languages for each problem on that magnificent site? ... > Turns out it's _not_ input speed that's the problem here, and not even > mainly the speed of integer mod: the bulk of the time is spent in > int(string) OK, I got an idea about how to do this fast. I started with Python code, and I now have C code that should beat int(string) always while getting a lot of speed making long values. The working tables can be used to do the reverse transformation (int or long to string in some base) with a little finesse, but I haven't done that yet in C. The code is pretty sprawly now (a lot left in for instrumentation and testing pieces), but can eventually get smaller. I gave myself time to do this as a birthday present to myself. It may take a while to build a patch, but perhaps you can let me know how much speedup you get using this code. if you build this module, I'd suggest using "from to_int import chomp" to get a function that works like int (producing a long when needed and so on). > If you can even track all the levels of C function calls that ends up > invoking , you find yourself in PyOS_strtoul(), which is a > nifty all-purpose routine that accepts inputs in bases 2 thru 36, can > auto-detect base, and does platform-independent overflow checking at > the cost of a division per digit. All those are features, but it > makes for slw conversion. OK, this code doesn't deal with unicode at all. The key observations are: A) to figure out the base, you pretty much need to get to the first digit; getting to the first non-zero digit is not that much worse. B) If you know the length of a string of digits (starting at the first non-zero digit) and the base, you know approximately how bits the result will have. You can do a single allocation if you are building a long. You can tell if you need to test for overflow in building an int; there is one length per base where you must. So the question becomes, is it worth taking two passes at the digits? Well, it sure looks like it to me, but I haven't timed one or two- character integers. I do longs in "megadigits" -- the largest set of digits that fits safely in SHIFT bits, so they have no need for overflow checks. For further excitement, you can use a similar technique to go from the number of bits to the string length. That should make for a fast convert int/long to string in any of 36 (or more, really) bases. I pass all of your mentioned test cases (including the one from a later message). I'm pretty much out of time for this project at the moment, but encouraging words would help me steal some time to finish. For anyone wanting to look at the code, or try it themselves: Installer: http://members.dsl-only.net/~daniels/dist/to_int-0.10.win32-py2.4.exe Just the 2.4 dll: http://members.dsl-only.net/~daniels/dist/to_int-0.10.win32.zip Sources: http://members.dsl-only.net/~daniels/dist/to_int-0.10.zip --Scott David Daniels [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter
Michiel Jan Laurens de Hoon wrote: > At this point, I can't propose a specific modification yet because I > don't know the reasoning that went behind the original choice of Tk as > the default GUI toolkit for Python Probably because at the time it was really the only cross-platform GUI toolkit that worked about equally well (or equally badly, depending on your point of view) on all the major platforms. I'm not sure the event-loop situation would be much different with another one, anyway. From what I've seen of GUI toolkits, they all have their own form of event loop, and they all provide some way of hooking other things into it (as does Tkinter), but whichever one you're using, it likes to be in charge. Code which blocks reading from standard input doesn't fit very well into any of them. -- Greg Ewing, Computer Science Dept, +--+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | [EMAIL PROTECTED] +--+ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter
On Thu, 10 Nov 2005 16:02:04 +1300, Greg Ewing <[EMAIL PROTECTED]> wrote: >Michiel Jan Laurens de Hoon wrote: > >> At this point, I can't propose a specific modification yet because I >> don't know the reasoning that went behind the original choice of Tk as >> the default GUI toolkit for Python > >Probably because at the time it was really the >only cross-platform GUI toolkit that worked >about equally well (or equally badly, depending >on your point of view) on all the major >platforms. > >I'm not sure the event-loop situation would be >much different with another one, anyway. From what >I've seen of GUI toolkits, they all have their own >form of event loop, and they all provide some way >of hooking other things into it (as does Tkinter), >but whichever one you're using, it likes to be in >charge. Code which blocks reading from standard >input doesn't fit very well into any of them. > Of course, the problem could be approached from the other direction: the blocking reads could be replaced with something else... Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
> This should work on a few platforms: > env PYTHONPATH=FILE.zip python -m some_module_in_the_zip Yeah, that's not bad, but I hate setting PYTHONPATH. I was thinking more along the line of python -z ZIPFILE where python would look at the ZIPFILE to see if there's a top-level module called "__init__", and if so, load it. That would allow existing PYTHONPATH settings to still be used if the user cares. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Unifying decimal numbers.
Excuse my English I think, we could just segregate tokens for decimal and real float and make them interoperable. Motivation: Most of us works with business databases - all "floats" are really decimals, algebraic operations should work without float inconsistency and those operations rare so speed is not important. But some of us use floats for speed in scientific and multimedia programs. with from __future__ import Decimal we could: a) interpret regular float constants as decimal b) interpret float constants with suffix 'f' as float (like1.5f 345.2e-5f etc) c) result of operation with decimal operands should be decimal >>> 1.0/3.0 0.3 d) result of operation with float operands should be float >>> 1.0f/3.0f 0.1f e) result of operation with decimal and float should be float (decimal converts into float and operation perfomed) >>> 1.0f/3.0 0.1f >>> 1.0/3.0f 0.1f ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] dev FAQ updated with day-to-day svn questions
I just finished fleshing out the dev FAQ (http://www.python.org/dev/devfaq.html) with questions covering what someone might need to know for regular usage. If anyone thinks I didn't cover something I should have, let me know. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter
> "Michiel" == Michiel Jan Laurens de Hoon <[EMAIL PROTECTED]> writes: Michiel> What is the advantage of Tk in comparison to other GUI Michiel> toolkits? IMO, Tk's _advantage_ is that it's there already. As a standard component, it works well for typical simple GUI applications (thus satisfying "batteries included" IMO), and it's self-contained. So I would say it's at _no disadvantage_ to other toolkits. Alternatives like PyGtk and wxWidgets are easily available and provide some degree of cross-platform support for those who need something more/different. Is there some reason why you can't require users to install a toolkit more suited to your application's needs? -- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of TsukubaTennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter
Greg Ewing wrote: >I'm not sure the event-loop situation would be >much different with another one, anyway. From what >I've seen of GUI toolkits, they all have their own >form of event loop, and they all provide some way >of hooking other things into it (as does Tkinter), >but whichever one you're using, it likes to be in >charge. > It's not because it likes to be in charge, it's because there's no other way to do it in Python. In our scientific visualization software, we also have our own event loop. I'd much rather let a Python event loop handle our messages. Not only would it save us time programming (setting up an event loop in an extension module that passes control back to Python when needed is tricky), it would also give better performance, it would work with IDLE (which an event loop in an extension module cannot as explained in my previous post), and it would let different extension modules live happily together all using the same event loop. Tkinter is a special case among GUI toolkits because it is married to Tcl. It doesn't just need to handle its GUI events, it also needs to run the Tcl interpreter in between. Which is why Tkinter needs to be in charge of the event loop. For other GUI toolkits, I don't see a reason why they'd need their own event loop. > Code which blocks reading from standard >input doesn't fit very well into any of them. > > Actually, this is not difficult to accomplish. For example, try Tcl's wish on Linux: It will pop up a (responsive) graphics window but continue to read Tcl commands from the terminal. This is done via a call to select (on Unix) or MsgWaitForMultipleObjects (on Windows). Both of these can listen for terminal input and GUI events at the same time. --Michiel. -- Michiel de Hoon Center for Computational Biology and Bioinformatics Columbia University 1150 St Nicholas Avenue New York, NY 10032 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter
Stephen J. Turnbull wrote: >Michiel> What is the advantage of Tk in comparison to other GUI >Michiel> toolkits? > >IMO, Tk's _advantage_ is that it's there already. As a standard >component, it works well for typical simple GUI applications (thus >satisfying "batteries included" IMO), and it's self-contained. So I >would say it's at _no disadvantage_ to other toolkits. > >Alternatives like PyGtk and wxWidgets are easily available and provide >some degree of cross-platform support for those who need something >more/different. > >Is there some reason why you can't require users to install a toolkit >more suited to your application's needs? > > My application doesn't need a toolkit at all. My problem is that because of Tkinter being the standard Python toolkit, we cannot have a decent event loop in Python. So this is the disadvantage I see in Tkinter. --Michiel. -- Michiel de Hoon Center for Computational Biology and Bioinformatics Columbia University 1150 St Nicholas Avenue New York, NY 10032 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak references: dereference notification
On 9-nov-2005, at 23:44, Gustavo J. A. M. Carneiro wrote: > On Wed, 2005-11-09 at 20:40 +0100, Ronald Oussoren wrote: >> On 9-nov-2005, at 18:52, Gustavo J. A. M. Carneiro wrote: >> >>> Qua, 2005-11-09 às 09:23 -0800, Guido van Rossum escreveu: >> Gustavo J. A. M. Carneiro wrote: >>> I have come across a situation where I find the current weak >>> references interface for extension types insufficient. >>> >>> Currently you only have a tp_weaklistoffset slot, pointing >>> to a >>> PyObject with weak references. However, in my case[1] I >>> _really_ need >>> to be notified when a weak reference is dereferenced. I find reading through the bug discussion a bit difficult to understand your use case. Could you explain it here? If you can't explain it you certainly won't get your problem solved! :-) >>> >>> This is a typical PyObject wrapping C object (GObject) problem. >>> Both >>> PyObject and GObject have independent reference counts. For each >>> GObject there is at most one PyObject wrapper. >>> >>> When the refcount on the wrapper drops to zero, tp_dealloc is >>> called. >>> In tp_dealloc, and if the GObject refcount is > 1, I do something >>> slightly evil: I 'resurect' the PyObject (calling PyObject_Init), >>> create >>> a weak reference to the GObject, and drop the "strong" reference. I >>> call this a 'hibernation state'. >> >> Why do you do that? The only reasons I can think of are that you hope >> to gain >> some speed from this or that you want to support weak references to >> the GObject. > > We want to support weak references to GObjects. Mainly because that > support has always been there and we don't want/can't break API. > And it > does have some uses... > >> >> For what its worth, in PyObjC we don't support weak references to the >> underlying >> Objective-C object and delete the proxy object when it is garbage >> collected. >> Objective-C also has reference counts, we increase that in the >> constructor for >> the proxy object and decrease it again in the destroctor. > > OK, but what if it is a subclass of a builtin type, with instance > variables? What if the PyObject is GC'ed but the ObjC object remains > alive, and later you get a new reference to it? Do you create a new > PyObject wrapper for it? What happened to the instance variables? Our main goal is that there is at most one wrapper for a python object alive at any one time. And likewise there is at most one Objective-C wrapper for a python object. If a PyObject is GC'ed and the ObjC object remains alive you will get a new PyObject when a reference to the ObjC object passes into python space again. That is no problem because the proxy object contains no state other than the pointer to the ObjC object. ObjC's runtime might be more flexible than that of GObject. If you create a subclass of an ObjC class the PyObjC runtime will create a real ObjC class for you and all object state, including Python instance variables, are stored on the ObjC side. > > Our goal in wrapping GObject is that, once a Python wrapper for a > GObject instance is created, it never dies until the GObject dies too. > At the same time, once the python wrapper loses all references, it > should not stop keeping the GObject alive. I tried that too, but ran into some very ugly issues and decided that weak references are not important enough for that. There's also the problem that this will keep the python proxy alive even when it is not needed anymore, which gives significant overhead of you traverse a large datastructure. What I don't quite understand is how you know that your python wrapper is the last reference to the GObject and your wrapper should not be forcefully kept alive. > > What happens currently, which is what I'm trying to change, is that > there is a reference loop between PyObject and GObject, so that > deallocation only happens with the help of the cyclic GC. But relying > on the GC for _everything_ causes annoying problems: At one time I used Python's reference counts as the reference count of the Objective-C object (ObjC's reference count management is done through method calls and can therefore be overridden in subclasses). That did work, but getting the semantics completely correct turned the code into a mess. Our current solution is much more satisfying. > > 1- The GC runs only once in a while, not soon enough if eg. you > have an > image object with several megabytes; > > 2- It makes it hard to debug reference counting bugs, as the symptom > only appears when the GC runs, far away from the code that cause the > problem in the first place; > > 3- Generally the GC has a lot more work, since every PyGTK object > needs > it, and a GUI app can have lots of PyGTK objects. > > Regards. > > -- > Gustavo J. A. M. Carneiro > <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> > The universe is always one step beyond logic > > __
Re: [Python-Dev] Weak references: dereference notification
Gustavo J. A. M. Carneiro wrote: > OK, but what if it is a subclass of a builtin type, with instance > variables? What if the PyObject is GC'ed but the ObjC object remains > alive, and later you get a new reference to it? Do you create a new > PyObject wrapper for it? What happened to the instance variables? Normally, wrappers don't have state. But if you do have state, this is how it could work: 1. Make two Python objects, PyState and PyWrapper (actually, PyState doesn't need to be a Python object) PyState holds the instance variables, and PyWrapper just holds a pointer to a GObject. 2. When a Python reference to a GObject is created for the first time, create both a PyState and a PyWrapper. Have the GObject point to the PyState, and the PyWrapper to the GObject. Have the PyState weakly reference the PyWrapper. 3. When the refcount to the PyWrapper drops to zero, discard it. 4. When somebody asks for the data in the PyWrapper, go to the GObject, then to the PyState, and return the data from there. 5. When somebody wants a reference to a GObject which already has a PyState, check the weak reference to find out whether there is a PyWrapper already. If yes, return it; if not, create a new one (and weakly reference it). 6. When the GObject is discarded, drop the PyState as well. This has the following properties: 1. There are no cyclic references for wrapping GObjects. 2. Weakly-referencing wrappers is supported; if there are no strong Python references to the wrapper, the wrapper goes away, and, potentially, the GObject as well. 3. The PyState object lives as long as the GObject. 4. Using "is" for GObjects/PyWrappers "works": there is at most one PyWrapper per GObject at any time. 5. id() of a GObject may change over time, if the wrapper becomes unreferenced and then recreated. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter
Michiel Jan Laurens de Hoon wrote: > At this point, I can't propose a specific modification yet because I > don't know the reasoning that went behind the original choice of Tk as > the default GUI toolkit for Python (and hence, I don't know if those > reasons are still valid today). I don't know, either, but I guess that it was the only option as a cross-platform GUI at the time when it was created. > I can see one disadvantage (using Tk > limits our options to run an event loop for other Python extensions), > and I am trying to find out why Tk was deemed more appropriate than > other GUI toolkits anyway. I don't think this is a disadvantage: my guess is that other GUI toolkits share the very same problems. So even though it looks like a limitation of Tkinter, it really is a fundamental limitation, and Tk is not any worse than the competitors. Also, I firmly believe that whatever your event processing requirements are, that there is a solution that meets all your end-user needs. That solution would fail the requirement to be easy to implement for you (IOW, it may take some work). > So let me rephrase the question: What is the advantage of Tk in > comparison to other GUI toolkits? It comes bundled with Python. If this sounds circular: It is. Whatever the historical reasons for original inclusion where, I'm sure they are not that important anymore. Today, what matters is that an actual implementation of a GUI toolkit integration is actively being maintained, in the Python core. This is something that is not the case for any other GUI toolkit. If you think it would be easy to change: it isn't. Somebody would have to step forward and say "I will maintain it for the next 10 years". Removal of Tkinter would meet strong resistance, so it would have to be maintained in addition to Tkinter. Nobody has stepped forward making such an offer. For Tkinter, it's different: because it *already* is part of Python, various maintainers fix problems as they find them, and contributors contribute improvements. > Is it Mac availability? More advanced > widget set? Installation is easier? Portability? These are all important, yes. But other GUI toolkits likely have the same properties. > Switching to a > different GUI toolkit would break too much existing code? Most definitely, yes. Switching would not be an option at all. Another GUI toolkit would have to be an addition, not a replacement. > I think that > having the answer to this will stimulate further development of > alternative GUI toolkits, which may give some future Python version a > toolkit at least as good as Tk, and one that doesn't interfere with > Python's event loop capabilities. I personally don't think so. The task is just too huge for volunteers to commit to. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] to_int -- oops, one step missing for use.
Well, wouldn't you know it. I get the code right and mess up the directions. Scott David Daniels wrote: > if you build this module, I'd suggest using > "from to_int import chomp" to get a function that works like int > (producing a long when needed and so on). Well, actually it is a bit more than that. "from to_int import chomp, _flag; _flag(1)" This sets a flag to suppress the return of the length along with the value from chomp. --Scott David Daniels [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter
Michiel Jan Laurens de Hoon wrote: > It's not because it likes to be in charge, it's because there's no other > way to do it in Python. As I said: this is simply not true. > Tkinter is a special case among GUI toolkits because it is married to > Tcl. It doesn't just need to handle its GUI events, it also needs to run > the Tcl interpreter in between. That statement is somewhat deceiving: there isn't much interpreter to run, really. > Which is why Tkinter needs to be in > charge of the event loop. For other GUI toolkits, I don't see a reason > why they'd need their own event loop. They need to fetch events from the operating system level, and dispatch them to the widgets. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
