Re: [Python-Dev] My summit notes (packaging)
On Mar 27, 2009, at 9:25 PM, P.J. Eby wrote:
At 03:06 PM 3/27/2009 -0500, Tarek Ziadé wrote:
They both aim at the
same goal besides a few differences, and they both rely
on a new metadata introduced by setuptools, wich is.
"install_requires". This new metadata extends the metadata.
described in PEP 314 but is slightly different from.
what is descibred in the Draft PEP 345 ("Requires").
..
PEP 345 introduces "Requires" and "Provides" wich are
are implemented in Distutils and PyP, but are not
widely used. 40 out of +4000 if I remember correctly. Martin will
correct me here if I am wrong.
FYI, The reason setuptools uses a different way of specifying
requirements is that the PEP-proposed way could not be used without
some kind of indexed repository of packages -- and PyPI did not
index "provides" at the time. Also, the PEP-proposed versioning
scheme was not compatible with the versioning schemes actually used
in the field at the time.
There's a deeper issue IMO. As Kevin pointed out, distutil's Requires
data works at the module and package level, rather than at the project
level. I can see some value in this, but I think working at the
project level is a lot simpler and more practically useful.
Jim
--
Jim Fulton
Zope Corporation
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's update CObject API so it is safe and regular!
On Mar 31, 2009, at 3:14 PM, Larry Hastings wrote: (Thanks for calling my attention to this. :) The CObject API has two flaws. First, there is no usable type safety mechanism. You can store a void *object, and a void *description. There is no established schema for the description; it could be an integer cast to a pointer, or it could point to memory of any configuration, or it could be NULL. Thus users of the CObject API generally ignore it--thus working without any type safety whatsoever. A programmer could crash the interpreter from pure Python by mixing and matching CObjects from different modules (e.g. give "curses" a CObject from "_ctypes"). The description field wasn't in the original CObject implementation that I was involved with many years ago. Looking at it now, I don't think it is intended as a type-safety mechanism at all, but as a way to pass data to the destructor. I don't know what motivated this. (I don't know why it it's called "description". This name seems to be very confusing.) The only type-safety mechanism for a CObject is it's identity. If you want to make sure you're using the foomodule api, make sure the address of the CObject is the same as the address of the api object exported by the module. The exporting module should automate use of the C API by providing an appropriate header file, as described in http://docs.python.org/extending/extending.html#providing-a-c-api-for-an-extension-module . Second, the destructor callback is defined as taking *either* one *or* two parameters, depending on whether the "descr" pointer is non- NULL. One can debate the finer points of what is and isn't defined behavior in C, but at its heart this is a sloppy API. It was necessary for backward compatibility. I don't know what motivated this, so I don't know if the benefit was worth the ugliness. MvL and I discussed this last night and decided to float a revision of the API. I wrote the patch, though, so don't blame Martin if you don't like my specific approach. The color of this particular bike shed is: * The PyCObject is now a private data structure; you must use accessors. I added accessors for all the members. The original implementation didn't expose the structure. I don't know why it was exposed. It would be backward incompatible to hide it again now. * The constructors and the main accessor (PyCObject_AsVoidPtr) now all *require* a "const char *type" parameter, which must be a non-NULL C string of non-zero length. If you call that accessor and the "type" is invalid *or doesn't match*, it fails. That would break backward compatibility. Are you proposing this for Python 3? What would be the gain in this? The CObject is already a type identifier for itself. In any case, client code generally doesn't mess with CObjects directly anyway. * The destructor now takes the PyObject *, not the PyCObject *. You must use accessors to get your hands on the data inside to free it. It currently isn't passed the CObject, but the C pointer that it holds. In any case, changing the API isn't practical, at least not for Python 2. Yes, you can easily skip around the "matching type" restriction by calling PyCObject_AsVoidPtr(cobj, PyCObject_GetType(cobj)). The point of my API changes is to *encourage* correct use. I've posted a patch implementing this change in the 3.1 trunk to the bug tracker: http://bugs.python.org/issue5630 I look forward to your comments! -1 I don't see that this gains anything. 1. All you're adding, afaict is a name for the API and the (address of the) CObject itself already provides this. 2. Only code provided by the module provider should be accessing the CObject exported by the module. Jim -- Jim Fulton Zope Corporation ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's update CObject API so it is safe and regular!
On Apr 1, 2009, at 11:51 PM, Guido van Rossum wrote: ... Note also this cheap exported-vtable hack isn't the only use of CObjects; for example _ctypes uses them to wrap plenty of one-off objects which are never set as attributes of the _ctypes module. We'd like a solution that enforces some safety for those too, without creating spurious module attributes. Why would you care about safety for ctypes? It's about as unsafe as it gets anyway. Coredump emptor I say. At which point, I wonder why we worry so much about someone intentionally breaking a CObject as in Larry's example. Jim -- Jim Fulton Zope Corporation ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's update CObject API so it is safe and regular!
On Apr 2, 2009, at 7:28 AM, Greg Ewing wrote: Jim Fulton wrote: The only type-safety mechanism for a CObject is it's identity. If you want to make sure you're using the foomodule api, make sure the address of the CObject is the same as the address of the api object exported by the module. I don't follow that. If you already have the address of the thing you want to use, you don't need a CObject. I was refering to the identity of the CObject itself. 2. Only code provided by the module provider should be accessing the CObject exported by the module. Not following that either. Without attaching some kind of metadata to a CObject, I don't see how you can know whether a CObject passed to you from Python code is one that you created yourself, or by some other unrelated piece of code. The original use case for CObjects was to export an API from a module, in which case, you'd be importing the API from the module. The presence in the module indicates the type. Of course, this doesn't account for someone intentionally replacing the module's CObject with a fake. Attaching some kind of type info to a CObject and having an easy way of checking it makes sense to me. If the existing CObject API can't be changed, maybe a new enhanced one could be added. I don't think backward compatibility needs to be a consideration for Python 3 at this point. I don't see much advantage in the proposal, but I can live with it for Python 3. Jim -- Jim Fulton Zope Corporation ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyObject_GC_UnTrack() no longer reliable in 2.7?
On Fri, Sep 24, 2010 at 3:36 PM, Antoine Pitrou wrote:
> On Fri, 24 Sep 2010 15:14:32 -0400
> Tim Peters wrote:
>> Looks like 2.7 changes introduced to exempt dicts and tuples from
>> cyclic gc if they obviously can't be in cycles has some unintended
>> consequences. Specifically, if an extension module calls
>> PyObject_GC_UnTrack() on a dict it _does not want tracked_, Python can
>> start tracking the dict again.
>>
>> I assume this is unintended because (a) the docs weren't changed to
>> warn about this; and, (b) it's wrong ;-)
>
> It was indeed unintended. I didn't know people were using
> PyObject_GC_(Un)Track in other places than constructors and destructors.
>
>> There are two main reasons an extension module may have been calling
>> PyObject_GC_UnTrack():
>>
>> 1. For correctness, if the extension is playing games with reference
>> counts Python isn't expecting.
>
> Yikes :)
>
>> 2. For speed, if the extension is creating dicts (or tuples) it knows
>> cannot participate in cycles.
>
> The optimization is now automated in the simple cases (as you've found
> out!).
>
>> This came up when Jim Fulton asked me for advice about assertion
>> failures coming out of cyclic gc in a debug build when running ZODB's
>> tests under 2.7. Turned out to be due to the "#1 reason" above: ZODB
>> hand-rolled its own version of weak references long before Python had
>> them, and has a dict mapping strings ("object IDs") to complex objects
>> where the referenced objects' refcounts intentionally do _not_ account
>> for the references due to this dict.
>
> Perhaps ZODB should switch to standard weak references these days?
> (as a bonus, chances are it will be faster)
This is the long term plan. Switching is not going to be a small
project and not high on the list of priorities.
(Actually, ZODB invented its own weakref mechanism after Python had
weakrefs, but before weakrefs were subclassable. Using standard
weakrefs was deemed too expensive in terms of memory use.)
For the record, I don't consider this a Python bug. This corner of
ZODB is living on the edge and deserves what it gets. :) I'm just
happy the fix was ultimately pretty simple.
>> Best if no changes had been needed. "Better than nothing" if the docs
>> are changed to warn that the effect of calling PyObject_GC_UnTrack()
>> may be undone by Python a nanosecond later ;-)
>
> A doc addition will be enough, hopefully.
Absolutely.
Jim
--
Jim Fulton
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] inst_persistent_id
Recently, I reviewed the documentation source for pickle and came across the following comment: BAW: Both pickle and cPickle support something called inst_persistent_id() which appears to give unknown types a second shot at producing a persistent id. Since Jim Fulton can't remember why it was added or what it's for, I'm leaving it undocumented. I couldn't remember this and decided to dig into it and realized that this was a very useful experimental feature I added way back when cPickle was in its infancy. This is a fairly useful optimization that I never got around to evaluating. (Yeah, I know.) It is a hook, like the persistent_id hook that is called with objects to determine if they should be pickled by persistent reference. Unlike persistent_id, it isn't called for built-in types (really types for which pickle has specific handlers), like strings, numbers, lists, tuples and so on. It is only called for "instances" (types for which pickle doesn't have specific handlers). This vastly reduces the number of calls to the hook. Some tests with ZODB indicated significant improvements in pickling speed when a hook is used. If there are no objections, I'll update the Python 2 documentation to describe this and add a test. The comment above suggests that this hook is in pickle and cPickle. It is in cPickle, but was removed from pickle. I propose to add it back to pickle -- mainly for consistency with cPickle. I'll need to double check how this interacts with the type dispatching in pickle protocol 2. Any objections? Jim -- Jim Fulton Zope Corporation ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] inst_persistent_id
I took Python-3000 out of the cc list as I originally just wanted to make them aware of this issue. On Jan 14, 2008, at 12:59 PM, Armin Rigo wrote: > Hi, > > On Sat, Jan 12, 2008 at 07:33:38PM -0500, Alexandre Vassalotti wrote: >> Well, in Python 3K, inst_persistent_id() won't be usable, since >> PyInstance_Type was removed. > > Looking at the code, inst_persistent_id() is just a confusing name. > It > has got nothing to do with PyInstance_Type; it's called for any object > type that cPickle.c doesn't know how to handle. It has to do with instances in a more general sense. > In fact, it seems that > cPickle.c never calls inst_persistent_id() for objects of type > PyInstance_Type... Hm, good point. At the time, all I cared about were ExtensionClass instances, which were akin to modern-day new-style instances. It doesn't make sense to not call this function for classic instances. This optimization is fairly useful. I propose to update the logic to call this function for classic instances too. I'm also open to renaming it if that would make people happier. :) Thoughts? Jim -- Jim Fulton Zope Corporation ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Released fixes for CVE-2008-2315 for Python 2.4?
Are there plans for a 2.4 release that includes: http://bugs.python.org/issue2620, and http://svn.python.org/view?rev=65335&view=rev ? These (and other fixes) address: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-2315 Jim -- Jim Fulton Zope Corporation ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python 2.4] PyInt_FromLong returning NULL
Andreas Jung wrote: While using Zope 2.7 with Python 2.4 we discovered some strange behaviour of the security machinery. I could track this down to some Zope code in cAccessControl.c where an Unauthorized exception is raised because of a call to PyInt_FromLong(1) which returns NULL. What could be the reason that such a "stupid" call return NULL in a reproducable way? Ugh. Part of the problem is that all of those calls are unchecked, Dang us. If they were checked, then, who knows, we might have gotten informative exceptions. I'd say the first step should be to add checks. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python 2.4] PyInt_FromLong returning NULL
Tim Peters wrote: [Jim Fulton] Ugh. Part of the problem is that all of those calls are unchecked, Dang us. If they were checked, then, who knows, we might have gotten informative exceptions. They certainly "should" be checked, but as a pragmatic matter PyInt_FromLong(1) can't fail -- Python allocates an int object for 1 (and for about 100 other popular little integers) when it starts up, and PyInt_FromLong() just returns a new reference to these pre-existing objects whenever possible. I know. I'm sure that's why we don't bother. But, obviously, it can fail. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python 2.4] PyInt_FromLong returning NULL
Tim Peters wrote:
[Andreas Jung]
Sorry, false alarm :-( There assignment of the NULL occurs in the
if-clause of the corresponding code (I have overseen the ASSIGN
call):
Thanks for the followup!
if (! PyInt_Check(p))
{
if (PyDict_Check(p))
{
if (PyString_Check(name) ||
PyUnicode_Check(name))
{
ASSIGN(p, PyObject_GetItem(p, name));
^^
if (p == NULL) {
puts("PyObject returned NULL");
PyErr_Clear();
}
}
else
p = PyInt_FromLong((long)1);
...doing some further investigations on that.
I note that all of this is nested inside another "if (p) {...}" block.
That implies the "p = PyInt_FromLong((long)1);" line is at least a
memory leak: it overwrites p without decref'ing p first.
The ASSIGN macro DECREFs it's first argument if it is non-NULL.
It loosly models a Python assignment, assuming that it owns the
reference to the second argument.
Jim
--
Jim Fulton mailto:[EMAIL PROTECTED] Python Powered!
CTO (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
___
Python-Dev mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's get rid of unbound methods
Guido van Rossum wrote:
In my blog I wrote:
Let's get rid of unbound methods. When class C defines a method f, C.f
should just return the function object, not an unbound method that
behaves almost, but not quite, the same as that function object. The
extra type checking on the first argument that unbound methods are
supposed to provide is not useful in practice (I can't remember that
it ever caught a bug in my code) and sometimes you have to work around
it; it complicates function attribute access;
I think this is probably a good thing as it potentially avoids
some unintential aliasing.
> and the overloading of
unbound and bound methods on the same object type is confusing. Also,
the type checking offered is wrong, because it checks for subclassing
rather than for duck typing.
duck typing?
This is a really simple change to begin with:
*** funcobject.c 28 Oct 2004 16:32:00 - 2.67
--- funcobject.c 4 Jan 2005 18:23:42 -
***
*** 564,571
static PyObject *
func_descr_get(PyObject *func, PyObject *obj, PyObject *type)
{
! if (obj == Py_None)
! obj = NULL;
return PyMethod_New(func, obj, type);
}
--- 564,573
static PyObject *
func_descr_get(PyObject *func, PyObject *obj, PyObject *type)
{
! if (obj == NULL || obj == Py_None) {
! Py_INCREF(func);
! return func;
! }
return PyMethod_New(func, obj, type);
}
There are some test suite failures but I suspect they all have to do
with checking this behavior.
Of course, more changes would be needed: docs, the test suite, and
some simplifications to the instance method object implementation in
classobject.c.
Does anyone think this is a bad idea?
It *feels* very disruptive to me, but I'm probably wrong.
We'll still need unbound builtin methods, so the concept won't
go away. In fact, the change would mean that the behavior between
builtin methods and python methods would become more inconsistent.
Jim
--
Jim Fulton mailto:[EMAIL PROTECTED] Python Powered!
CTO (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's get rid of unbound methods
Phillip J. Eby wrote: At 10:28 AM 1/4/05 -0800, Guido van Rossum wrote: Of course, more changes would be needed: docs, the test suite, and some simplifications to the instance method object implementation in classobject.c. Does anyone think this is a bad idea? Code that currently does 'aClass.aMethod.im_func' in order to access the function object would break, as would code that inspects 'im_self' to determine whether a method is a class or instance method. (Although code of the latter sort would already break with static methods, I suppose.) Code of the latter sort wouldn't break with the change. We'd still have bound methods. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's get rid of unbound methods
Guido van Rossum wrote: [Jim] We'll still need unbound builtin methods, so the concept won't go away. In fact, the change would mean that the behavior between builtin methods and python methods would become more inconsistent. Actually, unbound builtin methods are a different type than bound builtin methods: Of course, but conceptually they are similar. You would still encounter the concept if you got an unbound builtin method. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's get rid of unbound methods
Armin Rigo wrote: Hi Jim, On Tue, Jan 04, 2005 at 02:44:43PM -0500, Jim Fulton wrote: Actually, unbound builtin methods are a different type than bound builtin methods: Of course, but conceptually they are similar. You would still encounter the concept if you got an unbound builtin method. There are no such things as unbound builtin methods: list.append is list.__dict__['append'] True In other words 'list.append' just returns exactly the same object as stored in the list type's dict. Guido's proposal is to make Python methods behave in the same way. OK, interesting. I'm sold then. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 246, redux
Clark C. Evans wrote: On Tue, Jan 11, 2005 at 12:54:36PM -0500, Phillip J. Eby wrote: ... | * In my experience, incorrectly deriving an interface from another is the | most common source of unintended adaptation side-effects, not adapter | composition It'd be nice if interfaces had a way to specify a test-suite that could be run against a component which claims to be compliant. For example, it could provide invalid inputs and assert that the proper errors are returned, etc. We've tried this in Zope 3 with very limited success. In fact, so far, our attempts have provided more pain than their worth. The problem is that interfaces are usually abstract enough that it's very difficult to write generic tests. For example, many objects implement mapping protocols, but place restrictions on the values stored. It's hard to provide generic tests that don't require lots of inconvenient hooks. There are exceptions of course. Our ZODB storage tests use a generic storage-interface test, but this is possible because the ZODB storage interfaces are extremely concrete. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] rationale for the no-new-features approach
+1 (wish * could say +sys.maxint).
To emphasize:
- We want people to update to bug fix releases quickly. For people
to do that, they need to know the risk of breakage is low. Reducing the
scope of the release is important for that.
- Python has gotten much better at this than it used to be. I remember the
days when major features in 3rd-dot releases caused major headaches for us.
I think that made Python look bad.
Jim
Anthony Baxter wrote:
So it's only fair that I write down my rationale for why I'm being anal
about the no-new-features approach. Comments are more than welcome -
ideally, after discussion, I'll put some more words in the bugfix PEP.
Goal 1: When we cut a bugfix release, people will upgrade to it.
- This helps the users (they have bugs fixed)
- This helps us (python-dev) because people won't be logging
bugs against already-fixed-bugs.
- This makes us (Python) look good, because people using
Python have the most bug-free experience possible.
Goal 2: Packagers of Python will package up releases of Python
that are as close to the "official" release as possible.
- This, to me, is a huge win. If we don't have to worry about
whether someone is running 2.4.1, or DistroFoo's version of
2.4.1 with a couple of backported fixes from 2.4.2, or some
other horrible frankenstein's monster of a release, it makes
our lives much easier. (This is also why I've been on a bit of
a stomping exercise to attempt to get distros that broke Python
into pieces to stop doing this (notably, Debian/Ubuntu's distutils-
in-python-devel pain))
- And yes, we're always going to have the problem of some distros
stuck on old Python releases (Redhat -> Python 1.5.2, Debian stable
-> Python 2.1, &c) but we can hopefully encourage them to at least
roll out the latest bugfix version of whatever version they're stuck on.
Goal 3: Good PR.
- I've read a disturbing amount of words about other
languages (*cough*Java*cough*) that have no sanity about
minor and major releases. This seems to piss people off a
great deal. One of Python's selling points is that we're very
cautious and a suitable choice for grownups to use in business.
- This is good for us (Python community) because it makes it
less likely we'll be stuck in a job where we're forced to code
Perl, or C++, or Java, or some other horrible fate. It also boosts
the size of the community, meaning that there will be more
volunteers to work on Python (hurrah!)
Goal 4: Try and prevent something like
try:
True, False
except NameError:
True, False = 1, 0
from ever ever happening again.
- This, I hope, needs no further explanation
Anthony
--
Jim Fulton mailto:[EMAIL PROTECTED] Python Powered!
CTO (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Security capabilities in Python
You might take a look at zope.security: http://svn.zope.org/Zope3/trunk/src/zope/security/ It isn't a capability-based system, but it does address similar problems and might have some useful ideas. See the README.txt and untrustedinterpreter.txt. Jim Eyal Lotem wrote: I would like to experiment with security based on Python references as security capabilities. Unfortunatly, there are several problems that make Python references invalid as capabilities: * There is no way to create secure proxies because there are no private attributes. * Lots of Python objects are reachable unnecessarily breaking the principle of least privelege (i.e: object.__subclasses__() etc.) I was wondering if any such effort has already begun or if there are other considerations making Python unusable as a capability platform? (Please cc the reply to my email) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jim%40zope.com -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: anonymous blocks
Guido van Rossum wrote: I've written a PEP about this topic. It's PEP 340: Anonymous Block Statements (http://python.org/peps/pep-0340.html). Some highlights: - temporarily sidestepping the syntax by proposing 'block' instead of 'with' - __next__() argument simplified to StopIteration or ContinueIteration instance - use "continue EXPR" to pass a value to the generator - generator exception handling explained This looks pretty cool. Some observations: 1. It looks to me like a bare return or a return with an EXPR3 that happens to evaluate to None inside a block simply exits the block, rather than exiting a surrounding function. Did I miss something, or is this a bug? 2. I assume it would be a hack to try to use block statements to implement something like interfaces or classes, because doing so would require significant local-variable manipulation. I'm guessing that either implementing interfaces (or implementing a class statement in which the class was created before execution of a suite) is not a use case for this PEP. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: anonymous blocks
Duncan Booth wrote: Jim Fulton <[EMAIL PROTECTED]> wrote in news:[EMAIL PROTECTED]: Guido van Rossum wrote: I've written a PEP about this topic. It's PEP 340: Anonymous Block Statements (http://python.org/peps/pep-0340.html). Some observations: 1. It looks to me like a bare return or a return with an EXPR3 that happens to evaluate to None inside a block simply exits the block, rather than exiting a surrounding function. Did I miss something, or is this a bug? No, the return sets a flag and raises StopIteration which should make the iterator also raise StopIteration at which point the real return happens. Only if exc is not None The only return in the pseudocode is inside "if exc is not None". Is there another return that's not shown? ;) I agree that we leave the block, but it doesn't look like we leave the surrounding scope. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Anonymous blocks: Thunks or iterators?
Greg Ewing wrote: Elegant as the idea behind PEP 340 is, I can't shake the feeling that it's an abuse of generators. It seems to go to a lot of trouble and complication so you can write a generator and pretend it's a function taking a block argument. I'd like to reconsider a thunk implementation. It would be a lot simpler, doing just what is required without any jiggery pokery with exceptions and break/continue/return statements. It would be easy to explain what it does and why it's useful. "Simple is better than Complex." Is there a thunk PEP? Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
Tim Peters wrote: ... > I'm sending this to Jim Fulton because he did the conversion of Zope > Corp's code base to SVN. Unfortunately, Jim will soon be out of touch > for several weeks. Jim, do you have time to summarize the high bits > of the problems you hit? IIRC, you didn't find any conversion script > at the time capable of doing the whole job as-is. If that's wrong, it > would be good to know that too. I had two problems, one of which was zope specific: 1. We were making extensive use of symbolic links to share packages among various Zope projects. This requires special care and was the main reason I wrote my own scrips. I don't expect that this would be an issue for Python. 2. I initially tried to conver our entire repository, including all branches. This would have taken days. I finally decided to just convert our trunk, which took several hours. The main time sink was in the load step of the conversion process. I suspect that much of the time hit was due to the Berkely DB back end. I think I've heard that the file-system-based back end is much faster in general. We're using the BDB back end because that's all that was available at the time we converted. Every few weeks, the database gets wedged and I have to do a recovery operation. Every few months, the machine gets wedged and required a reboot. I'm pretty sure the later is due to locking issues. I definately recommend the file-system back-end, even though I haven't tried it myself. I haven't had the time to do the conversion myself. :( I'll also say that, on balance, I'm *very* *very* happy with subversion. I recommend it highly. > Other than that, I'd just like to see an explicit mention in the PEP > of a plan to preserve the pre-conversion CVS tarball forever. +1 Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
Tim Peters wrote: > [Jeff Rush] > >>The conversion script isn't perfect and does fail on complex CVS >>arrangements or where there is extensive history to migate. However it >>appears above that Martin has already tried the script out, with success. > > > I'd still like to hear from Jim, as I don't believe all serious > problems were identified by eyeball at once. I'm not aware of any errors in the conversion. ... > Ah, before I forget, "single repository" has worked very well for Zope Yup. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
Martin v. Löwis wrote: > Tim Peters wrote: > >>Ah, before I forget, "single repository" has worked very well for Zope >>(which includes top-level Zope2, Zope3, ZODB, ZConfig, zdaemon, ... >>projects): >> >>http://svn.zope.org/ >> >>Long URLs don't really get in the way in practice (rarely a need to >>type one after initial checkout; even "svn switch" is usually just a >>tail-end cmdline edit starting from a copy+paste of "svn info" >>output). > > > That would indeed give conversion problems: Nah > cvs2svn automatically > generates tags/trunk/branches in the root, with the original CVS > modules below; there is a single space for tags and branches > (as is in the original CVS repository). > > I would see two possible conversion procedures: > 1. convert the modules one-by-one, adding to the same repository. >I.e. python would get all the small revision numbers, then >peps, then sandbox, and so on. Cross-module tags would not >be supported (but likely don't exist in the first place), >and the revision number would not increase in historical order. > 2. convert the entire CVS, then rearrange things through >svn move. This would be tedious to do (atleast for tags/branches), >and it would cause all files to be renamed right from the >scratch (some svn clients fail to display history past moves). You reminded me of another reason why I used a custom conversion script. :) I did convert projects individually. I told cvs2svn to just create dump files. I then used svnload to load the dump files myself so that I could make each project a top-level directory with it's own trunk, branches and tags. I'd be happy to share my scrips, although there's nothing all that complicated about them. > So for all who would prefer to see a single repository, could > somebody please > 1. state how precisely the repository should be organized >(with exact URLs on svn.python.org - eg. which of the >non-dist directories becomes toplevel, which ones >get their own tags/branches/trunk, etc). I'm not close enough to the Python repo to offer much of an opinion other than: - IMO, a single repository is good - The repository should be orgabized by "projects", where separate projects reflect more or less independent development efforts with their own teams and schedules. (Maybe Python doesn't have many of these.) > 2. state how the conversion should be performed Once you decide what the projects are, I suggest converting each project separately. I'd be happy to share my scrips and experenice, although, as Tim noted I'll be off-line for two or three weeks starting in the next few days. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
Martin v. Löwis wrote: > Jim Fulton wrote: > >> I don't expect that this would be an issue for Python. > > > Right. > > >>2. I initially tried to conver our entire repository, including all >> branches. This would have taken days. I finally decided to just >> convert our trunk, which took several hours. The main time >> sink was in the load step of the conversion process. >> >> I suspect that much of the time hit was due to the Berkely DB >> back end. I think I've heard that the file-system-based back end >> is much faster in general. > > > Dunno. For the Python CVS (which translates into some 40,000 revisions), > on the machine which I was originally using (1GHz Pentium), conversion > took 7h. On my current workstation, it takes a little over an hour. Was this with the file-system back end? What is your current system? Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
Martin v. Löwis wrote: > Jim Fulton wrote: > >>I did convert projects individually. I told cvs2svn to just create dump >>files. I then used svnload to load the dump files myself so that >>I could make each project a top-level directory with it's own >>trunk, branches and tags. >> >>I'd be happy to share my scrips, although there's nothing all that >>complicated about them. > > > If that's how it worked, I'm sure I can cook my own. Just for > confirmation: the svn revision numbers don't increase > chronologically across 'modules', right: i.e. the first > revision of the module that was converted second has a higher > revision than the last revision of the first module, even > though historically, the order should have been reverse. > > Not that this worries me much, but I'd like to confirm there > is no other way. Right. The revision numbers reflect the order in which they are added to the svn repo. I'm pretty sure the revision meta data, in particular the date, was retained. This is an advantage advantage of using the low-level dump/load mechanism. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] RFC: readproperty
Read descriptors (defining only __get__) and data descripttors have (defining __get__ and __set__) different semantics. In particular, read descriptors are overridden by instance data, while data descriptors are not. A common use of read descriptors is for lazily computed data: class readproperty(object): "Create a read descriptor from a function" def __init__(self, func): self.func = func def __get__(self, inst, class_): if inst is None: return self return self.func(inst) class Spam: @readproperty def eggs(self): ... expensive computation of eggs self.eggs = result return result When we ask for the eggs attribute the first time, we call the descriptor, which calls the eggs function. The function sets the eggs attribute, overriding the descriptor. The next time the eggs attribute is accessed, the saved value will be used without calling the descriptor. I do this often enough that I think it would be useful to include it in python, either as a builtin (like property) or in the library. (Or possibly by adding an option to property to generate a read descriptor.) I'd be happy to add this for 2.5. Thoughts? Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: readproperty
Guido van Rossum wrote: > On 9/28/05, Jim Fulton <[EMAIL PROTECTED]> wrote: > ... > I think we need to be real careful with chosing a name -- in Jim's > example, *anyone* could assign to Spam().eggs to override the value. > The name "readproperty" is too close to "readonlyproperty", In fact, property creates read-only properties for new-style classes. (I hadn't realized, until reading this thread, that for classic classes, you could still set the attribute.) > but > read-only it ain't! "Lazy" also doesn't really describe what's going > on. I agree. > I believe some folks use a concept of "memo functions" which resemble > this proposal except the notation is different: IIRC a memo function > is always invoked as a function, but stores its result in a private > instance variable, which it returns upon subsequent calls. This is a > common pattern. Jim's proposal differs because the access looks like > an attribute, not a method call. Still, perhaps memoproperty would be > a possible name. > > Another way to look at the naming problem is to recognize that the > provided function really computes a default value if the attribute > isn't already set. So perhaps defaultproperty? Works for me. Oleg Broytmann wrote: > On Wed, Sep 28, 2005 at 10:16:12AM -0400, Jim Fulton wrote: > >> class readproperty(object): > > [skip] > >>I do this often enough > > >I use it since about 2000 often enough under the name CachedAttribute: > > http://cvs.sourceforge.net/viewcvs.py/ppa/qps/qUtils.py Steven Bethard wrote: > Jim Fulton wrote: > ... > I've also needed behavior like this a few times, but I use a variant > of Scott David Daniel's recipe[1]: > > class _LazyAttribute(object): Yup, the Zope 3 sources have something very similar: http://svn.zope.org/Zope3/trunk/src/zope/cachedescriptors/property.py?view=markup I actually think this does too much. All it saves me, compared to what I proposed is one assignment. I'd rather make that assignment explicit. Anyway, all I wanted with readproperty was a property that implemented only __get__, as opposed to property, which implements __get__, __set__, and __delete__. I'd be happy to call it readproprty or getproperty or defaulproperty or whatever. :) I'd prefer that it's semantics stay fairly simple though. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency
Michael Hudson wrote: > Bruce Eckel <[EMAIL PROTECTED]> writes: > > >>I'd like to restart this discussion; I didn't mean to put forth active >>objects as "the" solution, only that it seems to be one of the better, >>more OO solutions that I've seen so far. >> >>What I'd really like to figure out is the "pythonic" solution for >>concurrency. Guido and I got as far as agreeing that it wasn't >>threads. >> >>Here are my own criteria for what such a solution would look like: > > > Just because I've been mentioning it everywhere else since I read it, > have you seen this paper: > > http://research.microsoft.com/Users/simonpj/papers/stm/ > > ? I don't know how applicable it would be to Python but it's well > worth the time it takes to read. I haven't read more than the abstract and the responses to the paper here, but I think I have the gist. (I look forward to reading it later.) I'll note that we have experience with something less rigorous than but otherwise similar to this with ZODB. We often have tens of processes working on the same data, synchronizing through a common object store. Each process executes transactions which are written without any threading code at all. Processes synchronize through transaction commits. This model has worked very well for Zope, at least as long as conflicts can be kept to a minimum. :) The ZODB approach is less rigorous as it only works when your processes operate soley on database objects, but it provides a useful example, I think, of applying this model in Python and it probably covers a reasonably large set of interesting applications. There are hundreds (possibly thousands) of programmers who have used this technique in Zope, probably without even realizing that they were doing concurrent programming. J2EE systems use a similar approach. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Divorcing str and unicode (no more implicit conversions).
Martin Blais wrote: > Hi. > > Like a lot of people (or so I hear in the blogosphere...), I've been > experiencing some friction in my code with unicode conversion > problems. Even when being super extra careful with the types of str's > or unicode objects that my variables can contain, there is always some > case or oversight where something unexpected happens which results in > a conversion which triggers a decode error. str.join() of a list of > strs, where one unicode object appears unexpectedly, and voila! > exception galore. Sometimes the problem shows up late because your > test code doesn't always contain accented characters. I'm sure many > of you experienced that or some variant at some point. > > I came to realize recently that this problem shares strong similarity > with the problem of implicit type conversions in C++, or at least it > feels the same: Stuff just happens implicitly, and it's hard to track > down where and when it happens by just looking at the code. Part of > the problem is that the unicode object acts a lot like a str, which is > convenient, but... I agree. I think it was a mistake to implicitly convert mixed string expressions to unicode. > What if we could completely disable the implicit conversions between > unicode and str? In other words, if you would ALWAYS be forced to > call either .encode() or .decode() to convert between one and the > other... wouldn't that help a lot deal with that issue? Perhaps. > How hard would that be to implement? Not hard. We considered doing it for Zope 3, but ... > Would it break a lot of code? Yes. > Would some people want that? No, I wouldn't want lots of code to break. ;) > (I know I would, at least for some of my > code.) It seems to me that this would make the code more explicit and > force the programmer to become more aware of those conversions. Any > opinions welcome. I think it's too late to change this. I wish it had been done differently. (OTOH, I'm very happy we have Unicode support, so I'm not really complaining. :) I'll note that this hasn't been that much of a problem for us in Zope. We follow the strategy: Antoine Pitrou wrote: ... > A good rule of thumb is to convert to unicode everything that is > semantically textual, and to only use str for what is to be semantically > treated as a string of bytes (network packets, identifiers...). This is > also, AFAIU, the semantic model which is favoured for a hypothetical > future version of Python. This approach has worked pretty well for us. Still, when there is a problem, it's a real pain to debug because the error occurs too late, as you point out. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Divorcing str and unicode (no more implicit conversions).
M.-A. Lemburg wrote:
> Michael Hudson wrote:
>
>>Martin Blais <[EMAIL PROTECTED]> writes:
>>
>>
>>
>>>What if we could completely disable the implicit conversions between
>>>unicode and str? In other words, if you would ALWAYS be forced to
>>>call either .encode() or .decode() to convert between one and the
>>>other... wouldn't that help a lot deal with that issue?
>>
>>
>>I don't know. I've made one or two apps safe against this and it's
>>mostly just annoying.
>>
>>
>>>How hard would that be to implement?
>>
>>import sys
>>reload(sys)
>>sys.setdefaultencoding('undefined')
>
>
> You shouldn't post tricks like these :-)
>
> The correct way to change the default encoding is by
> providing a sitecustomize.py module which then call the
> sys.setdefaultencoding("undefined").
This is a much more evil trick IMO, as it affects all Python code,
rather than a single program.
I would argue that it's evil to change the default encoding
in the first place, except in this case to disable implicit
encoding or decoding.
Jim
--
Jim Fulton mailto:[EMAIL PROTECTED] Python Powered!
CTO (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency
Shane Hathaway wrote: > Antoine Pitrou wrote: > >>A relational database has a very strict and regular data model. Also, it >>has transactions. This makes it easy to precisely define concurrency at >>the engine level. >> >>To apply the same thing to Python you would at least need : >> 1. a way to define a subset of the current bag of reachable objects >>which has to stay consistent w.r.t. transactions that are applied to it >>(of course, you would have several such subsets in any non-trivial >>application) >> 2. a way to start and end a transaction on a bag of objects (begin / >>commit / rollback) >> 3. a precise definition of the semantics of "consistency" here : for >>example, only one thread could modify a bag of objects at any given >>time, and other threads would continue to see the frozen, stable version >>of that bag until the next version is committed by the writing thread >> >>For 1), a helpful paradigm would be to define an object as being the >>"root" of a bag, and all its properties would automatically and >>recursively (or not ?) belong to this bag. One has to be careful that no >>property "leaks" and makes the bag become the set of all reachable >>Python objects (one could provide a means to say that a specific >>property must not be transitively put in the bag). Then, use >>my_object.begin_transaction() and my_object.commit_transaction(). >> >>The implementation of 3) does not look very obvious ;-S > > > Well, I think you just described ZODB. ;-) I'd be happy to explain how > ZODB solves those problems, if you're interested. > > However, ZODB doesn't provide locking, and that bothers me somewhat. If > two threads try to modify an object at the same time, one of the threads > will be forced to abort, unless a method has been defined for resolving > the conflict. If there are too many writers, ZODB crawls. ZODB's > strategy works fine when there aren't many conflicting, concurrent > changes, but the complex locking done by relational databases seems to > be required for handling a lot of concurrent writers. I don't think it would be all that hard to use a locking (rather than a time-stamp) strategy for ZODB, although ZEO would make this extra challenging. In any case, the important thing to agree on here is that transactions provide a useful approach to concurrency control in the case where - separate control flows are independent, and - we need to mediate access to shared resources. Someone else pointed out essentially the same thing at the beginning of this thread. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] defaultproperty (was: Re: RFC: readproperty)
Based on the discussion, I think I'd go with defaultproperty. Questions: - Should this be in builtins, alongside property, or in a library module? (Oleg suggested propertytools.) - Do we need a short PEP? Jim Jim Fulton wrote: > Guido van Rossum wrote: > >>On 9/28/05, Jim Fulton <[EMAIL PROTECTED]> wrote: >> > > ... > >>I think we need to be real careful with chosing a name -- in Jim's >>example, *anyone* could assign to Spam().eggs to override the value. >>The name "readproperty" is too close to "readonlyproperty", > > > In fact, property creates read-only properties for new-style classes. > (I hadn't realized, until reading this thread, that for classic > classes, you could still set the attribute.) > > > but > >>read-only it ain't! "Lazy" also doesn't really describe what's going >>on. > > > I agree. > > >>I believe some folks use a concept of "memo functions" which resemble >>this proposal except the notation is different: IIRC a memo function >>is always invoked as a function, but stores its result in a private >>instance variable, which it returns upon subsequent calls. This is a >>common pattern. Jim's proposal differs because the access looks like >>an attribute, not a method call. Still, perhaps memoproperty would be >>a possible name. >> >>Another way to look at the naming problem is to recognize that the >>provided function really computes a default value if the attribute >>isn't already set. So perhaps defaultproperty? > > > Works for me. > > Oleg Broytmann wrote: > > On Wed, Sep 28, 2005 at 10:16:12AM -0400, Jim Fulton wrote: > > > >> class readproperty(object): > > > > [skip] > > > >>I do this often enough > > > > > >I use it since about 2000 often enough under the name CachedAttribute: > > > > http://cvs.sourceforge.net/viewcvs.py/ppa/qps/qUtils.py > > Steven Bethard wrote: > > Jim Fulton wrote: > > > ... > > I've also needed behavior like this a few times, but I use a variant > > of Scott David Daniel's recipe[1]: > > > > class _LazyAttribute(object): > > > Yup, the Zope 3 sources have something very similar: > > http://svn.zope.org/Zope3/trunk/src/zope/cachedescriptors/property.py?view=markup > > I actually think this does too much. All it saves me, compared to what I > proposed > is one assignment. I'd rather make that assignment explicit. > > Anyway, all I wanted with readproperty was a property that implemented only > __get__, as opposed to property, which implements __get__, __set__, and > __delete__. > > I'd be happy to call it readproprty or getproperty or defaulproperty or > whatever. :) > > I'd prefer that it's semantics stay fairly simple though. > > > Jim > -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] defaultproperty
Guido van Rossum wrote: > On 10/9/05, Jim Fulton <[EMAIL PROTECTED]> wrote: > >>Based on the discussion, I think I'd go with defaultproperty. > > > Great. > > >>Questions: >> >>- Should this be in builtins, alongside property, or in >> a library module? (Oleg suggested propertytools.) >> >>- Do we need a short PEP? > > > I think so. From the responses I'd say there's at most lukewarm > interest (including from me). Hm, I saw several responses from people who'd built something quite similar. This suggests to me that this is a common need. > You might also want to drop it and just > add it to your personal (or Zope's) library. I have something like this in Zope's library. I end up with a very small package that isn't logically part of other packages, but that is a dependency of lots of packages. I don't like that, but I guess I should get over it. I must say that I am of 2 minds about things like this. On the one hand, I'd like Python's standard library to be small with packaging systems to provide "extra batteries". OTOH, I often find small tools like this that would be nice to have readily available. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Comparing date+time w/ just time
[EMAIL PROTECTED] wrote: > With significant input from Fred I made some changes to xmlrpclib a couple > months ago to better integrate datetime objects into xmlrpclib. That raised > some problems because I neglected to add support for comparing datetime > objects with xmlrpclib.DateTime objects. (The problem showed up in > MoinMoin.) I've been working on that recently (adding rich comparison > methods to DateTime while retaining __cmp__ for backward compatibility), and > have second thoughts about one of the original changes. > > I tried to support datetime, date and time objects. My problems are with > support for time objects. Marshalling datetimes as xmlrpclib.DateTime > objects is no problem (though you lose fractions of a second). Marshalling > dates is reasonable if you treat the time as 00:00:00. I don't think that is reasonable at all. I would normally expect a date to represent the whole day, not a particular, unspecified time. Other people may have other expectations, but xmlrpclib should not assume a particular interpretation. > I decided to marshal > datetime.time objects by fixing the day portion of the xmlrpclib.DateTime > object as today's date. That's the suspect part. Very very suspect. :) > When I went back recently to add better comparison support, I decided to > compare xmlrpclib.DateTime objects with time objects by simply comparing the > HH:MM:SS part of the DateTime with the time object. That's making me a bit > queazy now. datetime.time(hour=23) would compare equal to any DateTime with > its time equal to 11PM. Under the rule, "in the face of ambiguity, refuse > the temptation to guess", I'm inclined to dump support for marshalling and > comparison of time objects altogether. Do others agree that was a bad idea? I agree that it was a bad idea and that you should not try to marshal time objects or compare time objects with DateTime objects. Similarly, I strongly recommend that you also stop trying to marshal date objects or compare date objects to DateTime objects. After all, if the datetime module doesn't allow compatison of date and datetime, why should you try to compare date and DateTime? Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Freezing the CVS on Oct 26 for SVN switchover
Jeremy Hylton wrote: > Can anyone point an old CVS/Perforce-Luddite at instructions for how > to use the new SVN repository? And can you remind us where to send our public keys? :) Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] For Python 3k, drop default/implicit hash, and comparison
The recent discussion about what the default hash and equality comparisons should do makes me want to chime in. IMO, the provision of defaults for hash, eq and other comparisons was a mistake. I'm especially sensitive to this because I do a lot of work with persistent data that outlives program execution. For such objects, memory address is meaningless. In particular, the default ordering of objects based in address has caused a great deal of pain to people who store data in persistent BTrees. Oddly, what I've read in these threads seems to be arguing about which implicit method is best. The answer, IMO, is to not do this implicitly at all. If programmers want their objects to be hashable, comparable, or orderable, then they should implement operators explicitly. There could even be a handy, but *optional*, base class that provides these operators based on ids. This would be too big a change for Python 2 but, IMO, should definately be made for Python 3k. I doubt any change in the default definition of these operations is practical for Python 2. Too many people rely on them, usually without really realizing it. Lets plan to stop guessing how to do hash and comparison. Explicit is better than implicit. :) Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] For Python 3k, drop default/implicit hash, and comparison
Guido van Rossum wrote: > On 11/6/05, Jim Fulton <[EMAIL PROTECTED]> wrote: > ... > Except that I really don't think that there's anything wrong with a > default __eq__ that uses object identity. As Martin pointed out, it's > just too weird that an object wouldn't be considered equal to itself. > It's the default __hash__ and __cmp__ that mess things up. Good point. I agree. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak references: dereference notification
Gustavo J. A. M. Carneiro wrote: > Hello, > > I have come across a situation where I find the current weak > references interface for extension types insufficient. > > Currently you only have a tp_weaklistoffset slot, pointing to a > PyObject with weak references. However, in my case[1] I _really_ need > to be notified when a weak reference is dereferenced. What happens now > is that, when you call a weakref object, a simple Py_INCREF is done on > the referenced object. It would be easy to implement a new slot to > contain a function that should be called when a weak reference is > dereferenced. Or, alternatively, a slot or class attribute that > indicates an alternative type that should be used to create weak > references: instead of the builtin weakref object, a subtype of it, so > you can override tp_call. > > Does this sounds acceptable? Since you can now (as of 2.4) subclass the weakref.ref class, you should be able to do this yourself in Python. See for example, weakref.KeyedRef. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ElementTree - Why not part of the core? (fwd)
Martin v. Löwis wrote: > [EMAIL PROTECTED] wrote: > >>Passing along from c.l.py. I think ElementTree is the poster child for >>best-of-breed code belonging in the standard distribution. > > > That's primarily for the author of the software to decide, at this > point. Fredrik Lundh would have to offer it for contribution first. > > I don't know what his current position is, but I think it is unlikely > that he will contribute it: in the past, he often indicated that he > a) dislikes the growth of the standard Python library, and > b) dislikes forking his own branch for inclusion in another package > (which would happen if he contributed one version for the > standard library, and would continue to maintain the code > outside of Python also). Ooh. Well said. I agree with both of these points. :) +1 > That said, I agree that ElementTree would be a valuable addition > to the Python library, and has certainly passed the "collect feedback > in the real world" test. I hope that packaging progress will someday make it matter much less whether something is in the standard library. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ElementTree - Why not part of the core? (fwd)
Steve Holden wrote: > Jim Fulton wrote: > [...] > >>I hope that packaging progress will someday make it matter much less >>whether something is in the standard library. >> > > For which we need a *mechanism* that all package providers can > implement, rather than a repository to which all package providers must > contribute. I think we need both. We need the mechanism and repositories, although non necessarily one repository. Phillip Eby and others seem to be making wonderful progress on the mechanism. I haven't had a chance to play with this yet, but I expect to over the next few months. People who have seem quite enthusiastic. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ElementTree - Why not part of the core? (fwd)
[EMAIL PROTECTED] wrote: > Jim> I hope that packaging progress will someday make it matter much > Jim> less whether something is in the standard library. > > It undoubtedly will. The point I was trying to raise here is that > ElementTree is so much better than the stuff we currently distribute (*) > that it should be included in the standard distribution if for no other > reason than to discourage use of the current stuff in new applications. Maybe we should deprecate the current stuff. It's been done before. Of course, much of the current XML support is still useful, if only because lots of existing 3rd-party code depend on it. When we have a packaging system we could move these out of the core without disparaging them and without breaking third-party modules. > > Here are a couple perhaps useful BDFL references: > > http://mail.python.org/pipermail/python-dev/2003-December/040928.html > http://mail.python.org/pipermail/python-dev/2003-April/034881.html > > In the first, Guido tells a potential submitter to "do the math" to make > sure his package is "best of breed". In the second Guido warns that having > code in the standard distribution tends to suppress usage of other packages, > even though they may be better: > > We can't put every approach in the core, but putting one package in > the core may damage the viability of another, possibly better (for > some users) solution. To some extent this has happened with GUI > toolkits: the presence of Tkinter in the core makes it harder for > other GUI toolkits to compete (leaving aside whether Tkinter is > better or not -- it's just not a level playing field). Well said. +1 I agree with this too. :) > I think that's sort of the reverse of the point I'm trying to make. But it's one of the reasons why I'd like to see fewer application-level facilities added to the core. I'd rather make it easier to try out different tools and figure out what's best for a particular situation. > ET > belongs in the standard distribution to create a level playing field for a > module many people feel is superior to the current XML-related modules. > Think of it as Pythonic affirmative action. ;-) I would only think of it as Pythonoc affirmative action if you also included the FourSuite stuff and lxml and ... which, of course, would be bad. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 8 updates/clarifications
e.g. class_ rather than klass. >(This last point is a bit controversial; if you prefer klass >over class_ then just be consistent. :). > > With class methods, this has become a more important. Can PEP 8 include > a preferred name for the class argument to classmethods? I personally > prefer cls, there are some who use klass, and I haven't see class_ used. FWIW, as a general rule, I like using a single trailing underscore, especially for keywords. It allows the use of meaningful and easy to remember names. When the name of a variable should be "class" or "for" or whatever, it's easy, as a Python programmer, to remember that I need to add a trailing _. As a reformed abuser of single-character variable names, I've come to really hate abbreviations. It's not only easier to use unabbreviated names, it's easier to remember them when reading code. (Note that ease of use hinges on editors that automate typeing of repeated names.) Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 8 updates/clarifications
[EMAIL PROTECTED] wrote: > Jim> This seems outdated. My impression, in part from time spent > Jim> working with the Python Labs guys, is that it is fine to have > Jim> public data sttributes even for non-"record" types. In fact, I > Jim> would argue that any time you would be tempted to provide > Jim> "getFoo()" and "setFoo(v)" for some "private attribute _foo", it > Jim> would be better to make it public. I certainly find "blah.foo" and > Jim> "blah.foo = v" to be much better than "blah.getFoo()" and > Jim> blah.setFoo(v)". > > Presuming the foo attribute provides some element of the API that you are > willing to support forever. If it is just an implementation detail you > should use accessor methods or properties. If foo is an implementation detail, then it shoudln't be exposed at all, even with accessors. Using attribute syntax makes no more of a commitment than accessor functions. The decision about wither to implement foo as a key in the instance dictionary *is* an implementation detail that can be hidden by a property. If the initial decision, following the rule of "do the simplest thing that works", is to use an instance dictionary item, then I wouldn't bother with a property until you change your mind. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 8 updates/clarifications
[EMAIL PROTECTED] wrote: >>>>>>"Jim" == Jim Fulton <[EMAIL PROTECTED]> writes: > > > Jim> The decision about wither to implement foo as a key in the instance > Jim> dictionary *is* an implementation detail that can be hidden by a > Jim> property. > > If it's not in the instance dictionary, where is it? It could be in a slot. It could be in the instance dictionary under another name. It could be in a subobject. It could be computed from other variables... (in a box, with a fox :) Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Deprecate __ private (was Re: PEP 8 updates/clarifications)
Ian Bicking wrote: > Jim Fulton wrote: > ... >>>Also decide whether your attributes should be private or not. >>>The difference between private and non-public is that the former >>>will never be useful for a derived class, while the latter might >>>be. Yes, you should design your classes with inheritence in >>>mind! >>> >>>Private attributes should have two leading underscores, no >>>trailing underscores. >>> >>> This conflicts with a previous suggestion "Generally, double leading >>> underscores should be used only to avoid name conflicts with >>> attributes in classes designed to be subclassed." Or perhaps >>> "private attributes" needs to be better explained. >> >> >> >> While, on some level, private variables seem attractive, I think that >> experience (for everyone I know) has shown them to be an attractive >> nuisance. I recommend discouraging them. > > > I really really hate double underscores, but I thought I'd let some > other people suggest stronger language first. I prefer explicit name > mangling for those cases where people justifiably use double underscores > now, e.g., self._MyPackage_variable instead of self.__variable, which I > think you also suggest below. Since it's all name mangling anyway, at > least explicit is better than implicit, especially when it's something > one could argue *should* look a little ugly. Perhaps all the > non-public/private language should be switched to just "private" (one > underscore) and "hidden from subclasses" (double underscore). I don't > like calling __ private at all, because it's not what people coming from > other languages think of as private. Can we officially mark __private as a mistake. Perhaps: - Strongly discourage it in the style guide - Mark it in the language reference as a deprecated feature - Generate deprecation warnings when it is used? (This might be too much.) Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 8 updates/clarifications
Ian Bicking wrote: > Jim Fulton wrote: > >>> Designing for inheritance >>> >>>Always decide whether a class's methods and instance variables >>>should be public or non-public. In general, never make data >>>variables public unless you're implementing essentially a >>>record. It's almost always preferrable to give a functional >> >> >> >interface to your class instead (and some Python 2.2 >> >developments will make this much nicer). >> > >> > Yes, Python 2.2 developments have made this better. Use of property() >> > should be suggested. >> >> This seems outdated. My impression, in part from time spent >> working with the Python Labs guys, is that it is fine to have public >> data sttributes even for non-"record" types. In fact, I would argue that >> any time you would be tempted to provide "getFoo()" and "setFoo(v)" >> for some "private attribute _foo", it would be better to make it >> public. I certainly find "blah.foo" and "blah.foo = v" to be much >> better than "blah.getFoo()" and blah.setFoo(v)". >> >> Certainly, properties provide a safety belt. I would argue it this >> way: Python APIs can include attributes as well as methods. >> Exposure of an attribute need not constrain the implementation, thanks >> to properties. OTOH, I wouldn't bother with a property unless it's >> needed. > > > So, getting back to the original paragraph, perhaps it could say: > > Decide whether a class's methods and instance variables should be public > or non-public. Non-public methods and variables should start with an > underscore. > > Do not use accessor methods, like ``obj.getFoo()`` and > ``obj.setFoo(v)``, instead just expose a public attribute (``obj.foo``). > If necessary you can use ``property`` to implement the same > functionality that accessor methods would give you. If you do use > properties, getting that property should never have a side effect. > [well... I think that certain side effects like caching and logging are > okay, but I'm not sure how to make that distinction] > > Potentially it could be added that the whole issue can often be avoided > when an object's methods perform actions instead of returning attributes > of the object. It's a long topic; maybe it could even just be a link, > if someone knows of a good discussion along those lines. I'm sure > there's some terminology here that I'm forgetting that describes the > design pattern. There's also a point when the style guide becomes an > API design guide, and I don't know how far it should go in that direction. Perhaps something like: "If you find yourself writing trivial accessor functions like: def getFoo(self): return self._foo def setFoo(self, v): self._foo = v Use attribute accessors instead. In the example above, just store foo in an attribute named "foo". If you need to store foo a different way later, you can use properties. On the other hand, if getting or setting a variable has other application- meaningful effects, then accessor methods might be better, or perhaps it would be best not to expose the attributes at all. " ... >> While, on some level, private variables seem attractive, I think that >> experience (for everyone I know) has shown them to be an attractive >> nuisance. I recommend discouraging them. > > > I really really hate double underscores, Doesn't everyone? :) > but I thought I'd let some > other people suggest stronger language first. I prefer explicit name > mangling for those cases where people justifiably use double underscores > now, e.g., self._MyPackage_variable instead of self.__variable, which I > think you also suggest below. Since it's all name mangling anyway, at > least explicit is better than implicit, especially when it's something > one could argue *should* look a little ugly. Perhaps all the > non-public/private language should be switched to just "private" (one > underscore) and "hidden from subclasses" (double underscore). I don't > like calling __ private at all, because it's not what people coming from > other languages think of as private. I think we should strongly discourage it in the style guide. I think we should go even further, as I pointed out in another post. >> I'll note that, IMO: >> >> - If you have to worry about protecting attributes from subclasses, >> maybe should sho
Re: [Python-Dev] PEP 8 updates/clarifications
[EMAIL PROTECTED] wrote: > Ian> Do not use accessor methods, like ``obj.getFoo()`` and > Ian> ``obj.setFoo(v)``, instead just expose a public attribute > Ian> (``obj.foo``). If necessary you can use ``property`` to implement > Ian> the same functionality that accessor methods would give you. > > Don't properties only work with new-style clsses? If so, this should > probably be noted. Read properties work with old-style classes. Write properties require old-stype classes. I'm always forgetting this for some reason. Yes, it should be noted. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecate __ private (was Re: PEP 8 updates/clarifications)
Tim Peters wrote: > [Neal Norwitz] > ... > That was the only point to `__` name-mangling. People who think it's > trying to, e.g., emulate C++'s `private` gimmick are enjoying a > semi-private fantasy ;-) It works fine for its intended use. In theory, I agree. In practice, I don't agree that it works fine. Inevitably, someone finds a need to access a "private" variable in a subclass. Or even in the original class, you find some need to use something like __getattr__ where the implicit name mangling doesn't come into play and you have to emulate the name mangling. Or perhaps someone wants to examine the value of one of these variables in the debugger. In my experience, almost every time someone uses the __private trick, they or someone else comes to regret it. OTOH, explicit name mangling provides the benefits of implicit name mangling without it's drawbacks. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 8 updates/clarifications
Raymond Hettinger wrote: >>Do not use accessor methods, like ``obj.getFoo()`` and >>``obj.setFoo(v)``, instead just expose a public attribute > > (``obj.foo``). > > This advice is, of course, not appropriate for all users (properties are > not typically in a Python beginner's skill set) Really? In any case, properties are only needed if you change your mind about the implementation. In my experience, they are rarely needed. > or all use cases. I think the advice gave a very narrow case, which was when you were going to write trivial accessors. > It is > closer to one person's view of the One-Right-Way(tm). Opinions on > programming best practices vary widely, evolve over time, and may be > context dependent. I thought I was reflecting more than just my opinion. Also, the original text had just as strong an admonition -- one that, as I mentioned, seem to be out of line with current thinking. ... >>>experience (for everyone I know) has shown them to be an attractive >>>nuisance. I recommend discouraging them. >> >>I really really hate double underscores > > > FWIW, I think we have no business dictating to others how they should > name their variables. This is doubly true for a convention that has a > long history and built-in language support. Even if, experience with a practice has shown it to be highly problematic? > My preference is to leave PEP 8 for the minimum practices necessary for > one programmer to be able to read and maintain another programmer's > code. I'm for making the style guide smaller. I do think it offers too much advice in places. Although I'm not sure we could all agree om what those places are. :) Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 8 updates/clarifications
Gustavo J. A. M. Carneiro wrote: ... > IMHO, if getting a property involves a potentially long computation, > it's better to have an accessor method rather than a property; > xpto.getFoobar() hints right away the programmer that to access that > value some code has to be run, so the programmer is more likely to store > the result in a temp variable for use in the same context, instead of > calling it multiple times. Similar reasoning applites for setter vs > property =. That's why, in my suggested writeup, I suggested that attributes should be used if the accessors are trivial. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 8 updates/clarifications
[EMAIL PROTECTED] wrote: > Jim> That's why, in my suggested writeup, I suggested that attributes > Jim> should be used if the accessors are trivial. > > In my experience it's difficult to find the locations where another module > mucks with your object's state. Using properties or accessor methods > coupling between modules is reduced and you can be more confident that the > only place an object's state is modified directly is in its own code. I don't understand this argument. Any mutating method or property invoked by foreign code changes an object's state. If you provide a property or a pair if accessors that just sets and gets an attribute with a slightly different name, that affords no more protection than if people were setting the attribute directly. If you don't want external code to change an attribute, don't expose it through a public API. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecate __ private (was Re: PEP 8 updates/clarifications)
Guido van Rossum wrote: > On 12/12/05, Jim Fulton <[EMAIL PROTECTED]> wrote: > >>In practice, I don't agree that it works fine. Inevitably, someone >>finds a need to access a "private" variable in a subclass. Or >>even in the original class, you find some need to use something like >>__getattr__ where the implicit name mangling doesn't come into play >>and you have to emulate the name mangling. Or perhaps someone wants >>to examine the value of one of these variables in the debugger. >>In my experience, almost every time someone uses the __private >>trick, they or someone else comes to regret it. >> >>OTOH, explicit name mangling provides the benefits of implicit >>name mangling without it's drawbacks. > > > I half agree. I've seen many classes overuse __private. As I point out above, it's not just a matter of overuse. It is only recognized by the compiler, so it doesn't work with getattr. And of couuse, there's the debugger issue. I've often found cases where, even when I was using it correctly, I had to do manual name mangling myself. Anytime one has to perform weird tricks in Python to work around magic should be a warning sign. > But that's a > separate issue from not having the feature at all; you might as well > argue against private in Java or C++. I'm not arguing against the feature but against it's implementation. My intuition is that the explicit name magling approach is more in keeping with Python's way of doing things. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 8 updates/clarifications
Ian Bicking wrote: > Barry Warsaw wrote: ... >>>This seems too complex to me for PEP 8. >> >> >>Really? ISTR adopting this convention from Guido, but I'm not 100% sure >>about that. After having used it for several years now, I do really >>like this style, but I'm willing to leave the recommendation out of PEP >>8. > > > It seems so exacting to me; Me too. > stdlib, external modules, internal modules > seems like enough ordering to me. If you want to order things more > exactly, sure, but I don't really see the point personally. Since I > can't assume as a reader that imports are ordered in any way I have to > search to be sure of what's there. The grouping help me browse, but I'd > hope that the import list is short enough that I don't need to use > alphabetization to scan for a module. Personally, I don't find the stdlib/external distinction to be useful. Personally, I'd rather just sort aphabetically based on dotted package name. Because packages provide meaningful groupings to begin with, this approach provides the most meaningful groupings to me. (All of my "internal" modules are in packages.) When scanning imports, I don't want to have to think about whether a module is internal or external. I've got enough to think about without that. :) Frankly, I'd be as happy t see the PEP be silent on module ordering. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Import order (was Re: PEP 8 updates/clarifications)
[EMAIL PROTECTED] wrote: ... > This is subjective enough that I would think some rationale explaining this > convention should be given. This is subjective enough that I don't think it should be in the PEP. Sometimes, less is more. JIm -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 8 updates/clarifications
[EMAIL PROTECTED] wrote: > Jim> I don't understand this argument. Any mutating method or property > Jim> invoked by foreign code changes an object's state. > > Sure, but the only place I need to look for direct changes to the object's > state are in the object's own code. > > Jim> If you provide a property or a pair if accessors that just sets and > Jim> gets an attribute with a slightly different name, that affords no > Jim> more protection than if people were setting the attribute directly. > > Sure it does. Suppose I get an exception in my code because some bit of > code somewhere broke my assumptions about the values an attribute could > assume. If that attribute is only set by the object's own code, I can more > easily debug it (stick in a print or an assert in the places where the > attribute changes, etc). If some external bit of code does something like > > self.foo = Foo() > ... > self.foo.attr = None > > then later in Foo's code I have something like > > self.attr.callme() > > The first thing I need to do is figure out who stomped on self.attr. That > can be time-consuming if I don't necessarily know where the stomping > occurred. I just don't buy this argument. For trivial accessors and properties, you can't just look at your code to know where the changes are initiated. For debugging purposes, it's easy to add a property to allow debugging of attribute assignment. > At work we use Python for very rapid development of trading applications. > Today I think we made about a half-dozen micro releases fixing bugs and our > traders tried each one immediately live. Much of the design is carried > around in our heads or consists of a few equations scribbled on sheets of > paper. As you might imagine, it's a very lively environment. Localizing > attribute modifications is a very good thing for us, even if they are simply > one-line set methods. Having to write accessors for all your public methods doesn't seem consistent with rapid development. It increases the ceremony of development and adds lots of meaningless boilerplate that readers of the code have to wade through. Note that they can't just skip over it, because they can't know if you've slipped something meaningful into one of these accessors. > Jim> If you don't want external code to change an attribute, don't > Jim> expose it through a public API. > > I suppose "public" is subject to some interpretation. Just because I don't > prefix an attribute with an underscore doesn't mean I've implicitly declared > it public. I assume that people will familiarize themselves with the > callable methods of an object and only use direct attribute access if I > haven't provided the necessary methods. A better approach is to document the API for your classes and expect people to use that API. Prepending an underscore documents that a variable or method is internal. (Of course, there's still the subclassing API to deal with, if you need one, but that's a separate issue.) Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 8 updates/clarifications
Barry Warsaw wrote: > On Sun, 2005-12-11 at 16:30 -0600, Ian Bicking wrote: > > >>Potentially it could be added that the whole issue can often be avoided >>when an object's methods perform actions instead of returning attributes >>of the object. It's a long topic; maybe it could even just be a link, >>if someone knows of a good discussion along those lines. I'm sure >>there's some terminology here that I'm forgetting that describes the >>design pattern. There's also a point when the style guide becomes an >>API design guide, and I don't know how far it should go in that direction. > > > I'm not exactly sure if this is what you're getting at, but one thing > that bothers me is using data attributes to trigger actions. Maybe this > gets into the "no side-effects" rule for data attributes, but attributes > that cause an object to perform some action should always be explicit > methods. Exactly. That's why I suggested the PEP start with the trivial case, which, BTW is extremely common. Let judgement guide when something is no-longer trivial. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 8 updates/clarifications
Barry Warsaw wrote: > On Sun, 2005-12-11 at 11:20 -0500, Jim Fulton wrote: > > >>This seems outdated. My impression, in part from time spent >>working with the Python Labs guys, is that it is fine to have public >>data sttributes even for non-"record" types. In fact, I would argue that >>any time you would be tempted to provide "getFoo()" and "setFoo(v)" >>for some "private attribute _foo", it would be better to make it >>public. I certainly find "blah.foo" and "blah.foo = v" to be much >>better than "blah.getFoo()" and blah.setFoo(v)". >> >>Certainly, properties provide a safety belt. I would argue it this >>way: Python APIs can include attributes as well as methods. >>Exposure of an attribute need not constrain the implementation, thanks >>to properties. OTOH, I wouldn't bother with a property unless it's needed. > > > Let me know what you think about this language (from my in-progress > update of PEP 8): > > Designing for inheritance > > Always decide whether a class's methods and instance variables > (collectively: "attributes") should be public or non-public. Public > attributes are those that you expect unrelated clients of your class to > use, with your commitment to avoid backward incompatible changes. > Non-public attributes are those that are not intended to be used by > third parties; you make no guarantees that non-pubic attributes won't > change or even be removed. I'd add somewhere: "If in doubt, chose non-public. You can always change your mind later." > We don't use the term "private" here, since no attribute is really > private in Python (without a generally unnecessary amount of work). > However, another category of attribute are those which, while not being > public, are intended for use by subclasses (often called "protected" in > other languages). Some classes are designed to be inherited from, > either to extend or modify aspects of the class's behavior. When > designing such a class, take care to make explicit decisions about which > attributes are public, which are non-public but useful for subclasses, > and > which are truly only to be used by your base class. A useful term might be "subclass API". Decide which non-public attributes are part of the subclass API. > With this in mind, here are the Pythonic guidelines: > > - Public attributes should have no leading underscores. > > - If your public attribute name collides with a reserved keyword, append > a single trailing underscore to your attribute name. This is > preferable to an abbreviation or corrupted spelling. E.g. "class_" > is preferable to "cls" or "klass". > > Note 1: See the argument name recommendation above for class methods. > > [BAW: I'll include this new text in a later followup] > > - For simple public data attributes, it is fine to expose just the > attribute name, without complicated accessor/mutator methods. Keep in > mind that Python provides an easy path to future enhancement, should > you find that a simple data attribute needs to grow functional > behavior. In that case, use properties to hide functional > implementation behind simple data attribute access syntax. > > Note 1: Properties only work on new-style classes. > > Note 2: Try to keep the functional behavior side-effect free, although > side-effects such as caching are generally fine. Personally, I'd actively discourage use of trivial accessors. Simple attribute access is not only "fine", IMO, but it is much better than trivial accessors. This is an important point, IMO, because, in my experience, the vast majority of accessors *are* trivial. > - If your class is intended to be subclassed, and you have attributes > that you do not want subclasses to use, consider naming them with > double leading underscores and no trailing underscores. This invokes > Python's name mangling algorithm, where the name of the class is > mangled into the attribute name. This helps avoid attribute name > collisions should subclasses inadvertently contain attributes with the > same name. > > Note 1: Note that only the simple class name is used in the mangled > name, so if a subclass chooses both the same class name and attribute > name, you can still get name collisions. > &g
Re: [Python-Dev] PEP 8 updates/clarifications
Michael Hoffman wrote: > [Ian Bickling] > >>>stdlib, external modules, internal modules seems like enough >>>ordering to me. > > > [Jim Fulton] > >>Personally, I don't find the stdlib/external distinction to be useful. > > > It's useful because it allows one to quickly see all the prerequisites > need to be installed in one place. Sure, if you only have one module, and if your module doesn't do any dynamic imports, and if the things your importing don't have dependencies, and ... I think it would be simpler to have a formal dependency system. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On moving to new-style classes
[EMAIL PROTECTED] wrote: > Jim> For debugging purposes, it's easy to add a property to allow > Jim> debugging of attribute assignment. > > Assuming you use new-style classes, which I often don't. The property/debug > idea that you and Gustavo have both now mentioned makes them a bit more > attractive. > > Is there a new-style class HOW-TO somewhere? See http://www.python.org/doc/newstyle.html > It would be useful to > summarize the advantages for them. I still have this thought stuck in my > head (from where, I don't know, probably incorrect) that one of the main > reasons for new-style classes was to get rid of __dict__. No, the main benefit is to begin to resolve the class/type dichotomy. Among other benefits, this allows you to subclass types written in C. Of course, there are other benefits, most notably descriptors, which make properties, among other things, possible. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Keep default comparisons - or add a second set?
Jim Jewett wrote: > PEP 3000 now suggests that dropping default comparison has become more > than an idle what-if. > > Unfortunately, one very common use case of comparisons is to get a > canonical order. If the order is sensible, all the better, but that > is not strictly required. One of Python's selling points (especially > compared to Java) is that getting a canonical order "just works", even > if the objects being sorted are not carefully homogenized by hand. > Python itself relies on this when comparing same-length dictionaries. > > There are times when a specific comparison doesn't make sense (date vs > a datetime that it includes), but these are corner cases best handled > by the specific class that understands the specific requirements -- > classes that already have to override the comparison operators anyhow. > > Even the recently posted "get rid of default comparisons" use case is > really just an argument to make the canonical ordering work better. > The problem Jim Fulton describes is that the (current default) > canonical order will change if objects are saved to a database and > then imported to a different session. Removing default comparisons > wouldn't really help much; the errors would (sometimes) show up at > saving instead of (maybe) at loading, but the solution would still be > to handcode a default comparison for every single class individually. I think you need to do a much better job of defining canonical ordering. You've given two properties: - It need not make sense. :) - It must be consistent accross sessions Does this also mean accross different versions of Python? How about different operating systems and hardware? If I create and pickle a BTree with a bunch of object keys and reload that pickle in a different session, with a later version of Python on a different OS and Hardware architecture, will the keys still have the same order? I consider (obviously) this second property to be crucial. Do you have any proposal for how to achieve these properties? Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] timeout options in high-level networking modules
Yesterday, I needed to make a web request in a program (actually a test) that could block indefinately, so I needed to set a socket timeout. Unfortunately, AFAICT none of urllib, urllib2, httplib provide options to set the timeout on the sockets they use. I ended up having to roll my own code to make the request. It would be nice if high-level network modules, like the ones mentioned above, had options to provide a timeout. (For example, urlopen could grow an optional timout argument.) Thoughts? If we think this is a good idea, then someone who has time could start chipping away at it. I'm happy to work on this *if* I can find time. This would make a nice easy sprint project at PyCon too. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] timeout options in high-level networking modules
Steve Holden wrote: > Jim Fulton wrote: > >>Yesterday, I needed to make a web request in a program (actually a test) >>that could block indefinately, so I needed to set a socket timeout. >>Unfortunately, AFAICT none of urllib, urllib2, httplib provide options to set >>the timeout on the sockets they use. I ended up having to roll my own >>code to make the request. >> >>It would be nice if high-level network modules, like the ones mentioned >>above, had options to provide a timeout. (For example, urlopen could >>grow an optional timout argument.) >> >>Thoughts? >> >>If we think this is a good idea, then someone who has time could start >>chipping >>away at it. I'm happy to work on this *if* I can find time. This would make >>a nice easy sprint project at PyCon too. >> > > That's a very good idea. At present the only option one has is to set a > global socket.defaulttimout() or somehow monkey-patch the modules you > want to use, and neither of those options are entirely satisfactory. Dang, I missed that. I could have abused that yesterday. :) > Basically any method that can create a new TCP connection should acquire > an optional timeout=None parameter, right? Yup, except that None shouldn't be the "I didn't pass anything" marker, since None is a valid settimeout parameter. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Names matter.
Tim Peters wrote: > [Georg Brandl -- or so he claimed on January 8] > >>today, when two Python developers here had approached me about the PSF, I >>realized that it is time to correct a mistake which I had made over three >>years >>ago, when I discovered Linux, free software, Usenet etc (I was sixteen at >>that time). >> >>I then assumed a different name, partly to anonymize myself as others had >>advised. >>When I came across Python, I instantly liked it, and since I had interest in >>Open Source development in general, I wanted to try and contribute to the >>development. >> >>And, as a matter of course, I used my different name for that. When I >>realized that >>I liked the community and the development of Python very much, I decided to >>"unveil" myself, but I could never overcome myself -- till now. >> >>I'm really sorry, and I hope you could at least partly understand what caused >>me to act so irrational. > > > It doesn't really matter, Georg: by default, Python compares by > object identity, not name. If _you_ haven't changed, your name > doesn't matter. I realize that folks are, rightly, trying to encourage Georg. I think protection of the identity of a minor on the Internet is understandable and justifiable. In general though, for adults, truthfulness and non-anonymity *do* matter. At least they matter to me, a *lot*. I don't think members of the PSF should be allowed to hide their identity and certainly, it should not be acceptable to contribute to Python under a false name. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
