Re: [Python-Dev] Add an optional timeout to lock operations

[Nick Coghlan]

Antoine Pitrou wrote:
> Guido van Rossum  python.org> writes:
>> Will locks be interruptible with ^C? That is an oft-requested feature
>> which also wasn't supported at that time; what's the situation
>> nowadays?
> 
> They still aren't interruptible. From what I can read it may be possible to 
> make
> them interruptible in the POSIX semaphore-based implementation, not in the 
> POSIX
> condition variable-based implementation (which is used as a fallback when 
> POSIX
> semaphores are not available, but I don't know whether this fallback is still
> useful).

I'm pretty sure at least some variants of *BSD still don't have OS level
semaphores - their lack is the reason multiprocessing doesn't
necessarily work everywhere that the threading module works (since mp
needs semaphores in order to work its magic).

Jesse would probably know the gory details.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add an optional timeout to lock operations

[Jesse Noller]

On Nov 18, 2009, at 5:38 AM, Nick Coghlan  wrote:


Antoine Pitrou wrote:

Guido van Rossum  python.org> writes:
Will locks be interruptible with ^C? That is an oft-requested  
feature

which also wasn't supported at that time; what's the situation
nowadays?


They still aren't interruptible. From what I can read it may be  
possible to make
them interruptible in the POSIX semaphore-based implementation, not  
in the POSIX
condition variable-based implementation (which is used as a  
fallback when POSIX
semaphores are not available, but I don't know whether this  
fallback is still

useful).


I'm pretty sure at least some variants of *BSD still don't have OS  
level

semaphores - their lack is the reason multiprocessing doesn't
necessarily work everywhere that the threading module works (since mp
needs semaphores in order to work its magic).

Jesse would probably know the gory details.

Cheers,
Nick.



Nick is right, many of the BSDs and FreeBSD up until fairly recently  
did not have named shared semaphore support. Still yet, the behavior  
is broken on some OSes such as OS X which you have to work around.


I wouldn't recommend using them for threading right now, there's an  
assumption that threading "just works" unlike multiprocessing, which  
people understand has caveats.


Jesse
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add an optional timeout to lock operations

[Antoine Pitrou]

Jesse Noller  gmail.com> writes:
> 
> Nick is right, many of the BSDs and FreeBSD up until fairly recently  
> did not have named shared semaphore support. Still yet, the behavior  
> is broken on some OSes such as OS X which you have to work around.

The core locking support only uses anonymous semaphores (they don't need to be
shared accross processes obviously).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add an optional timeout to lock operations

[Jesse Noller]

On Wed, Nov 18, 2009 at 8:50 AM, Antoine Pitrou  wrote:
> Jesse Noller  gmail.com> writes:
>>
>> Nick is right, many of the BSDs and FreeBSD up until fairly recently
>> did not have named shared semaphore support. Still yet, the behavior
>> is broken on some OSes such as OS X which you have to work around.
>
> The core locking support only uses anonymous semaphores (they don't need to be
> shared accross processes obviously).
>
> Regards
>
> Antoine.

That's what I get for replying from an iphone over morning coffee. See
also http://semanchuk.com/philip/sysv_ipc/ for Philip Semanchuk
semaphore/IPC work as well.

Given both multiprocessing and Philip's work are concerned about the
shared aspects, I think you're right - FBSD and others support the
posix semaphores for non-ipc stuff ok AFAIK (recent benchmark
http://www.ioremap.net/node/216).

jesse
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Static analysis of CPython using coccinelle/spatch

[David Malcolm]

On Tue, 2009-11-17 at 13:03 -0800, Brett Cannon wrote:
> On Mon, Nov 16, 2009 at 12:27, David Malcolm  wrote:
> > Has anyone else looked at using Coccinelle/spatch[1] on CPython source
> > code?
[snip]

> Running the tool over the code base and reporting the found bugs would
> be appreciated.

Discounting the false positives, the only issue it finds in python
itself (trunk) is the deliberate mistake in Modules/xxmodule.c

I also ran it on a random sample of extension modules and found some
real bugs (only reported downstream so far, within Fedora's bug
tracker):
- DBus python bindings assume in one place that "unsigned long" is
32 bits wide: https://bugzilla.redhat.com/show_bug.cgi?id=538225
- MySQL-python assumes in one place that sizeof(int) ==
sizeof(long):
https://bugzilla.redhat.com/show_bug.cgi?id=538234
- rpm.ps.append() uses unrecognized 'N' format specifier:
https://bugzilla.redhat.com/show_bug.cgi?id=538218


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] 2to3 for .c code (was Static analysis of CPython using coccinelle/spatch)

[David Malcolm]

On Tue, 2009-11-17 at 19:45 -0500, Terry Reedy wrote:
> A.M. Kuchling wrote:
> > On Mon, Nov 16, 2009 at 03:27:53PM -0500, David Malcolm wrote:
> >> Has anyone else looked at using Coccinelle/spatch[1] on CPython source
> >> code?
> > 
> > For an excellent explanation of Coccinelle, see
> > .
> 
> For those who have not looked, Coccinelle means ladybug (a bug-eating 
> bug ;-) in French. Its principle use to to take C code and a SmPl file 
> of high-level patch descriptions (fixers, in 2to3 talk) and produce a 
> standard diff file. I wonder if this could be used to help people 
> migrate C extensions to 3.1, by developing a SmPl file with the needed 
> changes dictated by API changes. This is similar to its motivating 
> application to Linux. From
> 
> http://coccinelle.lip6.fr/
> 
> "Coccinelle is a program matching and transformation engine which 
> provides the language SmPL (Semantic Patch Language) for specifying 
> desired matches and transformations in C code. Coccinelle was initially 
> targeted towards performing collateral evolutions in Linux. Such 
> evolutions comprise the changes that are needed in client code in 
> response to evolutions in library APIs, and may include modifications 
> such as renaming a function, adding a function argument whose value is 
> somehow context-dependent, and reorganizing a data structure. "
> 
> As I understand it, the problem with C extensions and 3.1 is the current 
> lack of a "collateral evolution" tool like 2to3 for Python code.
Indeed; I think it may be possible to use Coccinelle for this.  

Here's a .cocci semantic patch to convert non-PyObject* dereferences of
an "ob_type" field to use Py_TYPE macro instead. 

@@
PyObject *py_obj_ptr;
type T; 
T non_py_obj_ptr;
@@
(
  py_obj_ptr->ob_type
|
- non_py_obj_ptr->ob_type
+ Py_TYPE(non_py_obj_ptr) 
)


I was able to use this to generate the attached patch for the DBus
python bindings.  Note that it leaves dereferences of a PyObject*
untouched, and works inside sub-expressions.  (There's some noise at the
typedef of Server; I don't know why).

Hope this is helpful
Dave
diff --git a/_dbus_bindings/conn.c b/_dbus_bindings/conn.c
index c30f167..157218d 100644
--- a/_dbus_bindings/conn.c
+++ b/_dbus_bindings/conn.c
@@ -389,7 +389,7 @@ static void Connection_tp_dealloc(Connection *self)
 
 DBG("Connection at %p: freeing self", self);
 PyErr_Restore(et, ev, etb);
-(self->ob_type->tp_free)((PyObject *)self);
+(Py_TYPE(self)->tp_free)((PyObject *)self);
 }
 
 /* Connection type object === */
diff --git a/_dbus_bindings/libdbusconn.c b/_dbus_bindings/libdbusconn.c
index 9bd8def..c33bb7c 100644
--- a/_dbus_bindings/libdbusconn.c
+++ b/_dbus_bindings/libdbusconn.c
@@ -73,7 +73,7 @@ DBusPyLibDBusConnection_tp_dealloc(Connection *self)
 }
 
 PyErr_Restore(et, ev, etb);
-(self->ob_type->tp_free)((PyObject *) self);
+(Py_TYPE(self)->tp_free)((PyObject *) self);
 }
 
 PyTypeObject DBusPyLibDBusConnection_Type = {
diff --git a/_dbus_bindings/message.c b/_dbus_bindings/message.c
index a2c04c3..ee0cbd6 100644
--- a/_dbus_bindings/message.c
+++ b/_dbus_bindings/message.c
@@ -53,7 +53,7 @@ static void Message_tp_dealloc(Message *self)
 if (self->msg) {
 dbus_message_unref(self->msg);
 }
-self->ob_type->tp_free((PyObject *)self);
+Py_TYPE(self)->tp_free((PyObject *)self);
 }
 
 static PyObject *
diff --git a/_dbus_bindings/server.c b/_dbus_bindings/server.c
index 7fc4f70..ba39f74 100644
--- a/_dbus_bindings/server.c
+++ b/_dbus_bindings/server.c
@@ -40,7 +40,7 @@ typedef struct {
 PyObject *weaklist;
 
 PyObject *mainloop;
-} Server;
+}Server; 
 
 PyDoc_STRVAR(Server_tp_doc,
 "A D-Bus server.\n"
@@ -429,7 +429,7 @@ static void Server_tp_dealloc(Server *self)
 
 DBG("Server at %p: freeing self", self);
 PyErr_Restore(et, ev, etb);
-(self->ob_type->tp_free)((PyObject *)self);
+(Py_TYPE(self)->tp_free)((PyObject *)self);
 }
 
 PyDoc_STRVAR(Server_disconnect__doc__,
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add an optional timeout to lock operations

[David Bolen]

Antoine Pitrou  writes:

> I've submitted a patch (*) to add an optional timeout to locking
> operations (Lock.acquire() etc.). Since it's a pretty basic
> functionality, I would like to know if there was any good reason for
> not doing it.

I always assumed it was because as a least-common-denominator set of
functionality, some platforms didn't have the necessary support.

Providing the discussion on this ends up with the an implementation
being accepted, I'd absolutely love to see this then leveraged by
threading.Condition.wait() rather rather than the current
poll-with-timed-sleep approach.

The overhead (CPU, but most notably latency) of that approach (which
also directly impacts Queues) has historically been my top reason in
various applications on Windows to have to implement my own Queue or
Condition structures using native Windows calls.

-- David

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add an optional timeout to lock operations

[Antoine Pitrou]

Hello,

> > I've submitted a patch (*) to add an optional timeout to locking
> > operations (Lock.acquire() etc.). Since it's a pretty basic
> > functionality, I would like to know if there was any good reason for
> > not doing it.
> 
> I always assumed it was because as a least-common-denominator set of
> functionality, some platforms didn't have the necessary support.

Guido's answer says so indeed. Now py3k only needs to support POSIX and Windows
(and, provided Andrew MacIntyre maintains the port, OS/2), which both have
standard support for waiting-with-timeout.

> Providing the discussion on this ends up with the an implementation
> being accepted, I'd absolutely love to see this then leveraged by
> threading.Condition.wait() rather rather than the current
> poll-with-timed-sleep approach.

Agreed. The current patch (as proposed on http://bugs.python.org/issue7316)
includes exactly that.
Poll-with-timed-sleep is especially sub-optimal on laptops where short but
frequent wakeups can cause a significant decrease in battery life.
(the Linux community has been chasing this using powertop:
http://www.lesswatts.org/projects/powertop/)

Feel free to test or review if you're interested.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com