On 10/25/2011 09:01 PM, Dag Sverre Seljebotn wrote:
On 10/25/2011 06:58 PM, mark florisson wrote:
On 25 October 2011 12:22, Stefan Behnel<stefan...@behnel.de> wrote:
The problem is not so much the INCREF (which is just an indirect
add), it's
the DECREF, which contains a conditional jump based on an unknown
external
value, that may trigger external code. That can kill several C compiler
optimisations for the surrounding code. (And that would only get
worse by
using a dedicated locking mechanism.)
What you could do is a form of psuedo-garbage-collection where, when the
Cython refcount/acquisition count reaches 0, you enqueue a Python DECREF
until you're holding the GIL anyway. If sticking it into the queue is
unlikely(), and it is transparent to the compiler that it doesn't
dispatch into unknown code.
...then the C compiler optimizations should presumably not be killed.
DS
(And regarding Stefan's comment about Erlang: It's all about available
libraries. A language for concurrent computing running on CPython and
able to use all the libraries available for CPython would be awesome. It
doesn't need to be named Cython -- show me an Erlang port to the CPython
platform and I'd perhaps jump ship.)
Anyway, sorry for the long mail. I agree this is likely not feasible
to implement, although I would like the functionality to be there.
Perhaps I'm trying to solve problems which don't really need to be
solved. Maybe we should just use multiprocessing, or MPI and numpy
with global arrays and pickling. Maybe memoryviews could help out with
that as well.
Nice conclusion. I think prange was a very nice 80%-there-solution
(which is also the way we framed it when starting), but the GIL just
creates to many barriers. Real garbage collection is needed, and CPython
just isn't there.
What I'd like to see personally is:
- A convenient utility to allocate an array in shared memory, so that
when you pickle a view of it and send it to another Python process with
multiprocessing and it unpickles, it gets a slice into to the same
shared memory. People already do this but it's just a lot of jumping
through hoops. A good place would probably be in NumPy.
- Decent message passing using ZeroMQ in Cython code without any Python
overhead, for fine-grained communication in Cython code in Python
processes spawned using multiprocessing. I think this requires some
syntax candy in Cython to feel natural enough, but perhaps it can be put
on a form so that it is not ZeroMQ-specific.
Dag Sverre
_______________________________________________
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
