Re: [Python-Dev] GIL removal question

2011-08-17 Thread Sturla Molden
Den 10.08.2011 13:43, skrev Guido van Rossum: They have a specific plan, based on Software Transactional Memory: http://morepypy.blogspot.com/2011/06/global-interpreter-lock-or-how-to-kill.html Microsoft's experiment to use STM in .NET failed though. And Linux got rid of the BKL without STM.

Re: [Python-Dev] GIL removal question

2011-08-13 Thread Sturla Molden
Den 13.08.2011 17:43, skrev Antoine Pitrou: These days we have PyGILState_Ensure(): http://docs.python.org/dev/c-api/init.html#PyGILState_Ensure With the most recent Cython (0.15) we can just do: with gil: to ensure holding the GIL. And similarly from a thread holding the GIL with no

Re: [Python-Dev] GIL removal question

2011-08-13 Thread Nick Coghlan
On Sun, Aug 14, 2011 at 9:26 AM, Guido van Rossum wrote: >> These days we have PyGILState_Ensure(): >> http://docs.python.org/dev/c-api/init.html#PyGILState_Ensure >> >> and even dedicated documentation: >> http://docs.python.org/dev/c-api/init.html#non-python-created-threads >> >> ;) > > That is

Re: [Python-Dev] GIL removal question

2011-08-13 Thread Guido van Rossum
On Sat, Aug 13, 2011 at 8:43 AM, Antoine Pitrou wrote: > On Sat, 13 Aug 2011 09:08:16 -0400 > Guido van Rossum wrote: >> >> And, though mostly off-topic, the worst problem with C code, calling >> back into Python, and the GIL that I have seen (several times): >> Suppose you are calling some compl

Re: [Python-Dev] GIL removal question

2011-08-13 Thread Antoine Pitrou
On Sat, 13 Aug 2011 09:08:16 -0400 Guido van Rossum wrote: > > And, though mostly off-topic, the worst problem with C code, calling > back into Python, and the GIL that I have seen (several times): > Suppose you are calling some complex C library that creates threads > itself, where those threads

Re: [Python-Dev] GIL removal question

2011-08-13 Thread Guido van Rossum
On Sat, Aug 13, 2011 at 2:12 AM, Stefan Behnel wrote: > Guido van Rossum, 12.08.2011 23:38: >> >> On Fri, Aug 12, 2011 at 12:57 PM, Rene Nejsum wrote: >>> >>> I think I understand the background and need for GIL. Without it Python >>> programs would have been cluttered with lock/synchronized state

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Stefan Behnel
Guido van Rossum, 12.08.2011 23:38: On Fri, Aug 12, 2011 at 12:57 PM, Rene Nejsum wrote: I think I understand the background and need for GIL. Without it Python programs would have been cluttered with lock/synchronized statements and C-extensions would be harder to write. No, sorry, the first

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Greg Ewing
Sturla Molden wrote: With one interpreter per thread, and a malloc that does not let threads share memory pages (one heap per thread), Python could do the same. Wouldn't that be more or less equivalent to running each thread in a separate process? -- Greg _

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Rene Nejsum
Thank you for the clarification, I should have been more precise... On 12/08/2011, at 23.38, Guido van Rossum wrote: > On Fri, Aug 12, 2011 at 12:57 PM, Rene Nejsum wrote: >> I think I understand the background and need for GIL. Without it Python >> programs would have been cluttered with lock/s

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Guido van Rossum
On Fri, Aug 12, 2011 at 12:57 PM, Rene Nejsum wrote: > I think I understand the background and need for GIL. Without it Python > programs would have been cluttered with lock/synchronized statements and > C-extensions would be harder to write. No, sorry, the first half of this is incorrect: with o

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Xavier Morel
On 2011-08-12, at 20:59 , Sturla Molden wrote: > Den 12.08.2011 18:51, skrev Xavier Morel: >> * Erlang uses "erlang processes", which are very cheap preempted *processes* >> (no shared memory). There have always been tens to thousands to millions of >> erlang processes per interpreter source co

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Aaron Westendorf
Even in the Erlang model, the afore-mentioned issues of bus contention put a cap on the number of threads you can run in any given application assuming there's any amount of cross-thread synchronization. I wrote a blog post on this subject with respect to my experience in tuning RabbitMQ on NUMA ar

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Sturla Molden
Den 12.08.2011 18:57, skrev Rene Nejsum: My two danish kroner on GIL issues…. I think I understand the background and need for GIL. Without it Python programs would have been cluttered with lock/synchronized statements and C-extensions would be harder to write. Thanks to Sturla Molden for he'

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Sturla Molden
Den 12.08.2011 18:51, skrev Xavier Morel: * Erlang uses "erlang processes", which are very cheap preempted *processes* (no shared memory). There have always been tens to thousands to millions of erlang processes per interpreter source contention within the interpreter going back to pre-SMP by s

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Rene Nejsum
My two danish kroner on GIL issues…. I think I understand the background and need for GIL. Without it Python programs would have been cluttered with lock/synchronized statements and C-extensions would be harder to write. Thanks to Sturla Molden for he's explanation earlier in this thread. Howe

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Xavier Morel
On 2011-08-11, at 21:11 , Sturla Molden wrote: > > (b) another threading model (e.g. one interpreter per thread, as in Tcl, > Erlang, or .NET app domains). Nitpick: this is not correct re. erlang. While it is correct that it uses "another threading model" (one could even say "no threading model

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Antoine Pitrou
On Fri, 12 Aug 2011 09:32:23 -0500 VanL wrote: > On 8/11/2011 2:11 PM, Sturla Molden wrote: > > > > (b) another threading model (e.g. one interpreter per thread, as in Tcl, > > Erlang, or .NET app domains). > > We are close to this, in that we already have baked-in support for > subinterpreters.

Re: [Python-Dev] GIL removal question

2011-08-12 Thread VanL
On 8/11/2011 2:11 PM, Sturla Molden wrote: (b) another threading model (e.g. one interpreter per thread, as in Tcl, Erlang, or .NET app domains). We are close to this, in that we already have baked-in support for subinterpreters. Out of curiosity, why isn't this being pursued? _

Re: [Python-Dev] GIL removal question

2011-08-11 Thread Sturla Molden
Den 09.08.2011 11:33, skrev Марк Коренберг: Probably I want to re-invent a bicycle. I want developers to say me why we can not remove GIL in that way: 1. Remove GIL completely with all current logick. 2. Add it's own RW-locking to all mutable objects (like list or dict) 3. Add RW-locks to every

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Maciej Fijalkowski
On Wed, Aug 10, 2011 at 7:19 PM, Raymond Hettinger wrote: > > On Aug 10, 2011, at 4:15 AM, Nick Coghlan wrote: > > After implementing the aforementioned step 5, you will find that the > performance of everything, including the threaded code, will be quite a bit > worse.  Frankly, this is probably

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Raymond Hettinger
On Aug 10, 2011, at 4:15 AM, Nick Coghlan wrote: >> After implementing the aforementioned step 5, you will find that the >> performance of everything, including the threaded code, will be quite a bit >> worse. Frankly, this is probably the most significant obstacle to have any >> kind of GIL-

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Eric Snow
On Wed, Aug 10, 2011 at 10:19 AM, Brian Curtin wrote: > On Wed, Aug 10, 2011 at 11:14, Vlad Riscutia wrote: >> >> Removing GIL is interesting work and probably multiple people are willing >> to contribute. Threading and synchronization is a deep topic and it might be >> that if just one person to

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Brian Curtin
On Wed, Aug 10, 2011 at 11:14, Vlad Riscutia wrote: > Removing GIL is interesting work and probably multiple people are willing > to contribute. Threading and synchronization is a deep topic and it might be > that if just one person toys around with removing GIL he might not see > performance imp

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Vlad Riscutia
Removing GIL is interesting work and probably multiple people are willing to contribute. Threading and synchronization is a deep topic and it might be that if just one person toys around with removing GIL he might not see performance improvement (not meaning to offend anyone who tried this, honestl

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Maciej Fijalkowski
On Wed, Aug 10, 2011 at 1:43 PM, Guido van Rossum wrote: > On Wed, Aug 10, 2011 at 7:32 AM, David Beazley wrote: >> >> On Aug 10, 2011, at 6:15 AM, Nick Coghlan wrote: >> >>> On Wed, Aug 10, 2011 at 9:09 PM, David Beazley wrote: You're forgetting step 5. 5. Put fine-grain locks ar

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Guido van Rossum
On Wed, Aug 10, 2011 at 7:32 AM, David Beazley wrote: > > On Aug 10, 2011, at 6:15 AM, Nick Coghlan wrote: > >> On Wed, Aug 10, 2011 at 9:09 PM, David Beazley wrote: >>> You're forgetting step 5. >>> >>> 5. Put fine-grain locks around all reference counting operations (or >>> rewrite all of Pyth

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Nick Coghlan
On Wed, Aug 10, 2011 at 9:32 PM, David Beazley wrote: > On Aug 10, 2011, at 6:15 AM, Nick Coghlan wrote: >> PyPy would actually make a significantly better basis for this kind of >> experimentation, since they *don't* use reference counting for their >> memory management. > > That's an experiment

Re: [Python-Dev] GIL removal question

2011-08-10 Thread David Beazley
On Aug 10, 2011, at 6:15 AM, Nick Coghlan wrote: > On Wed, Aug 10, 2011 at 9:09 PM, David Beazley wrote: >> You're forgetting step 5. >> >> 5. Put fine-grain locks around all reference counting operations (or rewrite >> all of Python's memory management and garbage collection from scratch). >

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Nick Coghlan
On Wed, Aug 10, 2011 at 9:09 PM, David Beazley wrote: > You're forgetting step 5. > > 5. Put fine-grain locks around all reference counting operations (or rewrite > all of Python's memory management and garbage collection from scratch). ... > After implementing the aforementioned step 5, you will

Re: [Python-Dev] GIL removal question

2011-08-10 Thread David Beazley
> > Message: 1 > Date: Tue, 9 Aug 2011 15:31:47 +0600 > From: ? > To: python-dev@python.org > Subject: [Python-Dev] GIL removal question > Message-ID: > > Content-Type: text/plain; charset=UTF-8 > > Probably I want to re-invent a bicycle. I w

Re: [Python-Dev] GIL removal question

2011-08-09 Thread Stefan Behnel
Марк Коренберг, 09.08.2011 11:31: In a summary: Please say clearly why, actually, my variant is not still implemented. This question comes up on the different Python lists every once in a while. In general, if you want something to be implemented in a specific way, feel free to provide the im

[Python-Dev] GIL removal question

2011-08-09 Thread Марк Коренберг
Probably I want to re-invent a bicycle. I want developers to say me why we can not remove GIL in that way: 1. Remove GIL completely with all current logick. 2. Add it's own RW-locking to all mutable objects (like list or dict) 3. Add RW-locks to every context instance 4. use RW-locks when accessin

[Python-Dev] GIL removal question

2011-08-09 Thread Марк Коренберг
Probably I want to re-invent a bicycle. I want developers to say me why we can not remove GIL in that way: 1. Remove GIL completely with all current logick. 2. Add it's own RW-locking to all mutable objects (like list or dict) 3. Add RW-locks to every context instance 4. use RW-locks when accessin