Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Lennart Regebro
On Mon, May 17, 2010 at 15:05, Antoine Pitrou wrote: > On Mon, 17 May 2010 14:47:25 +0200 > Lennart Regebro wrote: >> On Mon, May 17, 2010 at 14:12, Antoine Pitrou wrote: >> > On Mon, 17 May 2010 08:28:08 +0200 >> > Lennart Regebro wrote: >> >> But the scheduler is so simplistic it ends up figh

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Nir Aides
I would like to restart this thread with 2 notes from the lively discussion: a) Issue 7946 (and this discussion?) concerns Python 3.2 b) The GIL problems are not specific to OSX. The old and new GIL misbehave on GNU/Linux and Windows too. [Putting on anti-frying-pan helmet] Nir _

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Martin v. Löwis
> But some of these, like JCC+pylucene, nltk, and vobject, were > developed with idiosyncratic funding resources which no longer exist. > Others, like pyglet, were developed for a completely different > purpose, and I doubt the developers care what I want. So, > realistically, I doubt it will be l

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Antoine Pitrou
Le lundi 17 mai 2010 à 14:36 -0700, Bill Janssen a écrit : > > Could the macro that releases the GIL also release the thread affinity? We release the GIL in a variety of situations which don't necessarily involve heavy computations (such as: waiting for IO or sleeping). Furthermore, having severa

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Bill Janssen
Antoine Pitrou wrote: > On Sun, 16 May 2010 15:13:44 PDT > Bill Janssen wrote: > > > > So the patch to the threading code would presumably, for those OSs where > > the capability exists, try to put all created threads in the same > > affinity set. > > This is not really a good idea. There's so

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Nick Coghlan
Florent Xicluna wrote: > I would like to have the possibility to "./configure > --without-broken-GIL" or "./configure --with-bfs-scheduler" in Python > 2.7 like we "./configure --with-computed-gotos" in 3.2. > > It will let the opportunity for the experienced user (or the > distribution packager)

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Nick Coghlan
Antoine Pitrou wrote: > On Sun, 16 May 2010 15:13:44 PDT > Bill Janssen wrote: >> So the patch to the threading code would presumably, for those OSs where >> the capability exists, try to put all created threads in the same >> affinity set. > > This is not really a good idea. There's some code wh

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Bill Janssen
Terry Reedy wrote: > On 5/17/2010 2:59 PM, Bill Janssen wrote: > > Yes, it would. As soon as I have working 3.x versions of BeautifulSoup, > > PIL, ReportLab, JCC, pylucene, pyglet, nltk, email, epydoc, feedparser, > > dictclient, docutils, hachoir, mutagen, medusa, python-dateutil, and > > vobj

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Nick Coghlan
Martin v. Löwis wrote: > People running into these problems will have a number of choices to > them: switch operating systems (i.e. drop OSX for something that > actually works), switch programming languages (i.e. drop Python for > something that actually works), switch application architectures (i

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Bill Janssen
Martin v. Löwis wrote: > > I'd hate to let a fundamental flaw like this go through simply because > > someone somewhere somewhen set a completely synthetic deadline. > > No, it's not like that. We set the deadline so that we are able to > cancel discussions like this one. It would be possible to

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Martin v. Löwis
> Not fixing this is a big problem. It relegates the only Python which > will be usable by many many people for many years, 2.x, to a toy > language status on modern machines. It will have threads, but the use > of them, either directly or indirectly by modules you may import, may > cause unpredi

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Terry Reedy
On 5/17/2010 2:59 PM, Bill Janssen wrote: Yes, it would. As soon as I have working 3.x versions of BeautifulSoup, PIL, ReportLab, JCC, pylucene, pyglet, nltk, email, epydoc, feedparser, dictclient, docutils, hachoir, mutagen, medusa, python-dateutil, and vobject, I'll let you know. :-) There /

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Bill Janssen
Antoine Pitrou wrote: > On Mon, 17 May 2010 11:15:49 PDT > Bill Janssen wrote: > > > > What I did know was that > > some of our big complicated Python multi-threaded daemons had shown > > puzzling resource hogging when moved from small Macs to large 8-core > > machines with hardware RAID and lo

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Florent Xicluna
2010/5/16 Nir Aides > > *What can be done with it?* > > Here are some options: > 1) Abandon it - no one is interested, yawn. > 2) Take ideas and workarounds from its code and apply to other patches. > 3) Include it in the interpreter as an auxiliary (turn on with a runtime > switch) scheduler. >

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Antoine Pitrou
On Mon, 17 May 2010 11:15:49 PDT Bill Janssen wrote: > > What I did know was that > some of our big complicated Python multi-threaded daemons had shown > puzzling resource hogging when moved from small Macs to large 8-core > machines with hardware RAID and lots of memory. Could you give detailed

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Bill Janssen
Antoine Pitrou wrote: > Well, if instead of gnashing your teeth, you had contributed to the > issue, perhaps a patch would have been committed by now (or perhaps > not, but who knows?). If you stay silent, you cannot assume that > someone else will stand up for *your* opinion (and the fact that n

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Bill Janssen
Bill Janssen wrote: > use it. Whether or not use explicitly know you are using threads > (because some other package may be using them under the covers). Of course, I meant to say, "Whether or not *youse* explicitly know you are using threads (because some other package may be using them under

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Antoine Pitrou
On Mon, 17 May 2010 10:11:03 PDT Bill Janssen wrote: > > I'd hate to let a fundamental flaw like this go through simply because > someone somewhere somewhen set a completely synthetic deadline. [...] > I've read through that issue (several times), and I have to say that I > wind up gnashing my te

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Bill Janssen
Antoine Pitrou wrote: > Well, 2.7rc1 is scheduled in less than three weeks now. IMO any patch > changing fundamental threading properties is a no-no (even the processor > affinity proposal). Unfortunately, our "fundamental threading properties" are broken for multicore machines. And multicore s

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Antoine Pitrou
Le lundi 17 mai 2010 à 09:05 -0700, Bill Janssen a écrit : > Antoine Pitrou wrote: > > > On Sun, 16 May 2010 15:13:44 PDT > > Bill Janssen wrote: > > > > > > So the patch to the threading code would presumably, for those OSs where > > > the capability exists, try to put all created threads in t

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Bill Janssen
Antoine Pitrou wrote: > On Sun, 16 May 2010 15:13:44 PDT > Bill Janssen wrote: > > > > So the patch to the threading code would presumably, for those OSs where > > the capability exists, try to put all created threads in the same > > affinity set. > > This is not really a good idea. Yes, fixi

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Antoine Pitrou
On Mon, 17 May 2010 14:47:25 +0200 Lennart Regebro wrote: > On Mon, May 17, 2010 at 14:12, Antoine Pitrou wrote: > > On Mon, 17 May 2010 08:28:08 +0200 > > Lennart Regebro wrote: > >> But the scheduler is so simplistic it ends up fighting with the > >> OS scheduler, and a large amount of CPU tim

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Lennart Regebro
On Mon, May 17, 2010 at 14:12, Antoine Pitrou wrote: > On Mon, 17 May 2010 08:28:08 +0200 > Lennart Regebro wrote: >> But the scheduler is so simplistic it ends up fighting with the >> OS scheduler, and a large amount of CPU time is used up switching >> instead of executing. > > This is already f

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Antoine Pitrou
On Sun, 16 May 2010 15:13:44 PDT Bill Janssen wrote: > > So the patch to the threading code would presumably, for those OSs where > the capability exists, try to put all created threads in the same > affinity set. This is not really a good idea. There's some code which releases the GIL, precisel

Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)

2010-05-17 Thread Antoine Pitrou
On Mon, 17 May 2010 08:28:08 +0200 Lennart Regebro wrote: > But the scheduler is so simplistic it ends up fighting with the > OS scheduler, and a large amount of CPU time is used up switching > instead of executing. This is already fixed with py3k. Antoine.

Re: [Python-Dev] Possible patch for functools partial - Interested?

2010-05-17 Thread Daniel Urban
On Mon, May 17, 2010 at 09:47, Hrvoje Niksic wrote: > On 05/14/2010 06:39 AM, Daniel Urban wrote: >> >> I've made a new patch, in which the keywords attribute is a read-only >> proxy of the dictionary. > > What about backward compatibility?  This looks like an incompatible change. You're probably

Re: [Python-Dev] Possible patch for functools partial - Interested?

2010-05-17 Thread Hrvoje Niksic
On 05/14/2010 06:39 AM, Daniel Urban wrote: I've made a new patch, in which the keywords attribute is a read-only proxy of the dictionary. What about backward compatibility? This looks like an incompatible change. ___ Python-Dev mailing list Python-D