Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-23 Thread Paul Moore
On 9/21/05, Josiah Carlson <[EMAIL PROTECTED]> wrote: > > Antoine Pitrou <[EMAIL PROTECTED]> wrote: > > > > > > > The best way to make people stop complaining about the GIL and start > > > using > > > process-based multiprogramming is to provide solid, standardized support > > > for process-based m

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-23 Thread Michael Hudson
"Gregory P. Smith" <[EMAIL PROTECTED]> writes: > On Mon, Sep 19, 2005 at 09:12:05PM +0100, Michael Hudson wrote: >> Martin Blais <[EMAIL PROTECTED]> writes: >> > http://www.gotw.ca/publications/concurrency-ddj.htm >> > The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software >> >

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-22 Thread Terry Reedy
"Bill Janssen" <[EMAIL PROTECTED]> wrote in message news:05Sep22.141518pdt."58617"@synergy1.parc.xerox.com... > Sokolov Jura writes: >> It is so simple to write application server in Python. >> It is so difficult to make it scallable in CPython. >> CPython will not be wide popular without real mu

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-22 Thread Bill Janssen
Sokolov Jura writes: > It is so simple to write application server in Python. > It is so difficult to make it scallable in CPython. > CPython will not be wide popular without real multithreading. He's right. Bill ___ Python-Dev mailing list Python-Dev@p

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-22 Thread Phillip J. Eby
At 08:42 PM 9/22/2005 +0200, Fredrik Lundh wrote: >Phillip J. Eby wrote: > >At 10:27 AM 9/22/2005 +0400, Sokolov Yura wrote: > >>It is so simple to write application server in Python. > >>It is so difficult to make it scallable in CPython. > > > > It seems you've never heard of fork(), which works

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-22 Thread Gregory P. Smith
On Mon, Sep 19, 2005 at 09:12:05PM +0100, Michael Hudson wrote: > Martin Blais <[EMAIL PROTECTED]> writes: > > On 9/18/05, Guido van Rossum <[EMAIL PROTECTED]> wrote: > >> On 9/17/05, John J Lee <[EMAIL PROTECTED]> wrote: > >> > I realize that not all algorithms (nor all computational problems) sca

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-22 Thread Fredrik Lundh
Phillip J. Eby wrote: >>It is so simple to write application server in Python. >>It is so difficult to make it scallable in CPython. > > It seems you've never heard of fork(), which works just fine to scale > Python processes on multiprocessor boxes. there's a version of fork hidden somewhere in

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-22 Thread Phillip J. Eby
At 10:27 AM 9/22/2005 +0400, Sokolov Yura wrote: >It is so simple to write application server in Python. >It is so difficult to make it scallable in CPython. It seems you've never heard of fork(), which works just fine to scale Python processes on multiprocessor boxes. I've actually done this, a

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-22 Thread Oleg Broytmann
On Thu, Sep 22, 2005 at 10:27:10AM +0400, Sokolov Yura wrote: > MULTIPROCESSING RULES!!! Everything in programming is about "divide and conquer". Separate and control. Modules. Classes. Namespaces. And now that multithreading with shared memory. That's an evil idea, it causes a lot of troubl

[Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-22 Thread Sokolov Yura
Ok. While windows already prefers threads, while linux-2.6 improves thread support and speed, while process startup expensive on time and memory, while we ought to dublicate our data and/or use obscure shared memory, while it is much simpler to make threaded program with care just about locks th

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-22 Thread Antoine Pitrou
Hi, > On *nix, use a unix domain socket (location in the filesystem which acts > as a listening socket). On Windows, you can use cTypes, pywin32, etc., > to create a global mutex and/or COM interface. Ok, but for a very simple and crude need like mine (the application code using this IPC means

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-21 Thread Josiah Carlson
Bill Janssen <[EMAIL PROTECTED]> wrote: > > > The best way to make people stop complaining about the GIL and start > > using > > process-based multiprogramming is to provide solid, standardized support > > for process-based multiprogramming. > > And the model provided by the thread abstraction

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-21 Thread Bill Janssen
> The best way to make people stop complaining about the GIL and start > using > process-based multiprogramming is to provide solid, standardized support > for process-based multiprogramming. And the model provided by the thread abstraction is a good API for that support... Bill ___

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-21 Thread Phillip J. Eby
At 12:04 PM 9/21/2005 -0700, Guido van Rossum wrote: >Actually Python itself has a hard time keeping multiple interpreters >truly separate. Also of course there are some shared resources >maintained by the operating system: current directory, open file >descriptors, signal settings, child processes

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-21 Thread Ronald Oussoren
On 21-sep-2005, at 21:04, Guido van Rossum wrote: A system like Java's classloader would be helpfull, where the classloader of a class is used to load the classes used by that class. I have no idea if this can be adapted to python at all. A strict coding style seems to work for now. You can

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-21 Thread Thomas Heller
Bob Ippolito <[EMAIL PROTECTED]> writes: > Personally my biggest issue with the way the CPython VM works is that > you can't reliably do multiple interpreters in a single process. If > that worked well, you could start an independent interpreter per > thread and with a per-interpreter GIL y

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-21 Thread Guido van Rossum
(Interestign to see the PyObjC folks disagree. :-) On 9/21/05, Ronald Oussoren <[EMAIL PROTECTED]> wrote: > > On 21-sep-2005, at 0:10, Bob Ippolito wrote: > > > > My use case for this isn't so much about threads, but plug-ins. > > Writing multiple Python-based plug-ins for an application is always

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-21 Thread Ronald Oussoren
On 21-sep-2005, at 0:10, Bob Ippolito wrote: My use case for this isn't so much about threads, but plug-ins. Writing multiple Python-based plug-ins for an application is always a mess, because they share too much (sys.modules, sys.path, etc.). PyObjC would benefit greatly from this feature, bec

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-21 Thread Josiah Carlson
Antoine Pitrou <[EMAIL PROTECTED]> wrote: > > > > The best way to make people stop complaining about the GIL and start > > using > > process-based multiprogramming is to provide solid, standardized support > > for process-based multiprogramming. > > 100% agreed. I needed a portable way to kno

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-21 Thread Jonathan LaCour
> The best way to make people stop complaining about the GIL and start > using process-based multiprogramming is to provide solid, standardized > support for process-based multiprogramming. +100 Huge amounts of effort would have to be invested to remove the GIL for the benefit of threads. Not

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-21 Thread Antoine Pitrou
> The best way to make people stop complaining about the GIL and start > using > process-based multiprogramming is to provide solid, standardized support > for process-based multiprogramming. 100% agreed. I needed a portable way to know if a program was already running (and then to send it a si

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-21 Thread Simon Percivall
On 21 sep 2005, at 12.33, Donovan Baarda wrote: > In the short term there will be various hacks to try and make the > existing plethora of threading applications run better on multiple > processors, but ultimately the overheads of shared memory will force > serious multi-processing to use IPC chann

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-21 Thread Guido van Rossum
On 9/21/05, Donovan Baarda <[EMAIL PROTECTED]> wrote: > The reality is threads were invented as a low overhead way of easily > implementing concurrent applications... ON A SINGLE PROCESSOR. Taking > into account threading's limitations and objectives, Python's GIL is the > best way to support threa

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-21 Thread Donovan Baarda
On Tue, 2005-09-20 at 22:43, Guido van Rossum wrote: > On 9/20/05, John J Lee <[EMAIL PROTECTED]> wrote: [...] > I don't know that any chips are designed with threading in mind. Fast > threading benefits from fast context switches which benefits from > small register sets. I believe the trend is to

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-20 Thread Phillip J. Eby
At 06:10 PM 9/20/2005 -0400, Bob Ippolito wrote: >My use case for this isn't so much about threads, but plug-ins. >Writing multiple Python-based plug-ins for an application is always a >mess, because they share too much (sys.modules, sys.path, etc.). >PyObjC would benefit greatly from this feature,

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-20 Thread Bob Ippolito
On Sep 20, 2005, at 5:43 PM, Guido van Rossum wrote: > On 9/20/05, John J Lee <[EMAIL PROTECTED]> wrote: > >> threading is not the only, nor the best, concurrency model. >> But maybe these chips designed with threading in mind blow that >> argument >> out of the water. I don't know enough to k

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-20 Thread Guido van Rossum
On 9/20/05, John J Lee <[EMAIL PROTECTED]> wrote: > threading is not the only, nor the best, concurrency model. > But maybe these chips designed with threading in mind blow that argument > out of the water. I don't know enough to know whether that's true or > not... I don't know that any chips ar

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-20 Thread John J Lee
On Tue, 20 Sep 2005, Brett Cannon wrote: > On 9/20/05, John J Lee <[EMAIL PROTECTED]> wrote: > > On Mon, 19 Sep 2005, Florian Weimer wrote: > > > > > The real problem is that you can ditch most extension modules. 8-( > > [...] > > > > *Is* that a showstopper for Python 3.0, though? > > Who know

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-20 Thread John J Lee
On Mon, 19 Sep 2005, Tim Lesher wrote: > On 9/19/05, Michael Hudson <[EMAIL PROTECTED]> wrote: > > I was disappointed that that article (hey, it was the only issue of > > ddj I've ever actually bought! :) didn't consider any concurrency > > models other than shared memory threading. > > The probl

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-20 Thread Brett Cannon
On 9/20/05, John J Lee <[EMAIL PROTECTED]> wrote: > On Mon, 19 Sep 2005, Florian Weimer wrote: > > > The real problem is that you can ditch most extension modules. 8-( > [...] > > *Is* that a showstopper for Python 3.0, though? Who knows. I bet Guido doesn't even know how much breakage he is go

Re: [Python-Dev] GIL, Python 3, and MP vs. UP (was Re: Variant of removing GIL.)

2005-09-20 Thread John J Lee
On Tue, 20 Sep 2005, John J Lee wrote: [...] > I don't actively want a GIL-free Python. I was just making some arguments [...] Actually, FWIW, I don't know if I even *passively* want a GIL-free Python, if it encourages threaded code (though I'd like to have that option for my occasional personal

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-20 Thread John J Lee
On Mon, 19 Sep 2005, Florian Weimer wrote: > The real problem is that you can ditch most extension modules. 8-( [...] *Is* that a showstopper for Python 3.0, though? John ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman

Re: [Python-Dev] GIL, Python 3, and MP vs. UP (was Re: Variant of removing GIL.)

2005-09-20 Thread John J Lee
On Sun, 18 Sep 2005, Guido van Rossum wrote: > On 9/17/05, John J Lee <[EMAIL PROTECTED]> wrote: [...snip...] [guido] > If my hunch is right, I expect that instead of writing massively > parallel applications, we will continue to write single-threaded > applications that are tied together at the p

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-19 Thread Barry Warsaw
On Mon, 2005-09-19 at 17:52, Scott David Daniels wrote: > Michael Hudson wrote: > > > How does a copying gc differ much from a non-copying non-refcounted gc > > here? > > One important issue for C coded modules is that addresses may change > when a GC is invoked, so no remembering addresses in yo

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-19 Thread Scott David Daniels
Michael Hudson wrote: > How does a copying gc differ much from a non-copying non-refcounted gc > here? One important issue for C coded modules is that addresses may change when a GC is invoked, so no remembering addresses in your module; you must recalculate before each use. -- Scott David Danie

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-19 Thread Florian Weimer
* Michael Hudson: > Not to my knowledge. I've always thought that it would be pretty > hard. I'd be interested in being proved wrong. The real problem is that you can ditch most extension modules. 8-( It sounds more like a fun project for the Python core, though. >> Copying GC might help to g

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-19 Thread Michael Hudson
Florian Weimer <[EMAIL PROTECTED]> writes: > By the way, has anybody ever tried to create a CPython variant which > uses a (mostly) copying garbage collector (or something else except > reference counting or Boehm GC)? Not to my knowledge. I've always thought that it would be pretty hard. I'd b

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-19 Thread Florian Weimer
* Guido van Rossum: > That assumes a very specific model for how all that MP power is going > to be used. Indeed. > I personally don't think the threaded programming model as found in > Java works all that well; without locks you end up with concurrent > modification errors, with locks you get d

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-19 Thread Florian Weimer
* Martin Blais: > http://www.gotw.ca/publications/concurrency-ddj.htm > The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software > Herb Sutter > March 2005 This piece is fundamentally wrong. We all have been writing concurrent server-side software for eons. I don't know what He

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-19 Thread Tim Lesher
On 9/19/05, Michael Hudson <[EMAIL PROTECTED]> wrote: > I was disappointed that that article (hey, it was the only issue of > ddj I've ever actually bought! :) didn't consider any concurrency > models other than shared memory threading. The problem is that, for all its limitations, shared-memory t

Re: [Python-Dev] GIL, Python 3, and MP vs. UP

2005-09-19 Thread Michael Hudson
Martin Blais <[EMAIL PROTECTED]> writes: > On 9/18/05, Guido van Rossum <[EMAIL PROTECTED]> wrote: >> On 9/17/05, John J Lee <[EMAIL PROTECTED]> wrote: >> > c. Since time is needed to iron out bugs (and perhaps also to reimplememt >> >some pieces of code "from scratch"), very early in the life

Re: [Python-Dev] GIL, Python 3, and MP vs. UP (was Re: Variant of removing GIL.)

2005-09-19 Thread Martin Blais
On 9/18/05, Guido van Rossum <[EMAIL PROTECTED]> wrote: > On 9/17/05, John J Lee <[EMAIL PROTECTED]> wrote: > > c. Since time is needed to iron out bugs (and perhaps also to reimplememt > >some pieces of code "from scratch"), very early in the life of Python 3 > >seems like the least-worst

Re: [Python-Dev] GIL, Python 3, and MP vs. UP (was Re: Variant of removing GIL.)

2005-09-18 Thread Guido van Rossum
On 9/17/05, John J Lee <[EMAIL PROTECTED]> wrote: > Given the points you make, and the facts that both Python 3 and real > problems with continuing to scale down semiconductor chip feature sizes > are on the horizon, it seems that now would be an excellent time to start > work on this, with the goa

[Python-Dev] GIL, Python 3, and MP vs. UP (was Re: Variant of removing GIL.)

2005-09-17 Thread John J Lee
On Fri, 16 Sep 2005, "Martin v. Löwis" wrote: > Sokolov Yura wrote: > > I think I know how to remove GIL Obviously I am an idiot. > > Not an idiot, just lazy :-) Please try to implement your ideas, > and I predict that you will find: > 1. it is a lot of work to implement > 2. it requires chan