[Python-Dev] Active Objects in Python
According to this list's welcome message, I should introduce myself. I'm Bruce Eckel, I write, consult and give seminars about computer programming, I use Python whenever I can get away with it, and I've spoken at Pycon a number of times. There are further URLs in my signature at the bottom. I'm joining this list because I was having a conversation with Guido about concurrency support in Python (in particular, using an actor/active-object approach) and he suggested I bring it over here. Here are the highlights of my most recent reply to him (Guido's remarks are marked by the '>' signs): > Agreed. IMO the pthreads-style solution doesn't work well no matter > *what* the programming model. Exactly. I think the problem may be that low-level threads people are one step behind (much like embedded systems programmers, which is where I began). They may be just now catching up to objects, but as far as concurrency goes their brains still think in terms of threads, so it seems natural to apply thread concepts to objects. But from an OO standpoint, pthread-style thinking is two steps backwards. You effectively throw open the innards of the object that you just spent time decoupling from the rest of your system, and the coupling is now unpredictable. It's the worst of all possible worlds. > I worked on Mobile Agents at CNRI from 1995 till 2000, but they were > very heavy-weight. Yet, I think that any solution that moves objects > between CPUs on a regular basis is going to suffer the same problem > that process switching in general gets you -- the move is so expensive > that it's hard to decide when it's justified. The examples I've seen haven't relied on mobility, so that's a question: how important is mobility really to an agent system? And could it be done in a trivial fashion in Python, by sending source code. And if it was really necessary sometimes, could it be something where the cost of mobility only occurred when moving an object? It's possible that either an argument can be made against mobility and/or some kind of trivial mobility system could be developed that would work when necessary. Possibly a Linda-like put-take system with pickles (off the top of my head). I don't know that much about mobility, but it does seem like mobile agents could be a powerful solution to the problem solved by enterprise messaging systems. > I believe you pointed me to Active Objects before, right? (Years ago, > maybe when you visited us at Zope.) I may have mentioned them in the past. > If multiple active objects can co-exist in the same *process*, but > nevertheless the language implementation prevents them from sharing > data except via channels, and in addition allows dynamic reallocation > of active objects across multiple CPUs, they just might be the ticket. > But it would require a complete implementation to prove it. Yes, defining an class as "active" would: 1) Install a worker thread and concurrent queue in each object of that class. 2) Automatically turn method calls into tasks and enqueue them 3) Prevent any other interaction other than enqueued messages At that point, the only time the GIL might be needed is to lock the queue at the point of putting ant taking objects. But there has been work done on lock-free data structures, which has been incorporated into Java 5 libraries, so it might even be possible to use a lock-free queue to do this and thus eliminate the need for the GIL at all. Since the enqueuing process serializes all requests to active objects, and since each message-task runs to completion before the next one starts, the problem of threads interfering with each other is eliminated. Also, the enqueuing process will always happen, so even if the queue blocks it will be for a very short, deterministic time. So the theory is that Active Object systems cannot deadlock (although I believe they can livelock). So yes, the best way for this to work might be some kind of enforced implementation -- but forcing you to do something is not particularly Pythonic, so I would think that it would be possible to build an active object framework along with some checking tools to warn you if you are breaking the rules. But what's not clear is whether this would require knowledge of the innards of the Python interpreter (which I don't have) or if it could be built using libraries. BTW: I think that Hoare's Communicating Sequential Processes (CSP) basically describes the idea of active objects but without objects. That is, I think active objects is CSP applied to OO. Bruce Eckelhttp://www.BruceEckel.com Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e" Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Active Objects in Python
Oops. I forgot to add that to the list. Yes, in the working example
of Active Objects that I've written in Java J2SE5, when you send a
message to an active object, you get back a Future, which
I suspect would be the same as your Deferred.
Tuesday, September 27, 2005, 7:41:27 PM, Christopher Armstrong wrote:
> On 9/28/05, Greg Ewing <[EMAIL PROTECTED]> wrote:
>> Nick Coghlan wrote:
>>
>> > PEP 342's yield expressions can probably be used to help address that
>> > problem,
>> > though:
>> >
>> >class SomeAO(ActiveObject):
>> > def processSomeMessage(self):
>> >msg = yield
>> ># Do something with the message
>> >next_msg = yield makeSomeBlockingCall(self)
>> ># Do something with the next message
>>
>> I don't see how that helps, since makeSomeBlockingCall()
>> is evaluated (and therefore blocks) *before* the yield
>> happens.
> Sounds like makeSomeBlockingCall is just misnamed (probably depending
> who you ask).
> I wrote a small library recently that wraps Twisted's Deferreds and
> asynchronous Failure objects such that you can do stuff like
> try:
> x = yield remoteObject.getSomething()
> except Foo:
> print "Oh no!"
> This is just a 2.5-ification of defgen, which is at
> twisted.internet.defer.{deferredGenerator,waitForDeferred}. So anyway,
> if your actor messages always return Deferreds, then this works quite
> nicely.
> --
> Twisted | Christopher Armstrong: International Man of Twistery
>Radix|-- http://radix.twistedmatrix.com
> | Release Manager, Twisted Project
> \\\V/// |-- http://twistedmatrix.com
>|o O||
> wvw-+
> _______
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/bruceeckel-python3234%40mailblocks.com
Bruce Eckelhttp://www.BruceEckel.com mailto:[EMAIL PROTECTED]
Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e"
Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel
Subscribe to my newsletter:
http://www.mindview.net/Newsletter
My schedule can be found at:
http://www.mindview.net/Calendar
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Pythonic concurrency
I'd like to restart this discussion; I didn't mean to put forth active objects as "the" solution, only that it seems to be one of the better, more OO solutions that I've seen so far. What I'd really like to figure out is the "pythonic" solution for concurrency. Guido and I got as far as agreeing that it wasn't threads. Here are my own criteria for what such a solution would look like: 1) It works by default, so that novices can use it without falling into the deep well of threading. That is, a program that you write using threading is broken by default, and the tool you have to fix it is "inspection." I want something that allows me to say "this is a task. Go." and have it work without the python programmer having to study and understand several tomes on the subject. 2) Tasks can be automatically distributed among processors, so it solves the problems of (a) making python run faster (b) how to utilize multiprocessor systems. 3) Tasks are cheap enough that I can make thousands of them, to solve modeling problems (in which I also lump games). This is really a solution to a cerain type of program complexity -- if I can just assign a task to each logical modeling unit, it makes such a system much easier to program. 4) Tasks are "self-guarding," so they prevent other tasks from interfering with them. The only way tasks can communicate with each other is through some kind of formal mechanism (something queue-ish, I'd imagine). 5) Deadlock is prevented by default. I suspect livelock could still happen; I don't know if it's possible to eliminate that. 6) It's natural to make an object that is actor-ish. That is, this concurrency approach works intuitively with objects. 7) Complexity should be eliminated as much as possible. If it requires greater limitations on what you can do in exchange for a clear, simple, and safe programming model, that sounds pythonic to me. The way I see it, if we can't easily use tasks without getting into trouble, people won't use them. But if we have a model that allows people to (for example) make easy use of multiple processors, they will use that approach and the (possible) extra overhead that you pay for the simplicity will be absorbed by the extra CPUs. 8) It should not exclude the possibility of mobile tasks/active objects, ideally with something relatively straightforward such as Linda-style tuple spaces. One thing that occurs to me is that a number of items on this wish list may conflict with each other, which may require a different way of thinking about the problem. For example, it may require two approaches: for "ordinary" non-OO tasks, a functional programming approach ala Erlang, in combination with an actor approach for objects. Bruce Eckelhttp://www.BruceEckel.com mailto:[EMAIL PROTECTED] Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e" Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency
This paper looks very interesting and promises some good ideas. It also looks like it will require time and effort to digest. I've only read the first few pages, but one thing that does leap out is at the beginning of section 3, they say: "... a purely-declarative language is a perfect setting for transactional memory." What's not clear to me from this is whether STM will work in a non-declarative language like Python. Thursday, September 29, 2005, 8:12:23 AM, Michael Hudson wrote: > Bruce Eckel <[EMAIL PROTECTED]> writes: >> I'd like to restart this discussion; I didn't mean to put forth active >> objects as "the" solution, only that it seems to be one of the better, >> more OO solutions that I've seen so far. >> >> What I'd really like to figure out is the "pythonic" solution for >> concurrency. Guido and I got as far as agreeing that it wasn't >> threads. >> >> Here are my own criteria for what such a solution would look like: > Just because I've been mentioning it everywhere else since I read it, > have you seen this paper: > http://research.microsoft.com/Users/simonpj/papers/stm/ > ? I don't know how applicable it would be to Python but it's well > worth the time it takes to read. > Cheers, > mwh Bruce Eckelhttp://www.BruceEckel.com mailto:[EMAIL PROTECTED] Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e" Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel Subscribe to my newsletter: http://www.mindview.net/Newsletter My schedule can be found at: http://www.mindview.net/Calendar ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency
> I spent a few weekends studying that paper earlier this year in order to > see if anything could be stolen for Python; my general impression was "not > easily" at the least. One notable feature of the presented concept was > that when code would otherwise block, they *rolled it back* to the last > nonblocking execution point. In a sense, they ran the code backwards, > albeit by undoing its effects. They then suspend execution until there's a > change to at least one of the variables read during the forward execution, > to avoid repeated retries. I haven't spent the weekends on the paper yet (but it looks like that is what it would take), but I had the impression that they were talking about the lock-free techniques such as the ones used in Java 5. Basically, you start a write operation "in the background" without locking the data structure, so reads can continue while the calculation is taking place but before the result is committed. When the result is ready, an atomic "test and write" operation is used to determine whether any other task has modified the value in the meantime, and if not to commit the new value. If another task did modify the value, then the calculation begins anew. That was my take, but I haven't studied everything about STM yet, so I'm probably missing something. The one thing about this paper is that it seems to be an orthogonal perspective to anything about concurrency that *I* have seen before. > Oddly enough, this paper actually demonstrates a situation where having > static type checking is in fact a solution to a non-trivial problem! It > uses static type checking of monads to ensure that you can't touch > untransacted things inside a transaction. Yes, because of some of my diatribes against static checking people get the impression that I think it's just a bad thing. However, I'm really trying to get across the idea that "static type checking as the solution to all problems is a bad idea," and that the cost is often much greater than the benefit. But if there really is a clear payoff then I'm certainly not averse to it. In general, I really *do* like to be told when something has gone wrong -- I think there's a huge benefit in that. But if I can learn about it at runtime rather than compile time, then that is often a reasonable solution. So with concurrency, I would like to know when I do something wrong, but if I am told at runtime that's OK with me as long as I'm told. Bruce Eckelhttp://www.BruceEckel.com mailto:[EMAIL PROTECTED] Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e" Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel Subscribe to my newsletter: http://www.mindview.net/Newsletter My schedule can be found at: http://www.mindview.net/Calendar ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency - cooperative MT
> I was just saying that it can be useful to mix cooperative threading and > preemptive threading in the same app, i.e. have different domains of > cooperative threading which are preemptively scheduled by the OS. That > has nothing to do with the GIL, I think (but I don't know much in Python > internals). This is one of the interesting questions I'd like to answer (but which probably requires some kind of mathematician to prove one way or another): Does a "complete" concurrency solution require both a preemptive task management system and a cooperative one? OS Processes are generally quite limited in number (although I understand that BeOS was designed to create tons of lightweight processes, which changed that particular picture). In Java, you can usually create several hundred threads before it craps out. Simulations and games could easily have thousands or even hundreds of thousands of independent simulation units. If there are limits to the number of concurrency drivers (threads, etc.), then that would suggest that at some point you have to move to a cooperative system. Which would suggest that a "complete" concurrency solution might require both. That wouldn't be my ideal. My ideal would be a single solution that would scale up to large numbers of concurrency units, and that wouldn't require the programmer to remember to explicitly yield control. Whether my ideal is possible is a question I'd like to answer. Bruce Eckelhttp://www.BruceEckel.com mailto:[EMAIL PROTECTED] Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e" Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel Subscribe to my newsletter: http://www.mindview.net/Newsletter My schedule can be found at: http://www.mindview.net/Calendar ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Pythonic concurrency
Jeremy Jones published a blog discussing some of the ideas we've talked about here: http://www.oreillynet.com/pub/wlg/8002 Although I hope our conversation isn't done, as he suggests! At some point when more ideas have been thrown about (and TIJ4 is done) I hope to summarize what we've talked about in an article. Bruce Eckelhttp://www.BruceEckel.com mailto:[EMAIL PROTECTED] Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e" Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel Subscribe to my newsletter: http://www.mindview.net/Newsletter My schedule can be found at: http://www.mindview.net/Calendar ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency
This does look quite fascinating, and I know there's a lot of really
interesting work going on at the BBC now -- looks like some really
pioneering stuff going on with respect to TV show distribution over
the internet, new compression formats, etc.
So yes indeed, this is quite high on my list to research. Looks like
people there have been doing some interesting work.
Right now I'm just trying to cast a net, so that people can put in
ideas, for when the Java book is done and I can spend more time on it.
Thursday, October 6, 2005, 1:54:56 PM, Michael Sparks wrote:
> Hi Bruce,
> On Thursday 06 October 2005 18:12, Bruce Eckel wrote:
>> Although I hope our conversation isn't done, as he suggests!
> ...
>> At some point when more ideas have been thrown about (and TIJ4 is
>> done) I hope to summarize what we've talked about in an article.
> I don't know if you saw my previous post[1] to python-dev on this topic, but
> Kamaelia is specifically aimed at making concurrency simple and easy to use.
> Initially we were focussed on using scheduled generators for co-operative
> CSP-style (but with buffers) concurrency.
>[1] http://tinyurl.com/dfnah, http://tinyurl.com/e4jfq
> We've tested the system so far on 2 relatively inexperienced programmers
> (as well as experienced, but the more interesting group is novices). The one
> who hadn't done much programming at all (a little bit of VB, pre-university)
> actually fared better IMO. This is probably because concurrency became
> part of his standard toolbox of approaches.
> I've placed the slides I've produced for Euro OSCON on Kamaelia here:
>* http://cerenity.org/KamaeliaEuroOSCON2005.pdf
> The corrected URL for the whitepaper based on work now 6 months old (we've
> come quite a way since then!) is here:
>* http://www.bbc.co.uk/rd/pubs/whp/whp113.shtml
> Consider a simple server for sending text (generated by a user typing into the
> server) to multiple clients connecting to a server. This is a naturally
> concurrent problem in various ways (user interaction, splitting, listening
> for connections, serving connections, etc). Why is that interesting to us?
> It's effectively a microcosm of how subtitling works. (I work at the BBC)
> In Kamaelia this looks like this:
> === start ===
> class ConsoleReader(threadedcomponent):
>def run(self):
> while 1:
> line = raw_input(">>> ")
> line = line + "\n"
> self.outqueues["outbox"].put(line)
> Backplane("subtitles").activate()
> pipeline(
> ConsoleReader(),
> publishTo("subtitles"),
> ).activate()
> def subtitles_protocol():
> return subscribeTo("subtitles")
> SimpleServer(subtitles_protocol, 5000).run()
> === end ===
> The ConsoleReader is threaded to allow the use of the naive way of
> reading from the input, whereas the server, backplane (a named splitter
> component in practice), pipelines, publishing, subscribing, splitting,
> etc are all single threaded co-operative concurrency.
> A possible client for this text service might be:
> pipeline(
> TCPClient("subtitles.rd.bbc.co.uk", 5000),
> Ticker(),
> ).run()
> (Though that would be a bit bare, even if it does use pygame :)
> The entire system is based around communicating generators, but we also
> have threads for blocking operations. (Though the entire network subsystem
> is non-blocking)
> What I'd be interested in, is hearing how our system doesn't match with
> the goals of the hypothetical concurrency system you'd like to see (if it
> doesn't). The main reason I'm interested in hearing this, is because the
> goals you listed are ones we want to achieve. If you don't think our system
> matches it (we don't have process migration as yet, so that's one area)
> I'd be interested in hearing what areas you think are deficient.
> However, the way we're beginning to refer to the project is to refer to
> just the component aspect rather than concurrency - for one simple
> reason - we're getting to stage where we can ignore /most/ concurrency
> issues(not all).
> If you have any time for feedback, it'd be appreciated. If you don't I hope
> it's useful food for thought!
> Best Regards,
> Michael
Bruce Eckelhttp://www.BruceEckel.com mailto:[EMAIL PROTECTED]
Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e"
Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel
Subscribe to my newsletter:
http://www.mindview.net/Newsletter
My schedule can be found at:
http://www.mindview.net/Calendar
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Pythonic concurrency
Early in this thread there was a comment to the effect that "if you don't know how to use threads, don't use them," which I pointedly avoided responding to because it seemed to me to simply be inflammatory. But Ian Bicking just posted a weblog entry: http://blog.ianbicking.org/concurrency-and-processes.html where he says "threads aren't as hard as they imply" and "An especially poor argument is one that tells me that I'm currently being beaten with a stick, but apparently don't know it." I always have a problem with this. After many years of studying concurrency on-and-off, I continue to believe that threading is very difficult (indeed, the more I study it, the more difficult I understand it to be). And I admit this. The comments I sometimes get back are to the effect that "threading really isn't that hard." Thus, I am just too dense to get it. It's hard to know how to answer. I've met enough brilliant people to know that it's just possible that the person posting really does easily grok concurrency issues and thus I must seem irreconcilably thick. This may actually be one of those people for whom threading is obvious (and Ian has always seemed like a smart guy, for example). But. I do happen to have contact with a lot of people who are at the forefront of the threading world, and *none* of them (many of whom have written the concurrency libraries for Java 5, for example) ever imply that threading is easy. In fact, they generally go out of their way to say that it's insanely difficult. And Java has taken until version 5 to (apparently) get it right, partly by defining a new memory model in order to accurately describe what goes on with threading issues. This same model is being adapted for the next version of C++. This is not stuff that was already out there, that everyone knew about -- this is new stuff. Also, look at the work that Scott Meyers, Andrei Alexandrescu, et al did on the "Double Checked Locking" idiom, showing that it was broken under threading. That was by no means "trivial and obvious" during all the years that people thought that it worked. My own experience in discussions with folks who think that threading is transparent usually uncovers, after a few appropriate questions, that said person doesn't actually understand the depth of the issues involved. A common story is someone who has written a few programs and convinced themselves that these programs work (the "it works for me" proof of correctness). Thus, concurrency must be easy. I know about this because I have learned the hard way throughout many years, over and over again. Every time I've thought that I understood concurrency, something new has popped up and shown me a whole new aspect of things that I have heretofore missed. Then I start thinking "OK, now I finally understand concurrency." One example: when I was rewriting the threading chapter for the 3rd (previous) edition of Thinking in Java, I decided to get a dual-processor machine so I could really test things. This way, I discovered that the behavior of a program on a single-processor machine could be dramatically different than the same program on a multiprocessor machine. That seems obvious, now, but at the time I thought I was writing pretty reasonable code. In addition, it turns out that some things in Java concurrency were broken (even the people who were creating thread support in the language weren't getting it right) so that threw in extra monkey wrenches. And when you start studying the new memory model, which takes into account instruction reordering and cache coherency issues, you realize that it's mind-numbingly far from trivial. Or maybe not, for those who think it's easy. But my experience is that the people who really do understand concurrency never suggest that it's easy. Bruce Eckelhttp://www.BruceEckel.com mailto:[EMAIL PROTECTED] Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e" Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel Subscribe to my newsletter: http://www.mindview.net/Newsletter My schedule can be found at: http://www.mindview.net/Calendar ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency
> //Theoretically// I suspect that the system /could/ perform as well as > traditional approaches to dealing with concurrent problems single threaded > (and multi-thread/process). I also think it's important to factor in the possibility of multiprocessors. If Kamaelia (for example) has a very safe and straightforward programming model so that more people are easily able to use it, but it has some performance impact over more complex systems, I think the ease of use issue opens up far greater possibilities if you include multiprocessing -- because if you can easily write concurrent programs in Python, then Python could gain a significant advantage over less agile languages when multiprocessors become common. That is, with multiprocessors, it could be way easier to write a program in Python that also runs way faster than the competition. Yes, of course given enough time they might theoretically be able to write a program that is as fast or faster using their threading mechanism, but it would be so hard by comparison that they'll either never get it done or never be sure if it's reliable. That's what I'm looking for. Bruce Eckelhttp://www.BruceEckel.com mailto:[EMAIL PROTECTED] Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e" Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel Subscribe to my newsletter: http://www.mindview.net/Newsletter My schedule can be found at: http://www.mindview.net/Calendar ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxed Threads in Python
> I can (but won't) point out examples for days of bad decisions made for > the sake of speed, or policy that has been ignored for the sake of speed > (some of these overlap and some don't). As long as you've entered premature-optimization land, how about decisions made because it's *assumed* that (A) We must have speed here and (B) This will make it happen. My hope would be that we could find a solution that would by default keep you out of trouble when writing concurrent programs, but provide a back door if you wanted to do something special. If you choose to go in the back door, you have to do it consciously and take responsibility for the outcome. With Java, in contrast, as soon as you step into the world of concurrency (even if you step in by accident, which is not uncommon), lots of rules change. What was an ordinary method call before is now something risky that can cause great damage. Should I make this variable volatile? Is an operation atomic? You have to learn a lot of things all over again. I don't want that for Python. I'd like the move into concurrency to be a gentle slope, not a sudden reality-shift. If a novice decides they want to try game programming with concurrency, I want there to be training wheels on by default, so that their first experience will be a successful one, and they can then start learning more features and ideas incrementally, without trying a feature and suddenly having the whole thing get weird and crash down on their heads and cause them to run screaming away ... I know there have been some technologies that have already been mentioned on this list and I hope that we can continue to experiment with and discuss those and also new ideas until we shake out the fundamental issues and maybe even come up with a list of possible solutions. Bruce Eckelhttp://www.BruceEckel.com mailto:[EMAIL PROTECTED] Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e" Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel Subscribe to my newsletter: http://www.mindview.net/Newsletter My schedule can be found at: http://www.mindview.net/Calendar ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency
> Yes, there's a troublesome meme in the world: "threads are hard". > They aren't, really. You just have to know what you're doing. I would say that the troublesome meme is that "threads are easy." I posted an earlier, rather longish message about this. The gist of which was: "when someone says that threads are easy, I have no idea what they mean by it." Perhaps this means "threads in Python are easier than threads in other languages." But I just finished a 150-page chapter on Concurrency in Java which took many months to write, based on a large chapter on Concurrency in C++ which probably took longer to write. I keep in reasonably good touch with some of the threading experts. I can't get any of them to say that it's easy, even though they really do understand the issues and think about it all the time. *Because* of that, they say that it's hard. So alright, I'll take the bait that you've laid down more than once, now. Perhaps you can go beyond saying that "threads really aren't hard" and explain the aspects of them that seem so easy to you. Perhaps you can give a nice clear explanation of cache coherency and memory barriers in multiprocessor machines? Or explain atomicity, volatility and visibility? Or, even better, maybe you can come up with a better concurrency model, which is what I think most of us are looking for in this discussion. Bruce Eckelhttp://www.BruceEckel.com mailto:[EMAIL PROTECTED] Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e" Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel Subscribe to my newsletter: http://www.mindview.net/Newsletter My schedule can be found at: http://www.mindview.net/Calendar ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency
> Java's condition variables don't (didn't? has this been fixed?) quite > work. The emphasis on portability and the resulting notions of > red/green threading packages at the beginning didn't help either. > Read Allen Holub's book. And Doug Lea's book. I understand much of > this has been addressed with a new package in Java 1.5. Not only are there significant new library components in java.util.concurrent in J2SE5, but perhaps more important is the new memory model that deals with issues that are (especially) revealed in multiprocessor environments. The new memory model represents new work in the computer science field; apparently the original paper is written by Ph.D.s and is a bit too theoretical for the normal person to follow. But the smart threading guys studied this and came up with the new Java memory model -- so that volatile, for example, which didn't work quite right before, does now. This is part of J2SE5, and this work is being incorporated into the upcoming C++0x. Java concurrency is certainly one of the bad examples of language design. Apparently, they grabbed stuff from C++ (mostly the volatile keyword) and combined it with what they new about pthreads, and decided that being able to declare a method as synchronized made the whole thing object-oriented. But you can see how ill-thought-out the design was because in later versions of Java some fundamental methods: stop(), suspend(), resume() and destroy(), were deprecated because ... oops, we didn't really think those out very well. And then finally, with J2SE5, it *appears* that all the kinks have been fixed, but only with some really smart folks like Doug Lea, Brian Goetz, and that gang, working long and hard on all these issues and (we hope) figuring them all out. I think threading *can* be much simpler, and I *want* it to be that way in Python. But that can only happen if the right model is chosen, and that model is not pthreads. People migrate to pthreads if they already understand it and so it might seem "simple" to them because of that. But I think we need something that supports an object-oriented approach to concurrency that doesn't prevent beginners from using it safely. Bruce Eckel ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency
I don't know of anything that exists. There is an upcoming book that may help: Java Concurrency in Practice, by Brian Goetz, Tim Peierls, Joshua Bloch, Joseph Bowbeer, David Holmes, and Doug Lea (Addison-Wesley 2006). I have had assistance from some of the authors, but don't know if it introduces the concepts from the research paper. Estimated publication is February. However, you might get something from Scott Meyer's analysis of the concurrency issues surrounding the double-checked locking algorithm: http://www.aristeia.com/Papers/DDJ_Jul_Aug_2004_revised.pdf Thursday, October 13, 2005, 8:36:21 AM, Michael Hudson wrote: > Bruce Eckel <[EMAIL PROTECTED]> writes: >> Not only are there significant new library components in >> java.util.concurrent in J2SE5, but perhaps more important is the new >> memory model that deals with issues that are (especially) revealed in >> multiprocessor environments. The new memory model represents new work >> in the computer science field; apparently the original paper is >> written by Ph.D.s and is a bit too theoretical for the normal person >> to follow. But the smart threading guys studied this and came up with >> the new Java memory model -- so that volatile, for example, which >> didn't work quite right before, does now. This is part of J2SE5, and >> this work is being incorporated into the upcoming C++0x. > Do you have a link that explains this sort of thing for the layman? > Cheers, > mwh Bruce Eckelhttp://www.BruceEckel.com mailto:[EMAIL PROTECTED] Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e" Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel Subscribe to my newsletter: http://www.mindview.net/Newsletter My schedule can be found at: http://www.mindview.net/Calendar ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Coroutines (PEP 342)
I just finished reading PEP 342, and it appears to follow Hoare's Communicating Sequential Processes (CSP) where a process is a coroutine, and the communicaion is via yield and send(). It seems that if you follow that form (and you don't seem forced to, pythonically), then synchronization is not an issue. What is not clear to me, and is not discussed in the PEP, is whether coroutines can be distributed among multiple processors. If that is or isn't possible I think it should be explained in the PEP, and I'd be interested in know about it here (and ideally why it would or wouldn't work). Thanks. Bruce Eckel ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
