Re: [Python-Dev] STM and python
Hi, On Thu, Dec 1, 2011 at 07:06, Matt Joiner wrote: > I saw this, I believe it just exposes an STM primitive to user code. > It doesn't make use of STM for Python internals. That's correct. > Explicit STM doesn't seem particularly useful for a language that > doesn't expose raw memory in its normal usage. In my opinion, that sentence could not be more wrong. It is true that, as I discuss on the blog post cited a few times in this thread, the first goal I see is to use STM to replace the GIL as an internal way of keeping the state of the interpreter consistent. This could quite possibly be achieved using the new GCC __transaction_atomic keyword, although I see already annoying issues (e.g. the keyword can only protect a _syntactically nested_ piece of code as a transaction). However there is another aspect: user-exposed STM, which I didn't explore much. While it is potentially even more important, it is a language design question, so I'm happy to delegate it to python-dev. In my opinion, explicit STM (like Clojure) is not only *a* way to write multithreaded Python programs, but it seems to be *the only* way that really makes sense in general, for more than small examples and more than examples where other hacks are enough (see http://en.wikipedia.org/wiki/Software_transactional_memory#Composable_operations ). In other words, locks are low-level and should not be used in a high-level language, like direct memory accesses, just because it forces the programmer to think about increasingly complicated situations. And of course there is the background idea that TM might be available in hardware someday. My own guess is that it will occur, and I bet that in 5 to 10 years all new Intel and AMD CPUs will have Hybrid TM. On such hardware, the performance penalty mostly disappears (which is also, I guess, the reasoning behind GCC 4.7, offering a future path to use Hybrid TM). If python-dev people are interested in exploring the language design space in that direction, I would be most happy to look in more detail at GCC 4.7. If we manage to make use of it, then we could get a version of CPython using STM internally with a very minimal patch. If it seems useful we can then turn that patch into #ifdefs into the normal CPython. It would of course be off by default because of the performance hit; still, it would give an optional alternate "CPythonSTM" to play with in order to come up with good user-level abstractions. (This is what I'm already trying to do with PyPy without using GCC 4.7, and it's progressing nicely.) (My existing patch to CPython emulating user-level STM with the GIL is not really satisfying, also for the reason that it cannot emulate some other potentially useful user constructs, like abort_and_retry().) A bientôt, Armin. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Warnings
Am 01.12.2011 07:10, schrieb Raymond Hettinger: > When updating the documentation, please don't go overboard with warnings. > The docs need to be worded affirmatively -- say what a tool does and show how > to > use it correctly. > See http://docs.python.org/documenting/style.html#affirmative-tone > > The docs for the subprocess module currently have SEVEN warning boxes on one > page: > http://docs.python.org/library/subprocess.html#module-subprocess > The implicit message is that our tools are hazardous and should be avoided. > > Please show some restraint and aim for clean looking, high-quality technical > writing without the FUD. > > Look at the SQLite3 docs for an example of good writing. The prevention of > SQL > injection attacks is discussed briefly and effectively without big red boxes > littering the page. Obviously, +1. Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] STM and python
Armin, thanks for weighing in on this. I'm keen to see a CPython making use of STM, maybe I'll give it a try over Christmas break. I'm willing to take the single threaded performance hit, as I have several applications that degrade due to significant contention with the GIL. The other benefits of STM you describe make it a lot more appealing. I actually tried out Haskell recently to make use of many of the advanced features but came crawling back. If anyone else is keen to try this, I'm happy to receive patches for testing and review. On Thu, Dec 1, 2011 at 10:01 PM, Armin Rigo wrote: > Hi, > > On Thu, Dec 1, 2011 at 07:06, Matt Joiner wrote: >> I saw this, I believe it just exposes an STM primitive to user code. >> It doesn't make use of STM for Python internals. > > That's correct. > >> Explicit STM doesn't seem particularly useful for a language that >> doesn't expose raw memory in its normal usage. > > In my opinion, that sentence could not be more wrong. > > It is true that, as I discuss on the blog post cited a few times in > this thread, the first goal I see is to use STM to replace the GIL as > an internal way of keeping the state of the interpreter consistent. > This could quite possibly be achieved using the new GCC > __transaction_atomic keyword, although I see already annoying issues > (e.g. the keyword can only protect a _syntactically nested_ piece of > code as a transaction). > > However there is another aspect: user-exposed STM, which I didn't > explore much. While it is potentially even more important, it is a > language design question, so I'm happy to delegate it to python-dev. > In my opinion, explicit STM (like Clojure) is not only *a* way to > write multithreaded Python programs, but it seems to be *the only* way > that really makes sense in general, for more than small examples and > more than examples where other hacks are enough (see > http://en.wikipedia.org/wiki/Software_transactional_memory#Composable_operations > ). In other words, locks are low-level and should not be used in a > high-level language, like direct memory accesses, just because it > forces the programmer to think about increasingly complicated > situations. > > And of course there is the background idea that TM might be available > in hardware someday. My own guess is that it will occur, and I bet > that in 5 to 10 years all new Intel and AMD CPUs will have Hybrid TM. > On such hardware, the performance penalty mostly disappears (which is > also, I guess, the reasoning behind GCC 4.7, offering a future path to > use Hybrid TM). > > If python-dev people are interested in exploring the language design > space in that direction, I would be most happy to look in more detail > at GCC 4.7. If we manage to make use of it, then we could get a > version of CPython using STM internally with a very minimal patch. If > it seems useful we can then turn that patch into #ifdefs into the > normal CPython. It would of course be off by default because of the > performance hit; still, it would give an optional alternate > "CPythonSTM" to play with in order to come up with good user-level > abstractions. (This is what I'm already trying to do with PyPy > without using GCC 4.7, and it's progressing nicely.) (My existing > patch to CPython emulating user-level STM with the GIL is not really > satisfying, also for the reason that it cannot emulate some other > potentially useful user constructs, like abort_and_retry().) > > > A bientôt, > > Armin. -- ಠ_ಠ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
