[Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt

2010-01-07 Thread Yoann Padioleau
Hi, I would like to use astgen.py to generate python classes corresponding to the AST of something I have defined in a .asdl file, along the line of what is apparently done for the python AST itself. I thought astgen.py would take as an argument a .asdl file, but apparently it instead process a f

[Python-Dev] Backported faster RLock to Python 2.6.

2010-01-07 Thread Johan Gill
Hi devs, the company where I work has done some work on Python, and the question is how this work, owned by the company, can be contributed to the community properly. Are there any license issues or other pitfalls we need to think about? I imagine that other companies have contributed before,

Re: [Python-Dev] Backported faster RLock to Python 2.6.

2010-01-07 Thread Lennart Regebro
On Thu, Jan 7, 2010 at 10:46, Johan Gill wrote: > Hi devs, > the company where I work has done some work on Python, and the question is > how this work, owned by the company, can be contributed to the community > properly. Are there any license issues or other pitfalls we need to think > about? I

Re: [Python-Dev] Backported faster RLock to Python 2.6.

2010-01-07 Thread Simon Cross
On Thu, Jan 7, 2010 at 1:12 PM, Lennart Regebro wrote: > I'm not a license lawyer, but typically your company needs to give the > code to the community. Yes, it means it stops owning it. This is incorrect. The correct information is at http://www.python.org/psf/contrib/. Schiavo Simon _

[Python-Dev] test_ctypes failure on AIX 5.3 using python-2.6.2 and libffi-3.0.9

2010-01-07 Thread swamy sangamesh
Hi All, I built the python-2.6.2 with the latest libffi-3.0.9 in AIX 5.3 using xlc compiler. When i try to run the ctypes test cases, two failures are seen in test_bitfields. *test_ints (ctypes.test.test_bitfields.C_Test) ... FAIL test_shorts (ctypes.test.test_bitfields.C_Test) ... FAIL* I have

Re: [Python-Dev] Backported faster RLock to Python 2.6.

2010-01-07 Thread Nick Coghlan
Lennart Regebro wrote: > On Thu, Jan 7, 2010 at 10:46, Johan Gill wrote: >> Hi devs, >> the company where I work has done some work on Python, and the question is >> how this work, owned by the company, can be contributed to the community >> properly. Are there any license issues or other pitfalls

Re: [Python-Dev] Backported faster RLock to Python 2.6.

2010-01-07 Thread Lennart Regebro
On Thu, Jan 7, 2010 at 13:23, Nick Coghlan wrote: > As Simon pointed out, while some organisations do work that way, the PSF > isn't one of them. > > The PSF only requires that the code be contributed under a license that > then allows us to turn around and redistribute it under a different open >

Re: [Python-Dev] Backported faster RLock to Python 2.6.

2010-01-07 Thread Michael Foord
On 07/01/2010 13:11, Lennart Regebro wrote: On Thu, Jan 7, 2010 at 13:23, Nick Coghlan wrote: As Simon pointed out, while some organisations do work that way, the PSF isn't one of them. The PSF only requires that the code be contributed under a license that then allows us to turn around an

Re: [Python-Dev] GIL required for _all_ Python calls?

2010-01-07 Thread Stefan Behnel
Guido van Rossum, 07.01.2010 05:29: A better rule would be "you may access the memory buffer in a PyString or PyUnicode object with the GIL released as long as you own a reference to the string object." Everything else is out of bounds (or not worth the bother). Is that a "yes" regarding the OP

Re: [Python-Dev] Backported faster RLock to Python 2.6.

2010-01-07 Thread Lennart Regebro
On Thu, Jan 7, 2010 at 14:15, Michael Foord wrote: > (i.e. copyright and ownership are legal terms that don't necessarily mean > anything *practical* in these situations.) OK, fair enough. :-) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 __

Re: [Python-Dev] GIL required for _all_ Python calls?

2010-01-07 Thread Stefan Behnel
MRAB, 07.01.2010 04:07: I've been wondering whether it's possible to release the GIL in the regex engine during matching. I know that it needs to have the GIL during memory-management calls, but does it for calls like Py_UNICODE_TOLOWER Py_UNICODE_TOLOWER looks safe to me at first glance. o

Re: [Python-Dev] GIL required for _all_ Python calls?

2010-01-07 Thread Antoine Pitrou
MRAB mrabarnett.plus.com> writes: > > I know that it needs to have the GIL during memory-management calls, but > does it for calls like Py_UNICODE_TOLOWER or PyErr_SetString? Is there > an easy way to find out? There is no "easy way" to do so. The only safe way is to examine all the functions or

Re: [Python-Dev] Question over splitting unittest into a package

2010-01-07 Thread Olemis Lang
On Mon, Jan 4, 2010 at 9:24 AM, Olemis Lang wrote: > On Thu, Dec 31, 2009 at 10:30 AM, Martin (gzlist) > wrote: >> Thanks for the quick response. >> >> On 30/12/2009, Benjamin Peterson wrote: >> >> but maybe a >> discussion could start about a new, less hacky, way of doing the same >> > > I am

Re: [Python-Dev] GIL required for _all_ Python calls?

2010-01-07 Thread Martin v. Löwis
>> A better rule would be "you may access the memory buffer in a PyString >> or PyUnicode object with the GIL released as long as you own a >> reference to the string object." Everything else is out of bounds (or >> not worth the bother). > > Is that a "yes" regarding the OP's original question ab

Re: [Python-Dev] GIL required for _all_ Python calls?

2010-01-07 Thread Martin v. Löwis
> I've been wondering whether it's possible to release the GIL in the > regex engine during matching. I don't think that's possible. The regex engine can also operate on objects whose representation may move in memory when you don't hold the GIL (e.g. buffers that get mutated). Even if they stay i

Re: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt

2010-01-07 Thread Martin v. Löwis
> I would like to use astgen.py to generate python classes corresponding to the > AST of something I have defined in a .asdl file, along the line of what is > apparently done for the python AST itself. I thought astgen.py would > take as an argument a .asdl file, but apparently it instead process

Re: [Python-Dev] GIL required for _all_ Python calls?

2010-01-07 Thread James Y Knight
On Jan 7, 2010, at 3:27 PM, Martin v. Löwis wrote: I've been wondering whether it's possible to release the GIL in the regex engine during matching. I don't think that's possible. The regex engine can also operate on objects whose representation may move in memory when you don't hold the GIL (

Re: [Python-Dev] GIL required for _all_ Python calls?

2010-01-07 Thread Martin v. Löwis
>>> I've been wondering whether it's possible to release the GIL in the >>> regex engine during matching. >> >> I don't think that's possible. The regex engine can also operate on >> objects whose representation may move in memory when you don't hold >> the GIL (e.g. buffers that get mutated). Even

Re: [Python-Dev] Backported faster RLock to Python 2.6.

2010-01-07 Thread Nick Coghlan
Johan Gill wrote: > Yes, it is the new RLock implementation. > If I understood this correctly, we should make a patch against trunk if > anything should be contributed. Yep. > Do you mean that we wouldn't need the paperwork for backporting the > original patch committed to py3k? Whether or not a

Re: [Python-Dev] GIL required for _all_ Python calls?

2010-01-07 Thread Antoine Pitrou
Martin v. Löwis v.loewis.de> writes: > > I don't think that's possible. The regex engine can also operate on > objects whose representation may move in memory when you don't hold > the GIL (e.g. buffers that get mutated). Why is it a problem? If we get a buffer through the new buffer API, the ob

Re: [Python-Dev] GIL required for _all_ Python calls?

2010-01-07 Thread Martin v. Löwis
> I've been wondering whether it's possible to release the GIL in the > regex engine during matching. Ok, here is another problem: SRE_OP_REPEAT uses PyObject_MALLOC, which requires the GIL (it then also may call PyErr_NoMemory, which also requires the GIL). Regards, Martin __

Re: [Python-Dev] Backported faster RLock to Python 2.6.

2010-01-07 Thread Johan Gill
On 01/07/2010 01:23 PM, Nick Coghlan wrote: As Simon pointed out, while some organisations do work that way, the PSF isn't one of them. The PSF only requires that the code be contributed under a license that then allows us to turn around and redistribute it under a different open source license

Re: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt

2010-01-07 Thread Yoann Padioleau
On Jan 7, 2010, at 12:31 PM, Martin v. Löwis wrote: >> I would like to use astgen.py to generate python classes corresponding to >> the >> AST of something I have defined in a .asdl file, along the line of what is >> apparently done for the python AST itself. I thought astgen.py would >> take a

Re: [Python-Dev] GIL required for _all_ Python calls?

2010-01-07 Thread Martin v. Löwis
>> I don't think that's possible. The regex engine can also operate on >> objects whose representation may move in memory when you don't hold >> the GIL (e.g. buffers that get mutated). > > Why is it a problem? If we get a buffer through the new buffer API, the object > should ensure that the repr

Re: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt

2010-01-07 Thread Martin v. Löwis
>> astgen.py is not used to process asdl files; ast.txt lives right >> next to astgen.py. Instead, the asdl file is processed by >> Parser/asdl_c.py. > > Yes, I know that. That's why I asked about the relation between > ast.txt and Python.adsl. If internally the parser uses the .adsl, but > expose

[Python-Dev] Improve open() to support reading file starting with an unicode BOM

2010-01-07 Thread Victor Stinner
Hi, Builtin open() function is unable to open an UTF-16/32 file starting with a BOM if the encoding is not specified (raise an unicode error). For an UTF-8 file starting with a BOM, read()/readline() returns also the BOM whereas the BOM should be "ignored". See recent issues related to reading

Re: [Python-Dev] Improve open() to support reading file starting with an unicode BOM

2010-01-07 Thread Guido van Rossum
I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy talk. And for the other two, perhaps it would make more sense to have a separate encoding-guessing function that takes a binary stream and returns a text stream wrapping it with the proper encoding? --Guido On Thu, Jan 7, 2010 a

Re: [Python-Dev] Improve open() to support reading file starting with an unicode BOM

2010-01-07 Thread MRAB
Guido van Rossum wrote: I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy talk. And for the other two, perhaps it would make more sense to have a separate encoding-guessing function that takes a binary stream and returns a text stream wrapping it with the proper encoding? Alte

Re: [Python-Dev] --enabled-shared broken on freebsd5?

2010-01-07 Thread Nicholas Bastin
I think this problem probably needs to move over to distutils-sig, as it doesn't seem to be specific to the way that Python itself uses distutils. distutils.command.build_ext tests for Py_ENABLE_SHARED on linux and solaris and automatically adds '.' to the library_dirs, and I suspect it just needs

Re: [Python-Dev] Improve open() to support reading file starting with an unicode BOM

2010-01-07 Thread Glyph Lefkowitz
On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: >> Hi, >> >> Builtin open() function is unable to open an UTF-16/32 file starting with a >> BOM if the encoding is not specified (raise an unicode error). For an UTF-8 >> file starting

Re: [Python-Dev] --enabled-shared broken on freebsd5?

2010-01-07 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Nicholas Bastin wrote: > I think this problem probably needs to move over to distutils-sig, as > it doesn't seem to be specific to the way that Python itself uses > distutils. distutils.command.build_ext tests for Py_ENABLE_SHARED on > linux and solar

Re: [Python-Dev] Improve open() to support reading file starting with an unicode BOM

2010-01-07 Thread Guido van Rossum
On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz wrote: > > > On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: > > On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner > wrote: > > Hi, > > Builtin open() function is unable to open an UTF-16/32 file starting with a > > BOM if the encoding is not speci

Re: [Python-Dev] Improve open() to support reading file starting with an unicode BOM

2010-01-07 Thread Stephen J. Turnbull
Guido van Rossum writes: > I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy > talk. That doesn't stop many applications from doing it. Python should perhaps not produce UTF-8 + BOM without a disclaimer of indemnification against all resulting damage, signed in blood, from t

Re: [Python-Dev] Improve open() to support reading file starting with an unicode BOM

2010-01-07 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz > wrote: >> >> On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: >> >> On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner >> wrote: >> >> Hi, >> >> Builtin open() function is un

Re: [Python-Dev] Improve open() to support reading file starting with an unicode BOM

2010-01-07 Thread Glyph Lefkowitz
On Jan 7, 2010, at 11:21 PM, Guido van Rossum wrote: > On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz > wrote: >> >> On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: >>> >>> I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy >>> talk. And for the other two, perhaps it wo