[Python-Dev] Undocumented view==NULL argument in PyObject_GetBuffer()
Hello, PyObject_GetBuffer() had an undocumented variant that was used internally: PyObject_GetBuffer(obj, NULL, flags) view==NULL has never been allowed by either PEP-3118 or the documentation: PEP: "The first variable is the "exporting" object. The second argument is the address to a bufferinfo structure. Both arguments must never be NULL." 3.2 docs: "view must point to an existing Py_buffer structure allocated by the caller." The internal use was to bump up the export count of e.g. a bytearray without bothering to have it fill in a complete Py_buffer. The increased export count would then prevent the bytearray from being resized. However, this feature appears to be unused in the source tree. The last traces of a middle NULL argument that I found are here: http://hg.python.org/cpython/file/df3b2b5db900/Modules/posixmodule.c#l561 So, currently the checks for NULL just slow down bytearray_getbuffer() and a couple of other getbufferprocs. Also, due to the absence of a use case it takes some VCS history digging to find out why the feature was there in the first place. The obvious question is: Will anyone need view==NULL in the future or can we remove the special case? Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Undocumented view==NULL argument in PyObject_GetBuffer()
On Tue, Mar 6, 2012 at 8:34 PM, Stefan Krah wrote: > The obvious question is: Will anyone need view==NULL in the future or > can we remove the special case? The public API will still need a guard (to report an error), but +1 for otherwise eliminating the undocumented special case. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Undocumented view==NULL argument in PyObject_GetBuffer()
Nick Coghlan, 06.03.2012 12:19: > On Tue, Mar 6, 2012 at 8:34 PM, Stefan Krah wrote: >> The obvious question is: Will anyone need view==NULL in the future or >> can we remove the special case? > > The public API will still need a guard (to report an error), but +1 > for otherwise eliminating the undocumented special case. +1 for removing it internally and only checking at the API level. I think it's just a left-over from the "old times" (pre 3.0) when the buffer protocol had an explicit option to lock a buffer. Back then, the code used to call into getbuffer() with a NULL pointer in order to acquire the (IIRC write-) lock. It took me some discussion back then to get this part of the protocol removed, but it's dead for good now. Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Windows uninstallation problem
I've built an MSI with 3.3 on Windows 7 and installed it - it seems to work OK in that it passes all tests except test_tcl (intermittent failure). However, when I uninstall, python33.dll is left behind in System32. If I rebuild the MSI after some changes and reinstall, the old python33.dll is not overwritten. Before raising an issue I thought I'd check here, in case it's something to do with my configuration. Can anyone else confirm my findings? Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Undocumented view==NULL argument in PyObject_GetBuffer()
Nick Coghlan wrote: > On Tue, Mar 6, 2012 at 8:34 PM, Stefan Krah wrote: > > The obvious question is: Will anyone need view==NULL in the future or > > can we remove the special case? > > The public API will still need a guard (to report an error), but +1 > for otherwise eliminating the undocumented special case. I'm looking at other getbufferprocs apart from bytearray_getbuffer() and the public API seems pretty dangerous to me: For example, bytes_buffer_getbuffer() just calls PyBuffer_FillInfo(), which instantly returns 0 (success). Now the reference count to the bytes object is *not* incremented, so it might disappear while the consumer still thinks it's valid. The same happens in _ctypes.c:PyCData_NewGetBuffer(). For array_buffer_getbuf() it looks different: The export count is increased, but not the reference count. So while the array is protected against resizing, it's not immediately obvious to me if it's protected against being deallocated (but I just skimmed the code). It seems to me that bytearray was the only place where the view==NULL scheme obviously worked. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP-393/PEP-3118: unicode format specifiers
Hello, In the array module the 'u' specifier previously meant "2-bytes, on wide builds 4-bytes". Currently in 3.3 the 'u' specifier is mapped to UCS4. I think it would be nice for Python3.3 to implement the PEP-3118 suggestion: 'c' -> UCS1 'u' -> UCS2 'w' -> UCS4 Actually we could even add 'a' -> ASCII, then a unicode object could be a buffer provider that gives the correct view according to the maxchar in the buffer. This opens the possibility for strongly typed memoryviews of strings. Not sure if this is useful, just an idea. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP-393/PEP-3118: unicode format specifiers
> In the array module the 'u' specifier previously meant "2-bytes, on wide > builds 4-bytes". Currently in 3.3 the 'u' specifier is mapped to UCS4. > > I think it would be nice for Python3.3 to implement the PEP-3118 > suggestion: > > 'c' -> UCS1 > > 'u' -> UCS2 > > 'w' -> UCS4 A Unicode string is an array of code point. Another approach is to expose such string as an array of uint8/uint16/uint32 integers. I don't know if you expect to get a character / a substring when you read the buffer of a string object. Using Python 3.2, I get: >>> memoryview(b"abc")[0] b'a' ... but using Python 3.3 I get a number :-) >>> memoryview(b'abc')[0] 97 It is no more possible to create a Unicode string containing characters outside U+-U+10 range. You might apply the same restriction in the buffer API for UCS4. It may be inefficient, the check can be done when you convert the buffer to a string. > Actually we could even add 'a' -> ASCII ASCII implies that the values are in the range U+-U+007F (0-127). Same as the UCS4: you may do the check in the buffer API or when the buffer is converted to string. I don't think that it would be useful to add an ASCII buffer type, because when the buffer is converted to string, Python has to recompute the maximum character (to choose between ASCII, UCS1, UCS2 and UCS4). For example, "abc\xe9"[:-1] is ASCII. UCS1 is enough for ASCII strings. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP-393/PEP-3118: unicode format specifiers
Victor Stinner wrote:
> > 'c' -> UCS1
> > 'u' -> UCS2
> > 'w' -> UCS4
>
> A Unicode string is an array of code point. Another approach is to
> expose such string as an array of uint8/uint16/uint32 integers. I
> don't know if you expect to get a character / a substring when you
> read the buffer of a string object. Using Python 3.2, I get:
>
> >>> memoryview(b"abc")[0]
> b'a'
>
> ... but using Python 3.3 I get a number :-)
Yes, that's changed because officially (see struct module) the format
is unsigned bytes, which are integers in struct module syntax:
>>> unsigned_bytes = memoryview(b"abc")
>>> unsigned_bytes.format
'B'
>>> char_array = unsigned_bytes.cast('c')
>>> char_array.format
'c'
>>> char_array[0]
b'a'
Possibly the uint8/uint16/uint32 integer approach that you mention
would make more sense.
Stefan Krah
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 3.3.0 alpha 1
On 05.03.2012 14:27, Ned Batchelder wrote: For a more extensive list of changes in 3.3.0, see http://docs.python.org/3.3/whatsnew/3.3.html The 3.3 whatsnews page doesn't seem to mention PEP 414 or Unicode literals at all. Indeed. Thanks to Nick, this is now fixed. Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] [RELEASED] Python 3.3.0 alpha 1
In http://mail.python.org/pipermail/python-dev/2012-March/117348.html Georg Brandl posted: > Python 3.3 includes a range of improvements of the 3.x series, as well as > easier > porting between 2.x and 3.x. Major new features in the 3.3 release series > are: As much as it is nice to just celebrate improvements, I think readers (particularly on the download page http://www.python.org/download/releases/3.3.0/ ) would be better served if there were an additional point about porting and the hash changes. http://docs.python.org/dev/whatsnew/3.3.html#porting-to-python-3-3 also failed to mention this, and even the changelog didn't seem to warn people about failing tests or tell them how to work around it. Perhaps something like: Hash Randomization (issue 13703) is now on by default. Unfortunately, this does break some tests; it can be temporarily turned off by setting the environment variable PYTHONHASHSEED to "0" before launching python. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 3.3.0 alpha 1
Il 06 marzo 2012 20:43, Jim J. Jewett ha scritto: > > > In http://mail.python.org/pipermail/python-dev/2012-March/117348.html > Georg Brandl posted: > >> Python 3.3 includes a range of improvements of the 3.x series, as well as >> easier >> porting between 2.x and 3.x. Major new features in the 3.3 release series >> are: > > As much as it is nice to just celebrate improvements, I think > readers (particularly on the download page > http://www.python.org/download/releases/3.3.0/ ) would be better > served if there were an additional point about porting and the > hash changes. > > http://docs.python.org/dev/whatsnew/3.3.html#porting-to-python-3-3 > also failed to mention this, and even the changelog didn't seem to > warn people about failing tests or tell them how to work around it. > > Perhaps something like: > > Hash Randomization (issue 13703) is now on by default. Unfortunately, > this does break some tests; it can be temporarily turned off by setting > the environment variable PYTHONHASHSEED to "0" before launching python. > > > -jJ > > -- > > If there are still threading problems with my replies, please > email me with details, so that I can try to resolve them. -jJ That's why I once proposed to include whatsnew.rst changes every time a new feature is added/committed. Assigning that effort to the release manager or whoever is supposed to take care of this, is both impractical and prone to forgetfulness. --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP-393/PEP-3118: unicode format specifiers
> I think it would be nice for Python3.3 to implement the PEP-3118 > suggestion: > > 'c' -> UCS1 > > 'u' -> UCS2 > > 'w' -> UCS4 What is the use case for these format codes? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows uninstallation problem
Am 06.03.2012 15:35, schrieb Vinay Sajip: > I've built an MSI with 3.3 on Windows 7 and installed it - it seems to work OK > in that it passes all tests except test_tcl (intermittent failure). However, > when I uninstall, python33.dll is left behind in System32. If I rebuild the > MSI > after some changes and reinstall, the old python33.dll is not overwritten. > > Before raising an issue I thought I'd check here, in case it's something to do > with my configuration. Can anyone else confirm my findings? It most likely is a misconfiguration of your system. I guess that the registry key for the DLL has a non-zero refcount before you started the installation, so that the refcount didn't drop to zero when you uninstalled. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows uninstallation problem
Martin v. Löwis v.loewis.de> writes: > It most likely is a misconfiguration of your system. I guess that the > registry key for the DLL has a non-zero refcount before you started the > installation, so that the refcount didn't drop to zero when you uninstalled. That must have been it - thanks. I uninstalled, removed the key from the SharedDLLs (it was still there with a refcount of 1, as per your analysis), reinstalled and uninstalled - all is well. I did raise an issue but I'll go and close it now. Thanks for the help. Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP-393/PEP-3118: unicode format specifiers
On Wed, Mar 7, 2012 at 4:15 AM, Stefan Krah wrote:
> Victor Stinner wrote:
>> A Unicode string is an array of code point. Another approach is to
>> expose such string as an array of uint8/uint16/uint32 integers. I
>> don't know if you expect to get a character / a substring when you
>> read the buffer of a string object. Using Python 3.2, I get:
>>
>> >>> memoryview(b"abc")[0]
>> b'a'
>>
>> ... but using Python 3.3 I get a number :-)
>
> Yes, that's changed because officially (see struct module) the format
> is unsigned bytes, which are integers in struct module syntax:
>
unsigned_bytes = memoryview(b"abc")
unsigned_bytes.format
> 'B'
char_array = unsigned_bytes.cast('c')
char_array.format
> 'c'
char_array[0]
> b'a'
To maintain backwards compatibility, we should probably take the
purity hit and officially change the default format of memoryview() to
'c', requiring the explicit cast to 'B' to get the new more bytes-like
behaviour.
Using 'c' as the default format is a little ugly, but not as ugly as
breaking currently working 3.2 code in the upgrade to 3.3.
> Possibly the uint8/uint16/uint32 integer approach that you mention
> would make more sense.
Any changes made in this area should be aimed specifically at making
life easier for developers dealing with ASCII puns in binary
protocols. Being able to ask a string for a memoryview, and receiving
one back with the format set to the appropriate value could
potentially help with that by indicating:
ASCII: each code point is mapped to an integer in the range 0-127
latin-1: each code point is mapped to an integer in the range 0-255
UCS2: each code point is mapped to an integer in the range 0-65535
UCS4: each code point is mapped to an integer in the range 0-0x10
Using the actual code point values rather than bytes representations
which may vary in length can help gloss over the differences in the
underlying data layout. However, use cases should be explored more
thoroughly *first* before any additional changes are made to the
supported formats.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 3.3.0 alpha 1
Jim J. Jewett, 06.03.2012 20:43: > Hash Randomization (issue 13703) is now on by default. Unfortunately, > this does break some tests; it can be temporarily turned off by setting > the environment variable PYTHONHASHSEED to "0" before launching python. I don't think that makes it clear enough that that's just a work-around and that it's the tests that need fixing in the first place. Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 3.3.0 alpha 1
On 07.03.2012 08:08, Stefan Behnel wrote: Jim J. Jewett, 06.03.2012 20:43: Hash Randomization (issue 13703) is now on by default. Unfortunately, this does break some tests; it can be temporarily turned off by setting the environment variable PYTHONHASHSEED to "0" before launching python. I don't think that makes it clear enough that that's just a work-around and that it's the tests that need fixing in the first place. It's much too much for the release announcement anyway. I'll add one bullet point in the future announcements. I've added a note to what's new so that someone will add an appropriate paragraph. Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
