Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467

[INADA Naoki]

I submit an issue about it.

See https://bugs.python.org/issue29178

On Fri, Jan 6, 2017 at 7:38 PM, INADA Naoki  wrote:
>>
>> Thinking more about this, and after looking at my own code in asyncpg and
>> uvloop, I'm now in favor of adding bytes.frombuffer() with the proposed
>> signature: ``bytes.frombuffer(byteslike, length=-1, offset=0)``
>>
>
> Do you prefer a signature proposed first?
> I thought it from asyncio usage too.  Slicing from head of bytearray
> is more common
> than offset usage.
>
> But in previous thread, Serhiy and Victor proposed signature same to
> old buffer API.
>
>> I like the idea with Serhiy's API (as Python 2 buffer constructor):
>>
>> bytes.frombuf(buffer, [offset, size])
>> bytearray.frombuf(buffer, [offset, size])
>> memoryview.frombuf(buffer, [offset, size])
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467

[INADA Naoki]

>
> Thinking more about this, and after looking at my own code in asyncpg and
> uvloop, I'm now in favor of adding bytes.frombuffer() with the proposed
> signature: ``bytes.frombuffer(byteslike, length=-1, offset=0)``
>

Do you prefer a signature proposed first?
I thought it from asyncio usage too.  Slicing from head of bytearray
is more common
than offset usage.

But in previous thread, Serhiy and Victor proposed signature same to
old buffer API.

> I like the idea with Serhiy's API (as Python 2 buffer constructor):
>
> bytes.frombuf(buffer, [offset, size])
> bytearray.frombuf(buffer, [offset, size])
> memoryview.frombuf(buffer, [offset, size])
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467

[Antoine Pitrou]

On Thu, 5 Jan 2017 20:28:26 -0500
Yury Selivanov  wrote:
> On 2017-01-05 7:11 PM, INADA Naoki wrote:
> >> bytes.frombuffer(x) is bytes(memoryview(x)) or memoryview(x).tobytes().
> >>  
> > There is pitfall: memoryview should be closed.
> > So b = bytes.frombuffer(x) is:
> >
> > with memoryview(x) as m:
> >  b = bytes(m)
> >  # or b = m.tobytes()  
> 
> Thinking more about this, and after looking at my own code in asyncpg 
> and uvloop, I'm now in favor of adding bytes.frombuffer() with the 
> proposed signature: ``bytes.frombuffer(byteslike, length=-1, offset=0)``

+1

Note this matches numpy.frombuffer():

"""
frombuffer(buffer, dtype=float, count=-1, offset=0)

Interpret a buffer as a 1-dimensional array.
"""

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Context manager lifetime rules Was: Re: Adding bytes.frombuffer() constructor to PEP 467

[Hans-Peter Jansen]

Hi Yury,

adjusted subject, since I'm dragging the discussion away from it.

On Donnerstag, 5. Januar 2017 20:28:26 Yury Selivanov wrote:
> On 2017-01-05 7:11 PM, INADA Naoki wrote:
> >> bytes.frombuffer(x) is bytes(memoryview(x)) or memoryview(x).tobytes().
> > 
> > There is pitfall: memoryview should be closed.
> > So b = bytes.frombuffer(x) is:
> > 
> > with memoryview(x) as m:
> >  b = bytes(m)
> >  # or b = m.tobytes()
> 
> Thinking more about this, and after looking at my own code in asyncpg
> and uvloop, I'm now in favor of adding bytes.frombuffer() with the
> proposed signature: ``bytes.frombuffer(byteslike, length=-1, offset=0)``
> 
> Inada-san is right, the memoryview should be explicitly released, but
> few do that. Instead, a lot of people simply rely on CPython refcounting
> semantics, which will cause the temporary memoryview be GCed asap.  That
> won't work so flawlessly in PyPy and will cause hard to understand bugs.

Did you noticed, that we're stirring in the same kind of soup lately?
(apart from my person, who's not that deep in the details)

Given above code snippet, my issue is caused from "m" surviving *too* long 
after the context manager block ended, resulting in a delayed release of the 
memoryview, which in turn interferes with subsequent code.

Simple minded, as I am, I would expect, that "m" is actively removed from 
local context as soon as the context manager block ends. I would even argue, 
that this is part of the contract, that the context manager offers here.

AFAICS, "m" is handed over the the GC, but:
* this is somewhat surprising for the (probably simple minded) 
programmer
   * leads to problems *already*

A workaround is deleting "m" actively:

with memoryview(x) as m:
b = bytes(m)
del m

I would like to discuss the rationals of this construct from a language design 
POV, not language implementation POV.

usually-hacking-del-free-code-ly y'rs,
Pete
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Context manager lifetime rules Was: Re: Adding bytes.frombuffer() constructor to PEP 467

[INADA Naoki]

>> > with memoryview(x) as m:
>> >  b = bytes(m)
>> >  # or b = m.tobytes()
>>
>> Thinking more about this, and after looking at my own code in asyncpg
>> and uvloop, I'm now in favor of adding bytes.frombuffer() with the
>> proposed signature: ``bytes.frombuffer(byteslike, length=-1, offset=0)``
>>
>> Inada-san is right, the memoryview should be explicitly released, but
>> few do that. Instead, a lot of people simply rely on CPython refcounting
>> semantics, which will cause the temporary memoryview be GCed asap.  That
>> won't work so flawlessly in PyPy and will cause hard to understand bugs.
>
> Did you noticed, that we're stirring in the same kind of soup lately?
> (apart from my person, who's not that deep in the details)
>
> Given above code snippet, my issue is caused from "m" surviving *too* long
> after the context manager block ended, resulting in a delayed release of the
> memoryview, which in turn interferes with subsequent code.

In case of memoryview, it detaches x.  Like closed file object, it
doesn't hold any resource
other than memory used for the object itself.

>
> Simple minded, as I am, I would expect, that "m" is actively removed from
> local context as soon as the context manager block ends. I would even argue,
> that this is part of the contract, that the context manager offers here.

In case of general context manager, there are some use cases which needs "m"
after context ended.
For example, 
https://docs.python.org/3.5/library/unittest.html#unittest.TestCase.assertRaises
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2016-12-30 - 2017-01-06)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open5685 (+22)
  closed 35253 (+45)
  total  40938 (+67)

Open issues with patches: 2460 


Issues opened (44)
==

#24725: test_socket testFDPassEmpty fails on OS X 10.11+ with "Cannot 
http://bugs.python.org/issue24725  reopened by ned.deily

#29013: zipfile: inconsistent doc for ZIP64 file size
http://bugs.python.org/issue29013  reopened by serhiy.storchaka

#29116: Make str and bytes error messages on concatenation conform wit
http://bugs.python.org/issue29116  opened by Jim Fasarakis-Hilliard

#29117: dir() should include dunder attributes of the un
http://bugs.python.org/issue29117  opened by Antony.Lee

#29120: Move hash randomisation initialisation out of Python/random.c
http://bugs.python.org/issue29120  opened by ncoghlan

#29121: sqlite3 Controlling Transactions documentation not updated
http://bugs.python.org/issue29121  opened by palaviv

#29124: Freeze fails to compile on windows
http://bugs.python.org/issue29124  opened by Liran Ringel

#29125: Shell injection via TIX_LIBRARY when using tkinter.tix
http://bugs.python.org/issue29125  opened by symphorien

#29126: Fix comments about _PyThreadState_Current
http://bugs.python.org/issue29126  opened by Jim Nasby

#29127: Incorrect reference names in asyncio.subprocess documentation
http://bugs.python.org/issue29127  opened by Eric Ahn

#29132: shlex.shlex with punctuation_chars and posix doesn't handle pu
http://bugs.python.org/issue29132  opened by evan_

#29133: Minor inaccuracy in shlex.shlex punctuation_chars example
http://bugs.python.org/issue29133  opened by evan_

#29136: Add OP_NO_TLSv1_3
http://bugs.python.org/issue29136  opened by christian.heimes

#29137: Fix fpectl-induced ABI breakage
http://bugs.python.org/issue29137  opened by njs

#29139: operator.concat/iconcat could only work if left operand is a s
http://bugs.python.org/issue29139  opened by xiang.zhang

#29140: time_hash() reads the wrong bytes to get microseconds
http://bugs.python.org/issue29140  opened by haypo

#29141: error in 2to3
http://bugs.python.org/issue29141  opened by Sandeep Srinivasa

#29142: urllib: no_proxy variable values with leading dot not properly
http://bugs.python.org/issue29142  opened by tloetzer

#29143: Logger should ignore propagate property for disabled handlers.
http://bugs.python.org/issue29143  opened by Oleg Serov

#29144: Implicit namespace packages in Python 3.6
http://bugs.python.org/issue29144  opened by frenzy

#29145: failing overflow checks in replace_*
http://bugs.python.org/issue29145  opened by matejcik

#29147: registry value to be cleared when python is uninstalled
http://bugs.python.org/issue29147  opened by jha.amit

#29151: test_asyncore.TestAPI_UseIPv4Select / test_asyncore.TestAPI_Us
http://bugs.python.org/issue29151  opened by patrila

#29152: unittest subTest does not call addFailure
http://bugs.python.org/issue29152  opened by kristall

#29153: Test test.test_asynchat.TestAsynchat / test.test_asynchat.Test
http://bugs.python.org/issue29153  opened by patrila

#29154: 5 failures in test_doctest: ModuleNotFoundError: No module nam
http://bugs.python.org/issue29154  opened by Gerrit.Holl

#29155: test.test_spwd.TestSpwdNonRoot  failure with FileNotFoundError
http://bugs.python.org/issue29155  opened by Gerrit.Holl

#29157: random.c: Prefer getrandom() over getentropy(), handle ENOSYS 
http://bugs.python.org/issue29157  opened by haypo

#29158: Possible glitch in the interaction of a thread and a multiproc
http://bugs.python.org/issue29158  opened by luke_16

#29164: make test always fail at 218/405 ( AssertionError: ',' not fou
http://bugs.python.org/issue29164  opened by OO O

#29165: Use forward compatible macro in example code for creating new 
http://bugs.python.org/issue29165  opened by inada.naoki

#29167: Race condition in enum.py:_decompose()
http://bugs.python.org/issue29167  opened by simon.percivall

#29169: update zlib to 1.2.10
http://bugs.python.org/issue29169  opened by doko

#29171: blake2: Remove unused function to avoid undefined references
http://bugs.python.org/issue29171  opened by ericvw

#29172: blake2: Use lowest-common denominator signature of #pragma pac
http://bugs.python.org/issue29172  opened by ericvw

#29174: 'NoneType' object is not callable in subprocess.py
http://bugs.python.org/issue29174  opened by ita1024

#29175: Tutorial links to file object methods are broken.
http://bugs.python.org/issue29175  opened by jonro...@icloud.com

#29176: /tmp does not exist on Android and is used by curses.window.pu
http://bugs.python.org/issue29176  opened by xdegaye

#29177: skip tests using socketserver.UnixStreamServer when bind() rai
http://bugs.python.org/issue29177  opened by xdegaye

#29178: Adding bytes.frombuffer(byteslike) constructor
http://bugs.python.org

[Python-Dev] enable-framework Vs. Naught

[André Lemos]

Hi,


I have a C++ module that I am compiling to use inside of my Python
installation under Mac OS.

If I compile & link it against a Framework enabled Python installation, it
works fine, but if I compile & link it against a *non* enabled Framework
installation that we use for distribution, I simply get a non inspiring:

Fatal Python error: PyThreadState_Get: no current thread


I am using python-config to get my flags on both the examples, but I simply
cannot get it to run (although it compiles fine) on a *non* enabled
Framework installation.


Thoughts/Help?



--
André Lemos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467

[Alexander Belopolsky]

On Thu, Jan 5, 2017 at 5:54 PM, Serhiy Storchaka 
wrote:

> On 05.01.17 22:37, Alexander Belopolsky wrote:
>
>> I propose the following:
>>
>> 1. For 3.6, restore and document 3.5 behavior.  Recommend that 3rd party
>> types that are both integer-like and buffer-like implement their own
>> __bytes__ method to resolve the bytes(x) ambiguity.
>>
>
> The __bytes__ method is used only by the bytes constructor, not by the
> bytearray constructor.


I am not sure this is deliberate.  See <
https://bugs.python.org/issue2415#msg71660>.

>
>
> 2. For 3.7, I would like to see a drastically simplified bytes(x):
>> 2.1.  Accept only objects with a __bytes__ method or a sequence of ints
>> in range(256).
>> 2.2.  Expand __bytes__ definition to accept optional encoding and errors
>> parameters.  Implement str.__bytes__(self, [encoding[, errors]]).
>>
>
> I think it is better to use the encode() method if you want to encode from
> non-strings.


Possibly, but the goal of my proposal is to lighten the logic in the
bytes(x, [encoding[, errors]])
constructor.  If it detects x.__bytes__, it should just call it with
whatever arguments are given.

>
>
> 2.3.  Implement new specialized bytes.fromsize and bytes.frombuffer
>> constructors as per PEP 467 and Inada Naoki proposals.
>>
>
> bytes.fromsize(n) is just b'\0'*n. I don't think this method is needed.
>

I don't care much about this.  If it helps removing bytes(int) case, I am
for it, otherwise ±0.


>
> bytes.frombuffer(x) is bytes(memoryview(x)) or memoryview(x).tobytes().


I've just tried Inada's patch < http://bugs.python.org/issue29178
>:

$ ./python.exe -m timeit -s "from array import array; x=array('f', [0])"
"bytes.frombuffer(x)"
200 loops, best of 5: 134 nsec per loop

$ ./python.exe -m timeit -s "from array import array; x=array('f', [0])"
"with memoryview(x) as m: bytes(m)"
50 loops, best of 5: 436 nsec per loop

A 3x speed-up seems to be worth it.


>
>
> 2.4. Implement memoryview.__bytes__ method so that bytes(memoryview(x))
>> works ad before.
>> 2.5.  Implement a fast bytearray.__bytes__ method.
>>
>
> This wouldn't help for the bytearray constructor. And wouldn't allow to
> avoid double copying in the constructor of bytes subclass.


I don't see why bytearray constructor should behave differently from bytes.


>
> 3. Consider promoting __bytes__ to a tp_bytes type slot.
>>
>
> The buffer protocol is more general than the __bytes__ method. It allows
> to avoid redundant memory copying in constructors of many types (bytes,
> bytearray, array.array, etc), not just bytes.
>

It looks like there are two different views on what the bytes type
represents.  Is it a sequence of small integers or a blob of binary data?

Compare these two calls:

>>> from array import array
>>> bytes(array('h', [1, 2, 3]))
b'\x01\x00\x02\x00\x03\x00'

and

>>> bytes(array('f', [1, 2, 3]))
b'\x00\x00\x80?\x00\x00\x00@\x00\x00@@'

For me the __bytes__ method is a way for types to specify their bytes
representation that may or may not be the same as memoryview(x).tobytes().
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] enable-framework Vs. Naught

[Terry Reedy]

On 1/6/2017 12:39 PM, André Lemos wrote:


I have a C++ module that I am compiling to use inside of my Python
installation under Mac OS.

If I compile & link it against a Framework enabled Python installation,
it works fine, but if I compile & link it against a /non/ enabled
Framework installation that we use for distribution, I simply get a non
inspiring:

Fatal Python error: PyThreadState_Get: no current thread


I am using python-config to get my flags on both the examples, but I
simply cannot get it to run (although it compiles fine) on a
/non/ enabled Framework installation.


This list is for development of furture releases.  Questions about using 
current releases should go to python-list, Stackoverflow, or other help 
forums.


--
Terry Jan Reedy


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467

[Serhiy Storchaka]

On 06.01.17 21:31, Alexander Belopolsky wrote:

On Thu, Jan 5, 2017 at 5:54 PM, Serhiy Storchaka mailto:storch...@gmail.com>> wrote:

On 05.01.17 22:37, Alexander Belopolsky wrote:

2. For 3.7, I would like to see a drastically simplified bytes(x):
2.1.  Accept only objects with a __bytes__ method or a sequence
of ints
in range(256).
2.2.  Expand __bytes__ definition to accept optional encoding
and errors
parameters.  Implement str.__bytes__(self, [encoding[, errors]]).


I think it is better to use the encode() method if you want to
encode from non-strings.


Possibly, but the goal of my proposal is to lighten the logic in the
bytes(x, [encoding[, errors]])
constructor.  If it detects x.__bytes__, it should just call it with
whatever arguments are given.


I think this would complicate the __bytes__ protocol. I don't know 
precedences of passing additional optional arguments to a special 
method. int() doesn't pass the base argument to __int__, str() doesn't 
pass encoding and errors to __str__, and pickle.dumps() passes the 
protocol argument to new special method __reduce_ex__ instead of __reduce__.



bytes.frombuffer(x) is bytes(memoryview(x)) or memoryview(x).tobytes().


I've just tried Inada's patch < http://bugs.python.org/issue29178
>:

$ ./python.exe -m timeit -s "from array import array; x=array('f', [0])"
"bytes..frombuffer(x)"
200 loops, best of 5: 134 nsec per loop

$ ./python.exe -m timeit -s "from array import array; x=array('f', [0])"
"with memoryview(x) as m: bytes(m)"
50 loops, best of 5: 436 nsec per loop

A 3x speed-up seems to be worth it.


There is a constant overhead for calling functions. It is dwarfen by 
memory copying for large arrays. I'm not sure that 300 ns is worth 
adding new method.



2.4. Implement memoryview.__bytes__ method so that
bytes(memoryview(x))
works ad before.
2.5.  Implement a fast bytearray.__bytes__ method.


This wouldn't help for the bytearray constructor. And wouldn't allow
to avoid double copying in the constructor of bytes subclass.


I don't see why bytearray constructor should behave differently from bytes.


bytes constructor can just return the result of __bytes__. bytearray 
constructor needs to do a double copying if support __bytes__. First 
copy a data to a bytes object returned by __bytes__, then copy it's 
content to the newly created bytearray object. Creating a bytearray 
object using the buffer protocol needs only one copying.


Perhaps this is the cause why the support of __bytes__ was not added in 
bytearray constructor after all.



Compare these two calls:


from array import array
bytes(array('h', [1, 2, 3]))

b'\x01\x00\x02\x00\x03\x00'

and


bytes(array('f', [1, 2, 3]))

b'\x00\x00\x80?\x00\x00\x00@\x00\x00@@'


I don't see a difference.


For me the __bytes__ method is a way for types to specify their bytes
representation that may or may not be the same as memoryview(x).tobytes().


It would be confusing if some type that supports the buffer protocol 
would implement __bytes__ returning a result different from 
memoryview(x).tobytes(). If you want to get b'\1\2\3' from array('h', 
[1, 2, 3]), use bytes(list(x)).


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467

[Alexander Belopolsky]

On Fri, Jan 6, 2017 at 4:43 PM, Serhiy Storchaka 
wrote:

> Compare these two calls:
>>
>> from array import array
> bytes(array('h', [1, 2, 3]))
>
 b'\x01\x00\x02\x00\x03\x00'
>>
>> and
>>
>> bytes(array('f', [1, 2, 3]))
>
 b'\x00\x00\x80?\x00\x00\x00@\x00\x00@@'
>>
>
> I don't see a difference.


Hmm indeed.  Doesn't this contradict the documentation?

>>> help(bytes)
..
class bytes(object)
 |  bytes(iterable_of_ints) -> bytes
 |  bytes(string, encoding[, errors]) -> bytes
 |  bytes(bytes_or_buffer) -> immutable copy of bytes_or_buffer
..

This suggests that iterable_of_ints is tried first.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com