Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467

2017-01-10 Thread Chris Barker - NOAA Federal
> This is what happens with numpy arrays: > > >>> bytes(numpy.array([2], 'i1')) > b'\x00\x00' > > >>> bytes(numpy.array([2, 2], 'i1')) > b'\x02\x02' > > For better or worse, single-element numpy arrays have a working __index__ > methods Ouch -- that probably is for the worse.. There are Numpy

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467

2017-01-06 Thread Alexander Belopolsky
On Fri, Jan 6, 2017 at 4:43 PM, Serhiy Storchaka wrote: > Compare these two calls: >> >> from array import array > bytes(array('h', [1, 2, 3])) > b'\x01\x00\x02\x00\x03\x00' >> >> and >> >> bytes(array('f', [1, 2, 3])) > b'\x00\x00\x80?\x00\x00\x00@\x00\x00@@' >> > > I don't

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467

2017-01-06 Thread Serhiy Storchaka
On 06.01.17 21:31, Alexander Belopolsky wrote: On Thu, Jan 5, 2017 at 5:54 PM, Serhiy Storchaka mailto:storch...@gmail.com>> wrote: On 05.01.17 22:37, Alexander Belopolsky wrote: 2. For 3.7, I would like to see a drastically simplified bytes(x): 2.1. Accept only objects wit

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467

2017-01-06 Thread Alexander Belopolsky
On Thu, Jan 5, 2017 at 5:54 PM, Serhiy Storchaka wrote: > On 05.01.17 22:37, Alexander Belopolsky wrote: > >> I propose the following: >> >> 1. For 3.6, restore and document 3.5 behavior. Recommend that 3rd party >> types that are both integer-like and buffer-like implement their own >> __bytes_

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467

2017-01-06 Thread Antoine Pitrou
On Thu, 5 Jan 2017 20:28:26 -0500 Yury Selivanov wrote: > On 2017-01-05 7:11 PM, INADA Naoki wrote: > >> bytes.frombuffer(x) is bytes(memoryview(x)) or memoryview(x).tobytes(). > >> > > There is pitfall: memoryview should be closed. > > So b = bytes.frombuffer(x) is: > > > > with memoryview(x) a

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467

2017-01-06 Thread INADA Naoki
> > Thinking more about this, and after looking at my own code in asyncpg and > uvloop, I'm now in favor of adding bytes.frombuffer() with the proposed > signature: ``bytes.frombuffer(byteslike, length=-1, offset=0)`` > Do you prefer a signature proposed first? I thought it from asyncio usage too.

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467

2017-01-06 Thread INADA Naoki
I submit an issue about it. See https://bugs.python.org/issue29178 On Fri, Jan 6, 2017 at 7:38 PM, INADA Naoki wrote: >> >> Thinking more about this, and after looking at my own code in asyncpg and >> uvloop, I'm now in favor of adding bytes.frombuffer() with the proposed >> signature: ``bytes.f

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467

2017-01-05 Thread Yury Selivanov
On 2017-01-05 7:11 PM, INADA Naoki wrote: bytes.frombuffer(x) is bytes(memoryview(x)) or memoryview(x).tobytes(). There is pitfall: memoryview should be closed. So b = bytes.frombuffer(x) is: with memoryview(x) as m: b = bytes(m) # or b = m.tobytes() Thinking more about this, an

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467

2017-01-05 Thread INADA Naoki
> > bytes.frombuffer(x) is bytes(memoryview(x)) or memoryview(x).tobytes(). > There is pitfall: memoryview should be closed. So b = bytes.frombuffer(x) is: with memoryview(x) as m: b = bytes(m) # or b = m.tobytes() ___ Python-Dev mailing list Py

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467

2017-01-05 Thread Serhiy Storchaka
On 05.01.17 22:37, Alexander Belopolsky wrote: I propose the following: 1. For 3.6, restore and document 3.5 behavior. Recommend that 3rd party types that are both integer-like and buffer-like implement their own __bytes__ method to resolve the bytes(x) ambiguity. The __bytes__ method is used

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467

2017-01-05 Thread Alexander Belopolsky
On Wed, Oct 12, 2016 at 12:08 AM, INADA Naoki wrote: > > Now I'm sure about bytes.frombuffer() is worth enough. I would like to revive this thread (taking a liberty to shorten the subject line.) The issue of how the bytes(x) constructor should behave when given objects of various types have come

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-25 Thread Chris Barker
On Tue, Oct 25, 2016 at 12:25 AM, Nick Coghlan wrote: > The suggestion came from folks working on asyncio performance > improvements, > I'm *absolutely* suggesting > that they put some paid time and energy into a lower level buffer > manipulation library between now and the Python 3.7 featur

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-25 Thread Nick Coghlan
On 25 October 2016 at 17:25, Nick Coghlan wrote: > as well as Yury's new > libuv-based ``uvloop`` asyncio event loop implementation. Oops, I phrased that badly - the default asyncio event loop implementation is still pure Python, but uvloop is a drop-in Cython based replacement. Cheers, Nick. -

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-25 Thread Nick Coghlan
On 25 October 2016 at 02:53, Chris Barker wrote: > IS this na either-or? IF someone is proposing a nice lib for "low level data > buffer > manipulation", then yes, putting frombuffer() in there would be a fine idea. > > But if there is no such proposal on the table, then I think adding a > frombuf

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-24 Thread Chris Barker
Before the horse is totally dead... (maybe it already is), a couple comments: > In such cases, an extension module written in something like Cython, C > or Rust would be a better fit, well, yes, but: > From that perspective, adding "[bytes/bytearray].frombuffer" this would be used for the f

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-22 Thread Nick Coghlan
On 22 October 2016 at 07:57, Chris Barker wrote: > I'm still confused about the "io" in "iobuffers" -- I've used buffers a lot > -- for passing data around between various C libs -- numpy, image > processing, etc... I never really thought of it as IO though. which is why a > simple frombuffer() se

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-21 Thread Chris Barker
On Thu, Oct 20, 2016 at 11:48 PM, Nick Coghlan wrote: > > len(get_builtin_methods()) > >>230 > > > > So what? No one looks in all the methods of builtins at once. > > Yes, Python implementation developers do, which is why it's a useful > part of defining the overall "size" of Python and h

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-20 Thread Nick Coghlan
On 19 October 2016 at 01:28, Chris Barker - NOAA Federal wrote: > def get_builtin_methods(): >>... return [(name, method_name) for name, obj in >> get_builtin_types().items() for method_name, method in >> vars(obj).items() if not method_name.startswith("__")] >>... > len(get_bu

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-18 Thread Chris Barker - NOAA Federal
>> >> >> Method proliferation on builtins is a Big Deal(TM) > > I wanted to quantify this concept, so here's a quick metric that helps > convey how every time we add a new builtin method we're immediately > making Python harder to comprehend: > def get_builtin_types(): >... import buil

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-12 Thread Nick Coghlan
On 13 October 2016 at 12:54, Nick Coghlan wrote: > Method proliferation on builtins is a Big Deal(TM) I wanted to quantify this concept, so here's a quick metric that helps convey how every time we add a new builtin method we're immediately making Python harder to comprehend: >>> def get_bui

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-12 Thread INADA Naoki
> > Also, why do the conversion from bytearray to bytes? It is definitely not > always needed. > ba = bytearray(b'abc') b = b'def' ba + b > bytearray(b'abcdef') b'%s %s' % (ba, b) > b'abc def' b + ba > b'defabc' ba.extend(b) ba > bytearray(b'abcdef') > > Even if

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-12 Thread Nick Coghlan
On 13 October 2016 at 02:37, Stephen J. Turnbull wrote: > Victor Stinner writes: > > 2016-10-12 11:34 GMT+02:00 INADA Naoki : > > > > I see. My proposal should be another PEP (if PEP is required). > > > > I don't think that adding a single method deserves its own method. > > You mean "deserve

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-12 Thread Terry Reedy
On 10/12/2016 5:42 AM, INADA Naoki wrote: On Wed, Oct 12, 2016 at 2:32 PM, Serhiy Storchaka wrote: On 12.10.16 07:08, INADA Naoki wrote: Sample code: def read_line(buf: bytearray) -> bytes: try: n = buf.index(b'\r\n') except ValueError: return b''

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-12 Thread Victor Stinner
Oops, right, I wanted to write "I don't think that adding a single method deserves its own PEP." Victor 2016-10-12 18:37 GMT+02:00 Stephen J. Turnbull : > Victor Stinner writes: > > 2016-10-12 11:34 GMT+02:00 INADA Naoki : > > > > I see. My proposal should be another PEP (if PEP is required).

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-12 Thread Stephen J. Turnbull
Victor Stinner writes: > 2016-10-12 11:34 GMT+02:00 INADA Naoki : > > I see. My proposal should be another PEP (if PEP is required). > > I don't think that adding a single method deserves its own method. You mean "deserves own PEP", right? I interpreted Nick to say that "the reasons that a

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467

2016-10-12 Thread Nikolaus Rath
On Oct 11 2016, Nathaniel Smith wrote: > On Tue, Oct 11, 2016 at 9:08 PM, INADA Naoki wrote: >> From Python 3.4, bytearray is good solution for I/O buffer, thanks to >> #19087 [1]. >> Actually, asyncio uses bytearray as I/O buffer often. > > Whoa what?! This is awesome, I had no idea that bytearr

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-12 Thread Victor Stinner
2016-10-12 11:34 GMT+02:00 INADA Naoki : > I see. My proposal should be another PEP (if PEP is required). I don't think that adding a single method deserves its own method. I like the idea with Serhiy's API (as Python 2 buffer constructor): bytes.frombuf(buffer, [offset, size]) bytearray.frombu

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-12 Thread INADA Naoki
On Wed, Oct 12, 2016 at 2:32 PM, Serhiy Storchaka wrote: > On 12.10.16 07:08, INADA Naoki wrote: >> >> Sample code: >> >> def read_line(buf: bytearray) -> bytes: >> try: >> n = buf.index(b'\r\n') >> except ValueError: >> return b'' >> >> line = b

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-12 Thread INADA Naoki
On Wed, Oct 12, 2016 at 2:07 PM, Nick Coghlan wrote: > I don't think it makes sense to add any more ideas to PEP 467. That > needed to be a PEP because it proposed breaking backwards > compatibility in a couple of areas, and because of the complex history > of Python 3's "bytes-as-tuple-of-ints" a

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-11 Thread Serhiy Storchaka
On 12.10.16 08:03, Nathaniel Smith wrote: On Tue, Oct 11, 2016 at 9:08 PM, INADA Naoki wrote: From Python 3.4, bytearray is good solution for I/O buffer, thanks to #19087 [1]. Actually, asyncio uses bytearray as I/O buffer often. Whoa what?! This is awesome, I had no idea that bytearray had O

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-11 Thread Serhiy Storchaka
On 12.10.16 07:08, INADA Naoki wrote: Sample code: def read_line(buf: bytearray) -> bytes: try: n = buf.index(b'\r\n') except ValueError: return b'' line = bytes(buf)[:n] # bytearray -> bytes -> bytes Wouldn't be more correct to write this

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-11 Thread Nick Coghlan
I don't think it makes sense to add any more ideas to PEP 467. That needed to be a PEP because it proposed breaking backwards compatibility in a couple of areas, and because of the complex history of Python 3's "bytes-as-tuple-of-ints" and Python 2's "bytes-as-str" semantics. Other enhancements to

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor

2016-10-11 Thread Nathaniel Smith
On Tue, Oct 11, 2016 at 9:08 PM, INADA Naoki wrote: > From Python 3.4, bytearray is good solution for I/O buffer, thanks to > #19087 [1]. > Actually, asyncio uses bytearray as I/O buffer often. Whoa what?! This is awesome, I had no idea that bytearray had O(1) deletes at the front. I literally re