Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Aug 1, 2012, at 1:46 AM, Mark Shannon wrote: > > ''' > Being able to pre-allocate lists based on the expected size, as estimated by > __length_hint__, > can be a significant optimization. > PyPy has been observed to run some code slower than CPython, purely because > this optimization is ab

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Wed, Aug 1, 2012 at 12:06 PM, Mark Shannon wrote: > Maciej Fijalkowski wrote: >> >> On Wed, Aug 1, 2012 at 10:46 AM, Mark Shannon wrote: >>> >>> While the idea behind PEP 424 is sound, the text of the PEP is rather >>> vague >>> and missing a lot of details. >>> There was extended discussion o

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Maciej Fijalkowski wrote: On Wed, Aug 1, 2012 at 10:46 AM, Mark Shannon wrote: While the idea behind PEP 424 is sound, the text of the PEP is rather vague and missing a lot of details. There was extended discussion on the details, but none of that has appeared in the PEP yet. So Alex, how abou

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Wed, Aug 1, 2012 at 10:46 AM, Mark Shannon wrote: > While the idea behind PEP 424 is sound, the text of the PEP is rather vague > and missing a lot of details. > There was extended discussion on the details, but none of that has appeared > in the PEP yet. > > So Alex, how about adding those det

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

While the idea behind PEP 424 is sound, the text of the PEP is rather vague and missing a lot of details. There was extended discussion on the details, but none of that has appeared in the PEP yet. So Alex, how about adding those details? Also the rationale is rather poor. Given that CPython i

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Sun, Jul 15, 2012 at 1:28 AM, Alexandre Zani wrote: > I'm +1 on not having a public API for this. Ultimately the contract > for a length hint will depend heavily upon what you need it for. Some > applications would require a length hint to be an "at least" others an > "at most" and others somet

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Tue, 17 Jul 2012 13:19:55 +1000 Nick Coghlan wrote: > > > There are no provisions for infinite iterators, that is not within the > > scope of > > this proposal. > > I'll repeat my observation that remaining silent on this point is > effectively identical to blessing the practice of raising a

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Sun, Jul 15, 2012 at 1:36 PM, Antoine Pitrou wrote: > On Sun, 15 Jul 2012 18:47:38 +1000 > Nick Coghlan wrote: >> I'm not seeing the value in returning None over 0 for the don't know >> case - it just makes the API harder to use. > > The point is that 0 is a legitimate value for a length hint.

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Tue, Jul 17, 2012 at 1:03 PM, Alex Gaynor wrote: > I've updated the PEP to reflect the discussion. There are two major changes: > > 1) NotImplemented may be used by __length_hint__ to indicate that there is no >finite length hint available. I've been thinking about this a bit more, and I t

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

I've updated the PEP to reflect the discussion. There are two major changes: 1) NotImplemented may be used by __length_hint__ to indicate that there is no finite length hint available. 2) callers of operator.length_hint() must provide their own default value, this is also required by the cu

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

M Stefan wrote: Also, what would iter([1,2,3]).__length_hint__() return? 3 or unknown? If 3, then the semantics of l=[1,2,3]; l += iter(l) will change (infinite loop without __length_hint__ vs. list of 6 elements with __length_hint__). What __length_hint__ returns is irrelevant -- it's only a

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On 7/16/2012 9:54 AM, Stefan Behnel wrote: Mark Shannon, 15.07.2012 16:14: Alex Gaynor wrote: CPython currently defines an ``__length_hint__`` method on several types, such as various iterators. This method is then used by various other functions (such as ``map``) to presize lists based on the

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Mon, Jul 16, 2012 at 3:36 AM, Stefan Behnel wrote: > Alex Gaynor, 15.07.2012 00:11: > > CPython currently defines an ``__length_hint__`` method on several > types, such > > as various iterators. This method is then used by various other > functions (such > > as ``map``) to presize lists based

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Jul 16, 2012, at 12:36 AM, Stefan Behnel wrote: > I'd like to more visibly repeat my suggestion to make this a slot method > "tp_length_hint()" in extension types that returns a Py_ssize_t. That is merely an implementation detail, but it would be a nice improvement. Raymond ___

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On 16.07.12 10:36, Stefan Behnel wrote: Return values could be -1 for "don't know" and -2 for "infinite" at the C level, and NotImplemented for "don't know" at the Python level. PY_SSIZE_T_MAX is better value for "infinite". In any case no difference for consumer between PY_SSIZE_T_MAX and a r

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Jul 15, 2012, at 7:22 AM, Antoine Pitrou wrote: > On Mon, 16 Jul 2012 00:08:41 +1000 > Nick Coghlan wrote: >> Right, I agree on the value in being able to return something to say "this >> cannot be converted to a concrete container". > > Who would be able to return that, apart from trivial c

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Nick Coghlan, 16.07.2012 10:26: > Proposing anything substantially more complicated than what is currently > implemented in CPython will just get the idea rejected. The substantial > immediate gain for PyPy is in skipping the memory resizing when building > containers from itertools iterators, not

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Proposing anything substantially more complicated than what is currently implemented in CPython will just get the idea rejected. The substantial immediate gain for PyPy is in skipping the memory resizing when building containers from itertools iterators, not anywhere else. -- Sent from my phone, t

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Alex Gaynor, 15.07.2012 00:11: > CPython currently defines an ``__length_hint__`` method on several types, such > as various iterators. This method is then used by various other functions > (such > as ``map``) to presize lists based on the estimated returned by > ``__length_hint__``. Types can th

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Antoine Pitrou, 15.07.2012 17:06: > On Sun, 15 Jul 2012 16:33:23 +0200 > Christian Heimes wrote: >> Am 15.07.2012 16:22, schrieb Antoine Pitrou: >>> On Mon, 16 Jul 2012 00:08:41 +1000 >>> Nick Coghlan wrote: Right, I agree on the value in being able to return something to say "this cannot

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Mark Shannon, 15.07.2012 16:14: > Alex Gaynor wrote: >> CPython currently defines an ``__length_hint__`` method on several types, >> such >> as various iterators. This method is then used by various other functions >> (such as ``map``) to presize lists based on the estimated returned by > > Don't

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Stephen J. Turnbull wrote: The point is that I don't really see the value in returning a precise estimate that cannot be relied on to be accurate. OK, Python is a "consenting adults" language, but returning an integer here seems like invitation to abuse. Since __length_hint__ already exists a

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Jul 16, 2012 1:52 PM, "Stephen J. Turnbull" wrote: > The point is that I don't really see the value in returning a precise > estimate that cannot be relied on to be accurate. OK, Python is a > "consenting adults" language, but returning an integer here seems like > invitation to abuse. Becaus

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Nick Coghlan writes: > Right, I agree on the value in being able to return something to say "this > cannot be converted to a concrete container". > > I still haven't seen a use case where the appropriate response to "I don't > know" differs from the appropriate response to a hint of zero - t

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Steven D'Aprano wrote: With a length hint, we could strengthen that promise: "if __length_hint__ returns a negative number, list, tuple and set will fail immediately with MemoryError" which I think is a good safety feature for some things which cannot possibly succeed, but risk DOSing your

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Mon, 16 Jul 2012 02:21:20 +1000 Steven D'Aprano wrote: > > > My conclusion is that an infinite iterator is a documentation issue. > > Just tell the user that it doesn't stop, and let them shoot themselves > > in the foot in they want to. > > Buffer overflows are a documentation issue. Just te

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Mon, 16 Jul 2012 02:00:58 +1000 Chris Angelico wrote: > On Mon, Jul 16, 2012 at 1:55 AM, Steven D'Aprano wrote: > > (I expect the difference in behaviour is due to the default ulimit under > > Debian/Mint and RedHat/Fedora systems.) > > Possibly also virtual memory settings. Allocating gobs o

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Antoine Pitrou wrote: First, you can't implement __length_hint__ for a generator, which is the preferred (the most practical) way of writing iterators in pure Python. Limitations of generators are no reason for not improving iterators which are not generators. __length_hint__ already exists;

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Mon, Jul 16, 2012 at 1:55 AM, Steven D'Aprano wrote: > (I expect the difference in behaviour is due to the default ulimit under > Debian/Mint and RedHat/Fedora systems.) Possibly also virtual memory settings. Allocating gobs of memory with a huge page file slows everything down without raising

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Mark Shannon wrote: So how does an iterator express infinite length? The suggestion was it should return -1. What should happen if I am silly enough to do this: >>> list(itertools.count()) This will fail; it should fail quickly. That depends on your OS. I've just tested it now on Linux

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Sun, Jul 15, 2012 at 8:08 AM, Mark Shannon wrote: > Brett Cannon wrote: > >> >> >> On Sun, Jul 15, 2012 at 10:39 AM, Mark Shannon > > wrote: >> >> Nick Coghlan wrote: >> >> Right, I agree on the value in being able to return something to >> say "this c

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Sun, 15 Jul 2012 16:08:00 +0100 Mark Shannon wrote: > > What should happen if I am silly enough to do this: > >>> list(itertools.count()) > > This will fail; it should fail quickly. Why should it? AFAIK it's not a common complaint. You said it yourself: it's a silly thing to do. Regards A

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Sun, 15 Jul 2012 16:33:23 +0200 Christian Heimes wrote: > Am 15.07.2012 16:22, schrieb Antoine Pitrou: > > On Mon, 16 Jul 2012 00:08:41 +1000 > > Nick Coghlan wrote: > >> Right, I agree on the value in being able to return something to say "this > >> cannot be converted to a concrete container

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Brett Cannon wrote: On Sun, Jul 15, 2012 at 10:39 AM, Mark Shannon > wrote: Nick Coghlan wrote: Right, I agree on the value in being able to return something to say "this cannot be converted to a concrete container". I still haven't seen a

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Am 15.07.2012 16:39, schrieb Mark Shannon: > 1. Don't implement it at all. > > 2. Implement __length_hint__() but don't want to return any value. >Either raise an exception (TypeError) -- As suggested in the PEP. >or return NotImplemented -- my preferred option. How is this different from

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Sun, Jul 15, 2012 at 10:39 AM, Mark Shannon wrote: > Nick Coghlan wrote: > >> Right, I agree on the value in being able to return something to say >> "this cannot be converted to a concrete container". >> >> I still haven't seen a use case where the appropriate response to "I >> don't know" di

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Alex Gaynor wrote: Hi all, I've just submitted a PEP proposing making __length_hint__ a public API for users to define and other VMs to implement: These seems back-to-front. __length_hint__ is *used* by the VM, not provided by it. It should be part of the object model, rather than the API.

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Nick Coghlan wrote: Right, I agree on the value in being able to return something to say "this cannot be converted to a concrete container". I still haven't seen a use case where the appropriate response to "I don't know" differs from the appropriate response to a hint of zero - that is, you

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Am 15.07.2012 16:22, schrieb Antoine Pitrou: > On Mon, 16 Jul 2012 00:08:41 +1000 > Nick Coghlan wrote: >> Right, I agree on the value in being able to return something to say "this >> cannot be converted to a concrete container". > > Who would be able to return that, apart from trivial cases lik

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Mon, 16 Jul 2012 00:08:41 +1000 Nick Coghlan wrote: > Right, I agree on the value in being able to return something to say "this > cannot be converted to a concrete container". Who would be able to return that, apart from trivial cases like itertools.cycle()? Regards Antoine. -- Software

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Right, I agree on the value in being able to return something to say "this cannot be converted to a concrete container". I still haven't seen a use case where the appropriate response to "I don't know" differs from the appropriate response to a hint of zero - that is, you don't preallocate, you ju

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Antoine Pitrou wrote: The point is that 0 is a legitimate value for a length hint. Simple implementations of __length_hint__ will start returning 0 as a legitimate value and you will wrongly interpret that as "don't know", which kinds of defeat the purpose of __length-hint__ ;) That said, I

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Sun, 15 Jul 2012 18:47:38 +1000 Nick Coghlan wrote: > > > * __length_hint__ should be allowed to return None to indicate "don't know" > > or -1 to indicate "infinite". > > > > Presumably anything that wishes to create a list or other sequence from an > > object with a hint of -1 could then r

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Alex Gaynor, 15.07.2012 07:20: > there's no way for the __lenght_hint__ to specify that > that particular instance can't have a length hint computed. e.g. imagine > some sort of lazy stream that cached itself, and only wanted to offer a > length hint if it had already been evaluated. Without an e

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Sun, Jul 15, 2012 at 6:21 PM, Steven D'Aprano wrote: > I suggest: > > * object (and hence all other types that don't explicitly override it) > should have a __length_hint__ that raises TypeError; We can keep it simpler than that just by changing the order of the checks. > * __length_hint__

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

Nick Coghlan wrote: On Sun, Jul 15, 2012 at 9:18 AM, Benjamin Peterson wrote: Open questions == There are two open questions for this PEP: * Should ``list`` expose a kwarg in it's constructor for supplying a length hint. * Should a function be added either to ``builtins`` or som

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Sat, Jul 14, 2012 at 10:16 PM, Nick Coghlan wrote: > On Sun, Jul 15, 2012 at 9:18 AM, Benjamin Peterson > wrote: > >> Open questions > >> == > >> > >> There are two open questions for this PEP: > >> > >> * Should ``list`` expose a kwarg in it's constructor for supplying a > length

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Sun, Jul 15, 2012 at 9:18 AM, Benjamin Peterson wrote: >> Open questions >> == >> >> There are two open questions for this PEP: >> >> * Should ``list`` expose a kwarg in it's constructor for supplying a length >> hint. >> * Should a function be added either to ``builtins`` or some

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On 7/14/2012 6:11 PM, Alex Gaynor wrote: ... Various thoughts: "This method is then used by various other functions (such +as ``map``) to presize lists" -- map no longer produces lists. This only makes sense in 3.x if you mean that map can pass along the value of its inputs. "Types can then

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

2012/7/14 Alex Gaynor : > > > On Sat, Jul 14, 2012 at 4:18 PM, Benjamin Peterson > wrote: >> >> 2012/7/14 Alex Gaynor : >> > >> > Proposal >> > >> > >> > This PEP proposes formally documenting ``__length_hint__`` for other >> > interpreter and non-standard library Python to implement. >>

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Sat, Jul 14, 2012 at 4:21 PM, Alex Gaynor wrote: > > > On Sat, Jul 14, 2012 at 4:18 PM, Benjamin Peterson > wrote: >> >> 2012/7/14 Alex Gaynor : >> > >> > Proposal >> > >> > >> > This PEP proposes formally documenting ``__length_hint__`` for other >> > interpreter and non-standard lib

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

On Sat, Jul 14, 2012 at 4:18 PM, Benjamin Peterson wrote: > 2012/7/14 Alex Gaynor : > > > > Proposal > > > > > > This PEP proposes formally documenting ``__length_hint__`` for other > > interpreter and non-standard library Python to implement. > > > > ``__length_hint__`` must return an in

Re: [Python-Dev] PEP 0424: A method for exposing a length hint

2012/7/14 Alex Gaynor : > > Proposal > > > This PEP proposes formally documenting ``__length_hint__`` for other > interpreter and non-standard library Python to implement. > > ``__length_hint__`` must return an integer, and is not required to be > accurate. > It may return a value that is

[Python-Dev] PEP 0424: A method for exposing a length hint

Hi all, I've just submitted a PEP proposing making __length_hint__ a public API for users to define and other VMs to implement: PEP: 424 Title: A method for exposing a length hint Version: $Revision$ Last-Modified: $Date Author: Alex Gaynor Status: Draft Type: Standards Track Content-Type: text