Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Ivan Levkivskyi wrote:
"People sometimes want to refactor for-loops containing `yield` into a 
comprehension but that doesn't work (particularly because of the hidden 
function scope) - lets make it a SyntaxError"


Personally I'd be fine with removing the implicit function
scope from comprehensions and allowing yield in them, since
the semantics of that are clear.

But I don't see a way to do anything equivalent with
generator expressions. Since the current effect of
yield in a generator expression is pretty useless,
it seems best just to disallow it.

That means a list comprehension won't be equivalent
to list(generator_expression) in all cases, but I
don't think there's any great need for it to be.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Ivan Levkivskyi wrote:
On 23 November 2017 at 05:44, Greg Ewing > wrote:


   def g():
  return ((yield i) for i in range(10))


I think this code should be just equivalent to this code

def g():
temp = [(yield i) for i in range(10)]
return (v for v in temp)


But then you get a non-lazy iterable, which defeats the
purpose of using a generator expression -- you might as
well have used a comprehension to begin with.

--
Greg

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Ivan Levkivskyi wrote:
"People sometimes want to refactor for-loops containing `yield` into a 
comprehension


By the way, do we have any real-life examples of people wanting to
do this? It might help us decide what the semantics should be.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 09:15, Greg Ewing 
wrote:

> Ivan Levkivskyi wrote:
>
>> On 23 November 2017 at 05:44, Greg Ewing > > wrote:
>>
>>def g():
>>   return ((yield i) for i in range(10))
>>
>>
>> I think this code should be just equivalent to this code
>>
>> def g():
>> temp = [(yield i) for i in range(10)]
>> return (v for v in temp)
>>
>
> But then you get a non-lazy iterable, which defeats the
> purpose of using a generator expression -- you might as
> well have used a comprehension to begin with.
>
>
This could be just a semantic equivalence (mental model), not how it should
be internally implemented.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 09:11, Greg Ewing 
wrote:

> Ivan Levkivskyi wrote:
>
>> "People sometimes want to refactor for-loops containing `yield` into a
>> comprehension but that doesn't work (particularly because of the hidden
>> function scope) - lets make it a SyntaxError"
>>
>
> Personally I'd be fine with removing the implicit function
> scope from comprehensions and allowing yield in them, since
> the semantics of that are clear.
>
> But I don't see a way to do anything equivalent with
> generator expressions. Since the current effect of
> yield in a generator expression is pretty useless,
> it seems best just to disallow it.
>
> That means a list comprehension won't be equivalent
> to list(generator_expression) in all cases, but I
> don't think there's any great need for it to be.
>

I am also fine with this. Generator expressions are indeed less clear.
Also different people have different mental model about them (especially re
implicit scope and equivalence to comprehensions).

On the contrary, vast majority agrees that comprehensions are just
for-loops without leaking variables.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 09:17, Greg Ewing 
wrote:

> Ivan Levkivskyi wrote:
>
>> "People sometimes want to refactor for-loops containing `yield` into a
>> comprehension
>>
>
> By the way, do we have any real-life examples of people wanting to
> do this? It might help us decide what the semantics should be.
>
>
Yes, there are two SO questions in two first posts here, also there are
some b.p.o. issues. It looks like in all case people expect:

def f():
return [(yield i) for i in range(3)]

to be roughly equivalent to

def f():
res = []
for i in range(3):
r = yield i
res.append(r)
return res

See Serhiy's original post for more detailed proposed semantic equivalence.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Steve Holden]

On Wed, Nov 22, 2017 at 8:48 PM, Sven R. Kunze  wrote:

> Isn't yield like a return?
>
​Enough like it to make a good case, I'd say.​


> A return in a list/dict/set comprehension makes no sense to me.
>
​Nor me, nor the vast majority of instance. But nowadays yield is more of a
synchronisation point. If something is valid syntax we should presumably
like to have defined semantics.

> So, +1 on SyntaxError from me too.
>
​I'd tend to agree. This would give more time to discuss the intended
semantics: giving it meaning later might be a more cautious approach that
would allow decisions to be made in the light of further experience.

I would urge developers, in their ​improvements to the language to support
asynchronous programming, to bear in mind that this is (currently) a
minority use case. Why the rush to set complex semantics in stone?

>
regards
 Steve​


> Cheers.
>
> On 22.11.2017 21:29, David Mertz wrote:
>
> Inasmuch as I get to opine, I'm +1 on SyntaxError. There is no behavior
> for that spelling that I would find intuitive or easy to explain to
> students. And as far as I can tell, the ONLY time anything has ever been
> spelled that way is in comments saying "look at this weird edge case
> behavior in Python."
>
> On Nov 22, 2017 10:57 AM, "Jelle Zijlstra" 
> wrote:
>
>
>
> 2017-11-22 9:58 GMT-08:00 Guido van Rossum :
>
>> Wow, 44 messages in 4 hours. That must be some kind of record.
>>
>> If/when there's an action item, can someone summarize for me?
>>
>> The main disagreement seems to be about what this code should do:
>
> g = [(yield i) for i in range(3)]
>
> Currently, this makes `g` into a generator, not a list. Everybody seems to
> agree this is nonintuitive and should be changed.
>
> One proposal is to make it so `g` gets assigned a list, and the `yield`
> happens in the enclosing scope (so the enclosing function would have to be
> a generator). This was the way things worked in Python 2, I believe.
>
> Another proposal is to make this code a syntax error, because it's
> confusing either way. (For what it's worth, that would be my preference.)
>
> There is related discussion about the semantics of list comprehensions
> versus calling list() on a generator expression, and of async semantics,
> but I don't think there's any clear point of action there.
>
>
>> --
>> --Guido van Rossum (python.org/~guido )
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/jelle.zij
>> lstra%40gmail.com
>>
>>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/mertz%
> 40gnosis.cx
>
>
>
>
> ___
> Python-Dev mailing 
> listPython-Dev@python.orghttps://mail.python.org/mailman/listinfo/python-dev
>
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> steve%40holdenweb.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python initialization and embedded Python

[M.-A. Lemburg]

On 18.11.2017 01:01, Victor Stinner wrote:
> Hi,
> 
> The CPython internals evolved during Python 3.7 cycle. I would like to
> know if we broke the C API or not.
> 
> Nick Coghlan and Eric Snow are working on cleaning up the Python
> initialization with the "on going" PEP 432:
> https://www.python.org/dev/peps/pep-0432/
> 
> Many global variables used by the "Python runtime" were move to a new
> single "_PyRuntime" variable (big structure made of sub-structures).
> See Include/internal/pystate.h.
> 
> A side effect of moving variables from random files into header files
> is that it's not more possible to fully initialize _PyRuntime at
> "compilation time". For example, previously, it was possible to refer
> to local C function (functions declared with "static", so only visible
> in the current file). Now a new "initialization function" is required
> to must be called.
> 
> In short, it means that using the "Python runtime" before it's
> initialized by _PyRuntime_Initialize() is now likely to crash. For
> example, calling PyMem_RawMalloc(), before calling
> _PyRuntime_Initialize(), now calls the function NULL: dereference a
> NULL pointer, and so immediately crash with a segmentation fault.

To prevent a complete crash, would it be possible to initialize
the struct entries to a generic function (or set of such functions
with the right signatures), which then issue a message to stderr
hinting to the missing call to _PyRuntime_Initialize()
before terminating ?

> I'm writing this email to ask if this change is an issue or not to
> embedded Python and the Python C API. Is it still possible to call
> "all" functions of the C API before calling Py_Initialize()?
> 
> I was bitten by the bug while reworking the Py_Main() function to
> split it into subfunctions and cleanup the code to handle the command
> line arguments and environment variables. I fixed the issue in main()
> by calling _PyRuntime_Initialize() as soon as possible: it's now the
> first instruction of main() :-) (See Programs/python.c)
> 
> To give a more concrete example: Py_DecodeLocale() is the recommanded
> function to decode bytes from the operating system, but this function
> calls PyMem_RawMalloc() which does crash before
> _PyRuntime_Initialize() is called. Is Py_DecodeLocale() used to
> initialize Python?
> 
> For example, "void Py_SetProgramName(wchar_t *);" expects a text
> string, whereas main() gives argv as bytes. Calling
> Py_SetProgramName() from argv requires to decode bytes... So use
> Py_DecodeLocale()...
> 
> Should we do something in Py_DecodeLocale()? Maybe crash if
> _PyRuntime_Initialize() wasn't called yet?
> 
> Maybe, the minimum change is to expose _PyRuntime_Initialize() in the
> public C API?
> 
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 23 2017)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 559 - built-in noop()

[Chris Jerdonek]

On Wed, Nov 22, 2017 at 4:32 PM, Victor Stinner 
wrote:

> Aha, contextlib.nullcontext() was just added, cool!
>

So is this equivalent to--

@contextmanager
def yielding(x):
yield x

I thought we were against adding one-line functions?

--Chris



>
> https://github.com/python/cpython/commit/0784a2e5b174d2dbf7b144d480559e
> 650c5cf64c
> https://bugs.python.org/issue10049
>
> Victor
>
> 2017-09-09 21:54 GMT+02:00 Victor Stinner :
> > I always wanted this feature (no kidding).
> >
> > Would it be possible to add support for the context manager?
> >
> > with noop(): ...
> >
> > Maybe noop can be an instance of:
> >
> > class Noop:
> >   def __enter__(self, *args, **kw): return self
> >   def __exit__(self, *args): pass
> >   def __call__(self, *args, **kw): return self
> >
> > Victor
> >
> > Le 9 sept. 2017 11:48 AM, "Barry Warsaw"  a écrit :
> >>
> >> I couldn’t resist one more PEP from the Core sprint.  I won’t reveal
> where
> >> or how this one came to me.
> >>
> >> -Barry
> >>
> >> PEP: 559
> >> Title: Built-in noop()
> >> Author: Barry Warsaw 
> >> Status: Draft
> >> Type: Standards Track
> >> Content-Type: text/x-rst
> >> Created: 2017-09-08
> >> Python-Version: 3.7
> >> Post-History: 2017-09-09
> >>
> >>
> >> Abstract
> >> 
> >>
> >> This PEP proposes adding a new built-in function called ``noop()`` which
> >> does
> >> nothing but return ``None``.
> >>
> >>
> >> Rationale
> >> =
> >>
> >> It is trivial to implement a no-op function in Python.  It's so easy in
> >> fact
> >> that many people do it many times over and over again.  It would be
> useful
> >> in
> >> many cases to have a common built-in function that does nothing.
> >>
> >> One use case would be for PEP 553, where you could set the breakpoint
> >> environment variable to the following in order to effectively disable
> it::
> >>
> >> $ setenv PYTHONBREAKPOINT=noop
> >>
> >>
> >> Implementation
> >> ==
> >>
> >> The Python equivalent of the ``noop()`` function is exactly::
> >>
> >> def noop(*args, **kws):
> >> return None
> >>
> >> The C built-in implementation is available as a pull request.
> >>
> >>
> >> Rejected alternatives
> >> =
> >>
> >> ``noop()`` returns something
> >> 
> >>
> >> YAGNI.
> >>
> >> This is rejected because it complicates the semantics.  For example, if
> >> you
> >> always return both ``*args`` and ``**kws``, what do you return when none
> >> of
> >> those are given?  Returning a tuple of ``((), {})`` is kind of ugly, but
> >> provides consistency.  But you might also want to just return ``None``
> >> since
> >> that's also conceptually what the function was passed.
> >>
> >> Or, what if you pass in exactly one positional argument, e.g.
> ``noop(7)``.
> >> Do
> >> you return ``7`` or ``((7,), {})``?  And so on.
> >>
> >> The author claims that you won't ever need the return value of
> ``noop()``
> >> so
> >> it will always return ``None``.
> >>
> >> Coghlin's Dialogs (edited for formatting):
> >>
> >> My counterargument to this would be ``map(noop, iterable)``,
> >> ``sorted(iterable, key=noop)``, etc. (``filter``, ``max``, and
> >> ``min`` all accept callables that accept a single argument, as do
> >> many of the itertools operations).
> >>
> >> Making ``noop()`` a useful default function in those cases just
> >> needs the definition to be::
> >>
> >>def noop(*args, **kwds):
> >>return args[0] if args else None
> >>
> >> The counterargument to the counterargument is that using ``None``
> >> as the default in all these cases is going to be faster, since it
> >> lets the algorithm skip the callback entirely, rather than calling
> >> it and having it do nothing useful.
> >>
> >>
> >> Copyright
> >> =
> >>
> >> This document has been placed in the public domain.
> >>
> >>
> >> ..
> >>Local Variables:
> >>mode: indented-text
> >>indent-tabs-mode: nil
> >>sentence-end-double-space: t
> >>fill-column: 70
> >>coding: utf-8
> >>End:
> >>
> >>
> >> ___
> >> Python-Dev mailing list
> >> Python-Dev@python.org
> >> https://mail.python.org/mailman/listinfo/python-dev
> >> Unsubscribe:
> >> https://mail.python.org/mailman/options/python-dev/
> victor.stinner%40gmail.com
> >>
> >
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> chris.jerdonek%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 23 November 2017 at 09:14, Steve Holden  wrote:
> I would urge developers, in their improvements to the language to support
> asynchronous programming, to bear in mind that this is (currently) a
> minority use case. Why the rush to set complex semantics in stone?

+1

Also, given that languages like C# have similar async/await
functionality, I'd be interested to know how they address questions
like this. If they have a parallel, we should probably follow it. If
they don't that would be further indication that no-one has much
experience of the "best answers" yet, and caution is indicated.

BTW, I'm assuming that the end goal is for async to be a natural and
fully-integrated part of the language, no more "minority use" than
generators, or context managers. Assuming that's the case, I think
that keeping a very careful eye on how intuitive async feels to
non-specialists is crucial (so thanks to Ivan and Yury for taking the
time to respond to this discussion).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python initialization and embedded Python

[Antoine Pitrou]

On Thu, 23 Nov 2017 10:37:59 +0100
"M.-A. Lemburg"  wrote:
> On 18.11.2017 01:01, Victor Stinner wrote:
> > Hi,
> > 
> > The CPython internals evolved during Python 3.7 cycle. I would like to
> > know if we broke the C API or not.
> > 
> > Nick Coghlan and Eric Snow are working on cleaning up the Python
> > initialization with the "on going" PEP 432:
> > https://www.python.org/dev/peps/pep-0432/
> > 
> > Many global variables used by the "Python runtime" were move to a new
> > single "_PyRuntime" variable (big structure made of sub-structures).
> > See Include/internal/pystate.h.
> > 
> > A side effect of moving variables from random files into header files
> > is that it's not more possible to fully initialize _PyRuntime at
> > "compilation time". For example, previously, it was possible to refer
> > to local C function (functions declared with "static", so only visible
> > in the current file). Now a new "initialization function" is required
> > to must be called.
> > 
> > In short, it means that using the "Python runtime" before it's
> > initialized by _PyRuntime_Initialize() is now likely to crash. For
> > example, calling PyMem_RawMalloc(), before calling
> > _PyRuntime_Initialize(), now calls the function NULL: dereference a
> > NULL pointer, and so immediately crash with a segmentation fault.  
> 
> To prevent a complete crash, would it be possible to initialize
> the struct entries to a generic function (or set of such functions
> with the right signatures), which then issue a message to stderr
> hinting to the missing call to _PyRuntime_Initialize()
> before terminating ?

+1.  This sounds like a good idea.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 559 - built-in noop()

[Nick Coghlan]

On 23 November 2017 at 19:42, Chris Jerdonek 
wrote:

> On Wed, Nov 22, 2017 at 4:32 PM, Victor Stinner 
> wrote:
>
>> Aha, contextlib.nullcontext() was just added, cool!
>>
>
> So is this equivalent to--
>
> @contextmanager
> def yielding(x):
> yield x
>
> I thought we were against adding one-line functions?
>

There's a lot of runtime complexity hiding behind that "@contextmanager"
line, so I'm open to `contextlib` additions that make it possible for
sufficiently common patterns to avoid it. (The explicit class based
nullcontext() implementation is 7 lines, the same as contextlib.closing())

After 7+ years, I'm happy that this one comes up often enough to be worth a
more obvious standard library level answer than we'd previously offered.
https://bugs.python.org/issue10049#msg281556 captures the point where I
really started changing my mind.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nick Coghlan]

On 23 November 2017 at 18:11, Greg Ewing 
wrote:

> Ivan Levkivskyi wrote:
>
>> "People sometimes want to refactor for-loops containing `yield` into a
>> comprehension but that doesn't work (particularly because of the hidden
>> function scope) - lets make it a SyntaxError"
>>
>
> Personally I'd be fine with removing the implicit function
> scope from comprehensions and allowing yield in them, since
> the semantics of that are clear.
>

People keep saying this, but seriously, those semantics aren't clear at all
once you actually start trying to implement it.

Yes, they're obvious in simple cases, but it isn't the simple cases that
are the problem.

Instead, things start getting hard once you're dealing with:

- unpacking to multiple variable names
- nested loops in the comprehension
- lexical closures inside the comprehension (e.g. lambda expressions,
comprehensions inside comprehensions)

Hence the approach we ended up going with for
https://bugs.python.org/issue1660500, which was to use a real function
scope that already handled all of those potential problems in a well
defined way.

Technically we *could* define new answers to all of those situations, but
then we're stuck explaining to everyone what those new behaviours actually
are, and I think that will actually be harder than the status quo, where we
only have to explain why these implicit scopes act much the same way that
"lambda: await expr" and "lambda: yield expr" do.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 11:55, Nick Coghlan  wrote:

> On 23 November 2017 at 18:11, Greg Ewing 
> wrote:
>
>> Ivan Levkivskyi wrote:
>>
>>> "People sometimes want to refactor for-loops containing `yield` into a
>>> comprehension but that doesn't work (particularly because of the hidden
>>> function scope) - lets make it a SyntaxError"
>>>
>>
>> Personally I'd be fine with removing the implicit function
>> scope from comprehensions and allowing yield in them, since
>> the semantics of that are clear.
>>
>
> People keep saying this, but seriously, those semantics aren't clear at
> all once you actually start trying to implement it.
>
>
If Serhiy will implement his idea (emitting for-loop bytecode inside a
try-finally), then I see no problems accepting it as a fix.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 12:38, Ivan Levkivskyi  wrote:

> On 23 November 2017 at 11:55, Nick Coghlan  wrote:
>
>> On 23 November 2017 at 18:11, Greg Ewing 
>> wrote:
>>
>>> Ivan Levkivskyi wrote:
>>>
 "People sometimes want to refactor for-loops containing `yield` into a
 comprehension but that doesn't work (particularly because of the hidden
 function scope) - lets make it a SyntaxError"

>>>
>>> Personally I'd be fine with removing the implicit function
>>> scope from comprehensions and allowing yield in them, since
>>> the semantics of that are clear.
>>>
>>
>> People keep saying this, but seriously, those semantics aren't clear at
>> all once you actually start trying to implement it.
>>
>>
> If Serhiy will implement his idea (emitting for-loop bytecode inside a
> try-finally), then I see no problems accepting it as a fix.
>

Also I think it makes sense to keep discussion in one place, i.e. either
here xor at https://bugs.python.org/issue10544

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 10:50, Paul Moore  wrote:

> On 23 November 2017 at 09:14, Steve Holden  wrote:
> > I would urge developers, in their improvements to the language to support
> > asynchronous programming, to bear in mind that this is (currently) a
> > minority use case. Why the rush to set complex semantics in stone?
>
> +1
>
> Also, given that languages like C# have similar async/await
> functionality, I'd be interested to know how they address questions
> like this. If they have a parallel, we should probably follow it. If
> they don't that would be further indication that no-one has much
> experience of the "best answers" yet, and caution is indicated.
>
>
Keeping this open for indefinite time is also not a good option.
Note that the issue about `yield` in comprehensions
https://bugs.python.org/issue10544 is 7 (seven) years old.
I don't say that we should fix it _right now_, but I think it makes sense
to spend some time and finally resolve it.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Antoine Pitrou]

On Thu, 23 Nov 2017 12:39:46 +0100
Ivan Levkivskyi  wrote:
> 
> Also I think it makes sense to keep discussion in one place, i.e. either
> here xor at https://bugs.python.org/issue10544

The bug tracker can be used for implementation discussions, but general
language design decisions (such as whether to allow or not a certain
construct) should take place on python-dev.

I'm still in favour of deprecating and then disallowing.  Nobody seems
to have presented a real-world use case that is made significantly
easier by trying to "fix" the current behaviour (as opposed to
spelling the loop explicitly).  I do asynchronous programming using
"yield" every day in may job (because of compatibility requirements
with Python 2) and I've never had once the need to write a "yield"
inside a comprehension or generator expression.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Antoine Pitrou]

On Thu, 23 Nov 2017 09:50:27 +
Paul Moore  wrote:
> On 23 November 2017 at 09:14, Steve Holden  wrote:
> > I would urge developers, in their improvements to the language to support
> > asynchronous programming, to bear in mind that this is (currently) a
> > minority use case. Why the rush to set complex semantics in stone?  
> 
> +1
> 
> Also, given that languages like C# have similar async/await
> functionality, I'd be interested to know how they address questions
> like this. If they have a parallel, we should probably follow it. If
> they don't that would be further indication that no-one has much
> experience of the "best answers" yet, and caution is indicated.

This discussion isn't about async/wait or asynchronous programming.

It's about "yield" (which used to be the standard for asynchronous
programming before async/await, but isn't anymore).  The fact that
"await" is now the standard still weakens the case for "yield" inside
comprehensions and generator expressions.

As someone who does asynchronous programming daily using "yield"
(because of compatibility requirements with Python 2), I don't think
I've even tried to use "yield" in a comprehension or generator
expression.  The use case doesn't seem to exist.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 12:49, Antoine Pitrou  wrote:

> On Thu, 23 Nov 2017 12:39:46 +0100
> Ivan Levkivskyi  wrote:
> >
> > Also I think it makes sense to keep discussion in one place, i.e. either
> > here xor at https://bugs.python.org/issue10544
>
> The bug tracker can be used for implementation discussions, but general
> language design decisions (such as whether to allow or not a certain
> construct) should take place on python-dev.
>
> I'm still in favour of deprecating and then disallowing.  Nobody seems
> to have presented a real-world use case that is made significantly
> easier by trying to "fix" the current behaviour (as opposed to
> spelling the loop explicitly).  I do asynchronous programming using
> "yield" every day in may job (because of compatibility requirements
> with Python 2) and I've never had once the need to write a "yield"
> inside a comprehension or generator expression.
>
>
"I don't use it, therefore it is not needed"  is a great argument, thanks.
Lets just forget about two SO questions and dozens people who up-voted it.
Do you use async comprehensions? If not, then we don't need them either.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Antoine Pitrou]

Le 23/11/2017 à 13:01, Ivan Levkivskyi a écrit :
> 
> "I don't use it, therefore it is not needed"  is a great argument, thanks.

This is just a data point.  Some people seem to think that the construct
is useful for asynchronous programming.  In my experience it isn't.
YMMV, etc.

> Lets just forget about two SO questions and dozens people who up-voted it.

Just because someone asks a question doesn't mean they have a pressing
use case.  People are curious and will try constructs just for the sake
it.  Today I looked up the Wikipedia page for Gregorian chant.  Did I
have a use case for it?  No, I was just curious.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Serhiy Storchaka]

23.11.17 13:49, Antoine Pitrou пише:

I'm still in favour of deprecating and then disallowing.


We could disallow it without deprecation. The current behavior 
definitely is wrong, nobody should depend on it. It should be either 
fixed or disallowed.



Nobody seems
to have presented a real-world use case that is made significantly
easier by trying to "fix" the current behaviour (as opposed to
spelling the loop explicitly).  I do asynchronous programming using
"yield" every day in may job (because of compatibility requirements
with Python 2) and I've never had once the need to write a "yield"
inside a comprehension or generator expression.


I used the "yield" statement, but I never used the "yield" expressions. 
And I can't found examples. Could you please present a real-world use 
case for the "yield" (not "yield from") expression?


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 13:11, Paul Moore  wrote:

> On 23 November 2017 at 12:01, Ivan Levkivskyi 
> wrote:
>
> > "I don't use it, therefore it is not needed"  is a great argument,
> thanks.
> > Lets just forget about two SO questions and dozens people who up-voted
> it.
> > Do you use async comprehensions? If not, then we don't need them either.
>
> For those of us trying to keep up with the discussion who don't have
> time to chase the various references, and in the interest of keeping
> the discussion in one place, can you summarise the real-world use
> cases from the SO questions here? (I assume they are real world cases,
> and not just theoretical questions)
>

OK, here are the links:

https://stackoverflow.com/questions/45190729/differences-between-generator-comprehension-expressions
https://stackoverflow.com/questions/29334054/why-am-i-getting-different-results-when-using-a-list-comprehension-with-coroutin
https://bugs.python.org/issue10544
https://bugs.python.org/issue3267

In all four cases pattern is the same, people were trying to refactor
something like this:

def f():
res = []
for x in y:
r = yield x
res.append(r)
return res

into something like this:

def f():
return [(yield x) for x in y]

My understanding is that none of the case is _pressing_, since they all
start with a for-loop, but
following this logic comprehensions themselves are not needed. Nevertheless
people use them because they like it.
The problem in all four cases is that they got hard to debug problem, since
calling `f()` returns a generator,
just not the one they would expect.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Antoine Pitrou]

On Thu, 23 Nov 2017 14:17:32 +0200
Serhiy Storchaka  wrote:
> 
> I used the "yield" statement, but I never used the "yield" expressions. 
> And I can't found examples. Could you please present a real-world use 
> case for the "yield" (not "yield from") expression?

Of course I can.  "yield" expressions are important for writing
Python 2-compatible asynchronous code while avoiding callback hell:

See e.g. http://www.tornadoweb.org/en/stable/gen.html
or https://jdb.github.io/concurrent/smartpython.html

There are tons of real-world code written using this scheme (as opposed
to almost no real-world code, even Python 2-only, using "yield" in
comprehensions or generation expressions).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 23 November 2017 at 12:01, Ivan Levkivskyi  wrote:

> "I don't use it, therefore it is not needed"  is a great argument, thanks.
> Lets just forget about two SO questions and dozens people who up-voted it.
> Do you use async comprehensions? If not, then we don't need them either.

For those of us trying to keep up with the discussion who don't have
time to chase the various references, and in the interest of keeping
the discussion in one place, can you summarise the real-world use
cases from the SO questions here? (I assume they are real world cases,
and not just theoretical questions)

Thanks,
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 13:30, Antoine Pitrou  wrote:

> On Thu, 23 Nov 2017 14:17:32 +0200
> Serhiy Storchaka  wrote:
> >
> > I used the "yield" statement, but I never used the "yield" expressions.
> > And I can't found examples. Could you please present a real-world use
> > case for the "yield" (not "yield from") expression?
>
> Of course I can.  "yield" expressions are important for writing
> Python 2-compatible asynchronous code while avoiding callback hell:
>
> See e.g. http://www.tornadoweb.org/en/stable/gen.html
>
> 


Great, so I open this page and see this code:

results = []
for future in list_of_futures:
results.append(yield future)

Interesting, why don't they use a comprehension for this and instead need
to invent a whole `tornado.gen.multi` function?

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 23 November 2017 at 12:28, Ivan Levkivskyi  wrote:
> On 23 November 2017 at 13:11, Paul Moore  wrote:
>>
>> On 23 November 2017 at 12:01, Ivan Levkivskyi 
>> wrote:
>>
>> > "I don't use it, therefore it is not needed"  is a great argument,
>> > thanks.
>> > Lets just forget about two SO questions and dozens people who up-voted
>> > it.
>> > Do you use async comprehensions? If not, then we don't need them either.
>>
>> For those of us trying to keep up with the discussion who don't have
>> time to chase the various references, and in the interest of keeping
>> the discussion in one place, can you summarise the real-world use
>> cases from the SO questions here? (I assume they are real world cases,
>> and not just theoretical questions)
>
>
> OK, here are the links:
>
> https://stackoverflow.com/questions/45190729/differences-between-generator-comprehension-expressions
> https://stackoverflow.com/questions/29334054/why-am-i-getting-different-results-when-using-a-list-comprehension-with-coroutin
> https://bugs.python.org/issue10544
> https://bugs.python.org/issue3267
>
> In all four cases pattern is the same, people were trying to refactor
> something like this:
>
> def f():
> res = []
> for x in y:
> r = yield x
> res.append(r)
> return res
>
> into something like this:
>
> def f():
> return [(yield x) for x in y]
>
> My understanding is that none of the case is _pressing_, since they all
> start with a for-loop, but
> following this logic comprehensions themselves are not needed. Nevertheless
> people use them because they like it.
> The problem in all four cases is that they got hard to debug problem, since
> calling `f()` returns a generator,
> just not the one they would expect.

OK, thanks. I can see why someone would want to do this. However, it
seems to me that the problem (a hard to debug error) could be solved
by disallowing yield in comprehensions and generator expressions
(giving an *easy* to debug error). I don't think the above is a
compelling argument that we have to support the one-line form. If
there was a non-trivial body of actual user code that uses the loop
form, which would be substantially improved by being able to use
comprehensions, that would be different. To put it another way, the
example you gave is still artificial. The second link is a real use
case, but as you say seems to be more a question about "why did this
not work as I expected" which could be solved with a SyntaxError
saying "yield expression not allowed in comprehensions".

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Antoine Pitrou]

Le 23/11/2017 à 13:42, Ivan Levkivskyi a écrit :
> 
> Great, so I open this page and see this code:
> 
> results = []
> for future in list_of_futures:
>     results.append(yield future)
> 
> Interesting, why don't they use a comprehension for this and instead
> need to invent a whole `tornado.gen.multi` function?

1) because it schedules the yielded coroutines in parallel (the "for"
loop isn't strictly equivalent, as AFAIU it would schedule the
coroutines serially)

2) because it accepts an optional argument to quiet some exceptions

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 23 November 2017 at 12:42, Ivan Levkivskyi  wrote:
>> See e.g. http://www.tornadoweb.org/en/stable/gen.html
>>
>
> Great, so I open this page and see this code:
>
> results = []
> for future in list_of_futures:
> results.append(yield future)
>
> Interesting, why don't they use a comprehension for this and instead need to
> invent a whole `tornado.gen.multi` function?

Because yield expressions in comprehensions are difficult to
understand, and the loop form is easy to understand? :-) (Certainly I
didn't find the explanation in that page confusing, I don't know if
I'd have found a comprehension form confusing, but I suspect I might
have...)

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Serhiy Storchaka]

23.11.17 14:30, Antoine Pitrou пише:

On Thu, 23 Nov 2017 14:17:32 +0200
Serhiy Storchaka  wrote:


I used the "yield" statement, but I never used the "yield" expressions.
And I can't found examples. Could you please present a real-world use
case for the "yield" (not "yield from") expression?


Of course I can.  "yield" expressions are important for writing
Python 2-compatible asynchronous code while avoiding callback hell:

See e.g. http://www.tornadoweb.org/en/stable/gen.html
or https://jdb.github.io/concurrent/smartpython.html

There are tons of real-world code written using this scheme (as opposed
to almost no real-world code, even Python 2-only, using "yield" in
comprehensions or generation expressions).


Thank you. The tornado examples contain the following equivalence code 
for `results = yield multi(list_of_futures)`:


results = []
for future in list_of_futures:
results.append(yield future)

Couldn't this by written as `results = [(yield future) for future in 
list_of_futures]`?


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 13:45, Paul Moore  wrote:

> On 23 November 2017 at 12:28, Ivan Levkivskyi 
> wrote:
> > On 23 November 2017 at 13:11, Paul Moore  wrote:
> >>
> >> On 23 November 2017 at 12:01, Ivan Levkivskyi 
> >> wrote:
> >>
> >> > "I don't use it, therefore it is not needed"  is a great argument,
> >> > thanks.
> >> > Lets just forget about two SO questions and dozens people who up-voted
> >> > it.
> >> > Do you use async comprehensions? If not, then we don't need them
> either.
> >>
> >> For those of us trying to keep up with the discussion who don't have
> >> time to chase the various references, and in the interest of keeping
> >> the discussion in one place, can you summarise the real-world use
> >> cases from the SO questions here? (I assume they are real world cases,
> >> and not just theoretical questions)
> [...]
> >
> > My understanding is that none of the case is _pressing_, since they all
> > start with a for-loop, but
> > following this logic comprehensions themselves are not needed.
> Nevertheless
> > people use them because they like it.
> > The problem in all four cases is that they got hard to debug problem,
> since
> > calling `f()` returns a generator,
> > just not the one they would expect.
>
> OK, thanks. I can see why someone would want to do this. However, it
> seems to me that the problem (a hard to debug error) could be solved
> by disallowing yield in comprehensions and generator expressions
> (giving an *easy* to debug error). I don't think the above is a
> compelling argument that we have to support the one-line form. If
> there was a non-trivial body of actual user code that uses the loop
> form, which would be substantially improved by being able to use
> comprehensions, that would be different. To put it another way, the
> example you gave is still artificial. The second link is a real use
> case, but as you say seems to be more a question about "why did this
> not work as I expected" which could be solved with a SyntaxError
> saying "yield expression not allowed in comprehensions".
>

The level of "artificialness" is quite subjective, this is rather matter of
taste (see the tornado example).
Let us forget for a moment about other problems and focus on this one: list
comprehension is currently not equivalent to a for-loop.
There are two options:
- Fix this, i.e. make comprehension equivalent to a for-loop even in edge
cases (Serhiy seems ready to do this)
- Prohibit all cases when they are not equivalent

I still prefer option one. But I see your point, option two is also an
acceptable fix.
Note that there were not so many situations when some code became
SyntaxError later.
I don't see why this particular case qualifies for such a radical measure
as an exception to syntactic rules,
instead of just fixing it (sorry Nick :-)

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Antoine Pitrou]

On Thu, 23 Nov 2017 14:54:27 +0200
Serhiy Storchaka  wrote:

> 23.11.17 14:30, Antoine Pitrou пише:
> > On Thu, 23 Nov 2017 14:17:32 +0200
> > Serhiy Storchaka  wrote:  
> >>
> >> I used the "yield" statement, but I never used the "yield" expressions.
> >> And I can't found examples. Could you please present a real-world use
> >> case for the "yield" (not "yield from") expression?  
> > 
> > Of course I can.  "yield" expressions are important for writing
> > Python 2-compatible asynchronous code while avoiding callback hell:
> > 
> > See e.g. http://www.tornadoweb.org/en/stable/gen.html
> > or https://jdb.github.io/concurrent/smartpython.html
> > 
> > There are tons of real-world code written using this scheme (as opposed
> > to almost no real-world code, even Python 2-only, using "yield" in
> > comprehensions or generation expressions).  
> 
> Thank you. The tornado examples contain the following equivalence code 
> for `results = yield multi(list_of_futures)`:
> 
>  results = []
>  for future in list_of_futures:
>  results.append(yield future)
> 
> Couldn't this by written as `results = [(yield future) for future in 
> list_of_futures]`?

See my answer to Ivan above.  The code isn't actually equivalent :-)
But, yes, this construct *could* be useful if you wanted to schedule
futures serially (as opposed to in parallel).

However, since it doesn't work on Python 3.x, and the main reason to
use "yield" coroutines (even with Tornado) instead of "async/await" is
for compatibility, solving the "yield in a comprehension" problem in
3.7 wouldn't make things any better IMO.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 23 November 2017 at 13:04, Ivan Levkivskyi  wrote:
> Let us forget for a moment about other problems and focus on this one: list
> comprehension is currently not equivalent to a for-loop.
> There are two options:
> - Fix this, i.e. make comprehension equivalent to a for-loop even in edge
> cases (Serhiy seems ready to do this)
> - Prohibit all cases when they are not equivalent
>
> I still prefer option one. But I see your point, option two is also an
> acceptable fix.
> Note that there were not so many situations when some code became
> SyntaxError later.
> I don't see why this particular case qualifies for such a radical measure as
> an exception to syntactic rules,
> instead of just fixing it (sorry Nick :-)

My main concern is that comprehension is not equivalent to a for loop
for a specific reason - the scope issue. Has anyone looked back at the
original discussions to confirm *why* a function was used?

My recollection:

>>> i = 1
>>> a = [i for i in (1,2,3)]
>>> print(i)
1

Serihy's approach (and your described expansion) would have print(i)
return NameError.

So - do we actually have a proposal to avoid the implied function that
*doesn't* break this example? I'm pretty sure this was a real-life
issue at the time we switched to the current implementation.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 15:21, Paul Moore  wrote:

> On 23 November 2017 at 13:04, Ivan Levkivskyi 
> wrote:
> > Let us forget for a moment about other problems and focus on this one:
> list
> > comprehension is currently not equivalent to a for-loop.
> > There are two options:
> > - Fix this, i.e. make comprehension equivalent to a for-loop even in edge
> > cases (Serhiy seems ready to do this)
> > - Prohibit all cases when they are not equivalent
> >
> > I still prefer option one. But I see your point, option two is also an
> > acceptable fix.
> > Note that there were not so many situations when some code became
> > SyntaxError later.
> > I don't see why this particular case qualifies for such a radical
> measure as
> > an exception to syntactic rules,
> > instead of just fixing it (sorry Nick :-)
>
> My main concern is that comprehension is not equivalent to a for loop
> for a specific reason - the scope issue. Has anyone looked back at the
> original discussions to confirm *why* a function was used?
>
> My recollection:
>
> >>> i = 1
> >>> a = [i for i in (1,2,3)]
> >>> print(i)
> 1
>
> Serihy's approach (and your described expansion) would have print(i)
> return NameError.
>

Absolutely no, it will still print 1. The internal implementation will use
unique ids internally (see https://bugs.python.org/issue10544 for details).

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Chris Angelico]

On Fri, Nov 24, 2017 at 1:21 AM, Paul Moore  wrote:
> On 23 November 2017 at 13:04, Ivan Levkivskyi  wrote:
>> Let us forget for a moment about other problems and focus on this one: list
>> comprehension is currently not equivalent to a for-loop.
>> There are two options:
>> - Fix this, i.e. make comprehension equivalent to a for-loop even in edge
>> cases (Serhiy seems ready to do this)
>> - Prohibit all cases when they are not equivalent
>>
>> I still prefer option one. But I see your point, option two is also an
>> acceptable fix.
>> Note that there were not so many situations when some code became
>> SyntaxError later.
>> I don't see why this particular case qualifies for such a radical measure as
>> an exception to syntactic rules,
>> instead of just fixing it (sorry Nick :-)
>
> My main concern is that comprehension is not equivalent to a for loop
> for a specific reason - the scope issue. Has anyone looked back at the
> original discussions to confirm *why* a function was used?
>
> My recollection:
>
 i = 1
 a = [i for i in (1,2,3)]
 print(i)
> 1
>
> Serihy's approach (and your described expansion) would have print(i)
> return NameError.
>
> So - do we actually have a proposal to avoid the implied function that
> *doesn't* break this example? I'm pretty sure this was a real-life
> issue at the time we switched to the current implementation.

A while back I had a POC patch that made "with EXPR as NAME:" create a
new subscope with NAME in it, such that the variable actually
disappeared at the end of the 'with' block. Should I try to track that
down and adapt the technique to comprehensions? The subscope shadows
names exactly the way a nested function does, but it's all within the
same function.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 23 November 2017 at 14:24, Ivan Levkivskyi  wrote:
>> My main concern is that comprehension is not equivalent to a for loop
>> for a specific reason - the scope issue. Has anyone looked back at the
>> original discussions to confirm *why* a function was used?
>>
>> My recollection:
>>
>> >>> i = 1
>> >>> a = [i for i in (1,2,3)]
>> >>> print(i)
>> 1
>>
>> Serihy's approach (and your described expansion) would have print(i)
>> return NameError.
>
>
> Absolutely no, it will still print 1. The internal implementation will use
> unique ids internally (see https://bugs.python.org/issue10544 for details).
>

Ok, cool. My main point still applies though - has anyone confirmed
why a function scope was considered necessary at the time of the
original implementation, but it's apparently not now? I'm pretty sure
it was a deliberate choice, not an accident.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Serhiy Storchaka]

23.11.17 16:30, Paul Moore пише:

Ok, cool. My main point still applies though - has anyone confirmed
why a function scope was considered necessary at the time of the
original implementation, but it's apparently not now? I'm pretty sure
it was a deliberate choice, not an accident.


The implementation with an intermediate one-time function is just 
simpler. The one of purposes of Python 3 was simplifying the 
implementation, even at the cost of some performance penalty. I'm pretty 
sure the corner case with "yield" was just missed.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 15:30, Paul Moore  wrote:

> On 23 November 2017 at 14:24, Ivan Levkivskyi 
> wrote:
> >> My main concern is that comprehension is not equivalent to a for loop
> >> for a specific reason - the scope issue. Has anyone looked back at the
> >> original discussions to confirm *why* a function was used?
> >>
> >> My recollection:
> >>
> >> >>> i = 1
> >> >>> a = [i for i in (1,2,3)]
> >> >>> print(i)
> >> 1
> >>
> >> Serihy's approach (and your described expansion) would have print(i)
> >> return NameError.
> >
> >
> > Absolutely no, it will still print 1. The internal implementation will
> use
> > unique ids internally (see https://bugs.python.org/issue10544 for
> details).
> >
>
> Ok, cool. My main point still applies though - has anyone confirmed
> why a function scope was considered necessary at the time of the
> original implementation, but it's apparently not now? I'm pretty sure
> it was a deliberate choice, not an accident.


>From what Nick explained on b.p.o. I understand that this is closer to the
"accident" definition.
Also the original issue https://bugs.python.org/issue1660500 doesn't have
any discussion of the implementation _strategy_.
So I tried to dig the mailing list, in the latest Guido's message I have
found
https://mail.python.org/pipermail/python-3000/2006-December/005218.html
he still likes the idea of unique hidden ids (like Serhiy proposes now) and
no function scopes. After that there is Nick's message
https://mail.python.org/pipermail/python-3000/2006-December/005229.html
where he says that he still likes pseudo-scopes more.
Then I lost the track of discussion.

It may well be Nick's intentional decision (and it has its merits) but I am
not sure it was a conscious consensus.
Nick could probably add more. Also I propose to wait and see when Serhiy
will show us his complete implementation.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ethan Furman]

On 11/22/2017 11:51 PM, Sven R. Kunze wrote:


A "yield" within a comprehension is like a "return" in a comprehension. It 
makes no sense at all.
Also a "yield" and a "return with value" is also rarely seen.

Comprehensions build new objects, they are not for control flow, IMO.


+1

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 559 - built-in noop()

[Barry Warsaw]

On Nov 22, 2017, at 19:32, Victor Stinner  wrote:
> 
> Aha, contextlib.nullcontext() was just added, cool!

So, if I rewrite PEP 559 in terms of decorators it won’t get rejected?

from functools import wraps

def noop(func):
@wraps(func)
def wrapper(*args, **kws):
return None
return wrapper


@noop
def do_something_important(x, y, z):
return blah_blah_blah(x, y, z)


print(do_something_important(1, 2, z=3))

Cheers?
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 23 November 2017 at 15:37, Ethan Furman  wrote:
> On 11/22/2017 11:51 PM, Sven R. Kunze wrote:
>
>> A "yield" within a comprehension is like a "return" in a comprehension. It
>> makes no sense at all.
>> Also a "yield" and a "return with value" is also rarely seen.
>>
>> Comprehensions build new objects, they are not for control flow, IMO.
>
>
> +1

That isn't a principle I've seen described explicitly before, but I
agree it makes sense to me. So +1 here as well.

And yes, I know this might seem to contradict my position that
comprehensions translate to loops, but remember that the translation
is a mental model, not an exact definition (to my way of thinking).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ethan Furman]

On 11/23/2017 04:01 AM, Ivan Levkivskyi wrote:


Lets just forget about two SO questions and dozens people who up-voted it.


Questions/answers are routinely up-voted because they are well-written and/or informative, not just because somebody had 
a need for it or a use for the answer.


The SO question linked in the OP wasn't even about using the yield/yield-from in real code, just about timings and 
differences in byte code and whether or not they "worked" (no real-world use-case).


Hardly a clarion call for a fix.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Guido van Rossum]

This thread is still going over the speed limit. Don't commit anything
without my explicit approval.

I know one thing for sure. The choice to make all comprehensions functions
was quite intentional (even though alternatives were also discussed) and
the extra scope is now part of the language definition. It can't be removed
as a "bug fix". One thing that can be done without a PEP is making yield
inside a comprehension a syntax error, since I don't think *that* was
considered when the function scopes were introduced.

A problem with dropping the "function-ness" of the comprehension by
renaming the variables (as Ivan/Serhiy's plan seems to be?) would be what
does it look like in the debugger -- can I still step through the loop and
print the values of expressions? Or do I have to know the "renamed" names?

And my mind boggles when considering a generator expression containing
yield that is returned from a function. I tried this and cannot say I
expected the outcome:

  def f():
  return ((yield i) for i in range(3))
  print(list(f()))

In both Python 2 and Python 3 this prints

  [0, None, 1, None, 2, None]

Even if there's a totally logical explanation for that, I still don't like
it, and I think yield in a comprehension should be banned. From this it
follows that we should also simply ban yield  from comprehensions.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Brett Cannon]

I've now ended up in Guido's boat of needing a summary since I think this
thread has grown to cover whether yield should be allowed in
comprehensions, something about await in comprehensions, and now about
leaking the loop variable (or some implementation detail). IOW there seems
to be 3 separate discussions going on in a single thread.

Any chance we can get some clear threads going for each independent topic
to make it easier to discuss? I do have an opinion on all three topics (if
I understand the topics accurately 😉), but I don't want to contribute to
the confusion of the discussion by mentioning them here.

On Thu, Nov 23, 2017, 06:50 Ivan Levkivskyi,  wrote:

> On 23 November 2017 at 15:30, Paul Moore  wrote:
>
>> On 23 November 2017 at 14:24, Ivan Levkivskyi 
>> wrote:
>> >> My main concern is that comprehension is not equivalent to a for loop
>> >> for a specific reason - the scope issue. Has anyone looked back at the
>> >> original discussions to confirm *why* a function was used?
>> >>
>> >> My recollection:
>> >>
>> >> >>> i = 1
>> >> >>> a = [i for i in (1,2,3)]
>> >> >>> print(i)
>> >> 1
>> >>
>> >> Serihy's approach (and your described expansion) would have print(i)
>> >> return NameError.
>> >
>> >
>> > Absolutely no, it will still print 1. The internal implementation will
>> use
>> > unique ids internally (see https://bugs.python.org/issue10544 for
>> details).
>> >
>>
>> Ok, cool. My main point still applies though - has anyone confirmed
>> why a function scope was considered necessary at the time of the
>> original implementation, but it's apparently not now? I'm pretty sure
>> it was a deliberate choice, not an accident.
>
>
> From what Nick explained on b.p.o. I understand that this is closer to the
> "accident" definition.
> Also the original issue https://bugs.python.org/issue1660500 doesn't have
> any discussion of the implementation _strategy_.
> So I tried to dig the mailing list, in the latest Guido's message I have
> found
> https://mail.python.org/pipermail/python-3000/2006-December/005218.html
> he still likes the idea of unique hidden ids (like Serhiy proposes now)
> and no function scopes. After that there is Nick's message
> https://mail.python.org/pipermail/python-3000/2006-December/005229.html
> where he says that he still likes pseudo-scopes more.
> Then I lost the track of discussion.
>
> It may well be Nick's intentional decision (and it has its merits) but I
> am not sure it was a conscious consensus.
> Nick could probably add more. Also I propose to wait and see when Serhiy
> will show us his complete implementation.
>
> --
> Ivan
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Serhiy Storchaka]

23.11.17 18:08, Guido van Rossum пише:
This thread is still going over the speed limit. Don't commit anything 
without my explicit approval.


I'm not going to write a single line of code while the decision about 
this issue is not made. This is not easy issue.


A problem with dropping the "function-ness" of the comprehension by 
renaming the variables (as Ivan/Serhiy's plan seems to be?) would be 
what does it look like in the debugger -- can I still step through the 
loop and print the values of expressions? Or do I have to know the 
"renamed" names?


Does the debugger supports stepping inside a comprehension? Isn't the 
whole comprehension a single logical line? It this is supported the the 
debugger should see a local variable with a strange name ".0". This is a 
leaked implementation detail. It wouldn't be surprised if details of 
other implementation will be leaked too in the debugger. The debugger 
can unmangle the inner variable names and hide local variables in the 
outer scope.


And my mind boggles when considering a generator expression containing 
yield that is returned from a function. I tried this and cannot say I 
expected the outcome:


   def f():
       return ((yield i) for i in range(3))
   print(list(f()))

In both Python 2 and Python 3 this prints

   [0, None, 1, None, 2, None]

Even if there's a totally logical explanation for that, I still don't 
like it, and I think yield in a comprehension should be banned. From 
this it follows that we should also simply ban yield  from comprehensions.


This behavior doesn't look correct to me and Ivan. Ivan explained that 
this function should be rough equivalent to


   def f():
   t = [(yield i) for i in range(3)]
   return (x for x in t)

which should be equivalent to

   def f():
   t = []
   for i in range(3):
  t.append((yield i))
   return (x for x in t)

and list(f()) should be [1, 2, 3].

But while I know how to fix yield in comprehensions (just inline the 
code with some additions), I have no ideas how to fix yield in 
generators. Me and Ivan agreed that SyntaxError in this case is better 
than the current behavior. Maybe we will find how to implement the 
expected behavior.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Guido van Rossum]

On Thu, Nov 23, 2017 at 9:06 AM, Serhiy Storchaka 
wrote:

> 23.11.17 18:08, Guido van Rossum пише:
>
>> This thread is still going over the speed limit. Don't commit anything
>> without my explicit approval.
>>
>
> I'm not going to write a single line of code while the decision about this
> issue is not made. This is not easy issue.
>

OK, I think Ivan at some point said he was waiting how your fix would look.


> A problem with dropping the "function-ness" of the comprehension by
>> renaming the variables (as Ivan/Serhiy's plan seems to be?) would be what
>> does it look like in the debugger -- can I still step through the loop and
>> print the values of expressions? Or do I have to know the "renamed" names?
>>
>
> Does the debugger supports stepping inside a comprehension? Isn't the
> whole comprehension a single logical line? It this is supported the the
> debugger should see a local variable with a strange name ".0". This is a
> leaked implementation detail. It wouldn't be surprised if details of other
> implementation will be leaked too in the debugger. The debugger can
> unmangle the inner variable names and hide local variables in the outer
> scope.
>

The debugger does stop at each iteration. It does see a local named ".0"
(which of course cannot be printed directly, only using vars()). I suppose
there currently is no way for the debugger to map the variable names to
what they are named in the source, right? (If you think this is a useful
thing to add, please open a separate issue.)


> And my mind boggles when considering a generator expression containing
>> yield that is returned from a function. I tried this and cannot say I
>> expected the outcome:
>>
>>def f():
>>return ((yield i) for i in range(3))
>>print(list(f()))
>>
>> In both Python 2 and Python 3 this prints
>>
>>[0, None, 1, None, 2, None]
>>
>> Even if there's a totally logical explanation for that, I still don't
>> like it, and I think yield in a comprehension should be banned. From this
>> it follows that we should also simply ban yield  from comprehensions.
>>
>
> This behavior doesn't look correct to me and Ivan. Ivan explained that
> this function should be rough equivalent to
>
>def f():
>t = [(yield i) for i in range(3)]
>return (x for x in t)
>
> which should be equivalent to
>
>def f():
>t = []
>for i in range(3):
>   t.append((yield i))
>return (x for x in t)
>
> and list(f()) should be [1, 2, 3].
>
> But while I know how to fix yield in comprehensions (just inline the code
> with some additions), I have no ideas how to fix yield in generators. Me
> and Ivan agreed that SyntaxError in this case is better than the current
> behavior. Maybe we will find how to implement the expected behavior.
>

I doubt that anyone who actually wrote that had a valid expectation for it
-- most likely they were coding by random modification or it was the result
of a lack of understanding of generator expressions in general. I think the
syntax error is the best option here.

I also think we should all stop posting for 24 hours as tempers have gotten
heated.

Happy Thanksgiving!

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Paul Moore wrote:

has anyone confirmed
why a function scope was considered necessary at the time of the
original implementation, but it's apparently not now?


At the time I got the impression that nobody wanted to
spend the time necessary to design and implement a subscope
mechanism. What's changed is that we now have someone
offering to do that.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Serhiy Storchaka wrote:
Ivan explained that 
this function should be rough equivalent to


   def f():
   t = [(yield i) for i in range(3)]
   return (x for x in t)


This seems useless to me. It turns a lazy iterator
into an eager one, which is a gross violation of the
author's intent in using a generator expression.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Guido van Rossum wrote:


The debugger does stop at each iteration. It does see a local named ".0" 
I suppose there currently is no way for the debugger to map the variable 
names to what they are named in the source, right?


If the hidden local were named "a.0" where "a" is the original
name, mapping it back would be easy. It would also be easily
understood even if it weren't mapped back.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Guido van Rossum wrote:
the extra scope is now part of the language definition. 
It can't be removed as a "bug fix".


Does anyone actually rely on the scope-ness of comprehensions
in any way other than the fact that it prevents local variable
leakage?

If not, probably nobody would notice if it were changed back.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python initialization and embedded Python

[Victor Stinner]

Hi,

We are close to the 3.7a3 release and the bug is not fixed yet. I
propose to revert the changes on memory allocators right now, and take
time to design a proper fix which will respect all constraints.

https://github.com/python/cpython/pull/4532

Today, someone came to me on IRC to complain that calling
Py_DecodeLocale() does now crash on Python 3.7. He is doing tests to
embed Python on Android. Later he asks me about
PyImport_AppendInittab(), but I don't know this function. He told me
that it does crash in PyMem_Realloc()... But PyImport_AppendInittab()
must be called before Py_Initialize()...

It confirms that Python is embedded and that the C API is used before
Py_Initialize().

We don't know yet exactly how the the C API is used, which functions
are called before Py_Initialize(). Moreover, PEP 432 implementation is
still incomplete, and calling _PyRuntime_Initialize() is just not
possible, since it's a private API which is not exported...

Victor

2017-11-18 1:01 GMT+01:00 Victor Stinner :
> Hi,
>
> The CPython internals evolved during Python 3.7 cycle. I would like to
> know if we broke the C API or not.
>
> Nick Coghlan and Eric Snow are working on cleaning up the Python
> initialization with the "on going" PEP 432:
> https://www.python.org/dev/peps/pep-0432/
>
> Many global variables used by the "Python runtime" were move to a new
> single "_PyRuntime" variable (big structure made of sub-structures).
> See Include/internal/pystate.h.
>
> A side effect of moving variables from random files into header files
> is that it's not more possible to fully initialize _PyRuntime at
> "compilation time". For example, previously, it was possible to refer
> to local C function (functions declared with "static", so only visible
> in the current file). Now a new "initialization function" is required
> to must be called.
>
> In short, it means that using the "Python runtime" before it's
> initialized by _PyRuntime_Initialize() is now likely to crash. For
> example, calling PyMem_RawMalloc(), before calling
> _PyRuntime_Initialize(), now calls the function NULL: dereference a
> NULL pointer, and so immediately crash with a segmentation fault.
>
> I'm writing this email to ask if this change is an issue or not to
> embedded Python and the Python C API. Is it still possible to call
> "all" functions of the C API before calling Py_Initialize()?
>
> I was bitten by the bug while reworking the Py_Main() function to
> split it into subfunctions and cleanup the code to handle the command
> line arguments and environment variables. I fixed the issue in main()
> by calling _PyRuntime_Initialize() as soon as possible: it's now the
> first instruction of main() :-) (See Programs/python.c)
>
> To give a more concrete example: Py_DecodeLocale() is the recommanded
> function to decode bytes from the operating system, but this function
> calls PyMem_RawMalloc() which does crash before
> _PyRuntime_Initialize() is called. Is Py_DecodeLocale() used to
> initialize Python?
>
> For example, "void Py_SetProgramName(wchar_t *);" expects a text
> string, whereas main() gives argv as bytes. Calling
> Py_SetProgramName() from argv requires to decode bytes... So use
> Py_DecodeLocale()...
>
> Should we do something in Py_DecodeLocale()? Maybe crash if
> _PyRuntime_Initialize() wasn't called yet?
>
> Maybe, the minimum change is to expose _PyRuntime_Initialize() in the
> public C API?
>
> Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nick Coghlan]

On 23 November 2017 at 23:04, Ivan Levkivskyi  wrote:

> I don't see why this particular case qualifies for such a radical measure
> as an exception to syntactic rules,
> instead of just fixing it (sorry Nick :-)
>

I've posted in more detail about this to the issue tracker, but the
argument here is: because making it behave differently from the way it does
now while still hiding the loop iteration variable potentially requires
even more radical revisions to the lexical scoping rules :)

If somebody can come up with a clever trick to allow yield inside a
comprehension to jump levels in a relatively intuitive way, that would
actually be genuinely cool, but the lexical scoping rules mean it's
trickier than it sounds.

Now that I frame the question that way, though, I'm also remembering that
we didn't have "yield from" yet when I wrote the current comprehension
implementation, and given that, it may be as simple as having an explicit
yield expression in a comprehension imply delegation to a subgenerator.

If we went down that path, then a list comprehension like the following:

results = [(yield future) for future in list_of_futures]

might be compiled as being equivalent to:

def __listcomp_generator(iterable):
result = []
for future in iterable:
result.append((yield future))
return result

results = yield from _listcomp_generator(list_of_futures)

The only difference between the current comprehension code and this idea is
"an explicit yield expression in a comprehension implies the use of 'yield
from' when calling the nested function".

For generator expressions, the adjustment would need to be slightly
different: for those, we'd either need to prohibit yield expressions, or
else say that if there's an explicit yield expression present anywhere,
then we drop the otherwise implied yield expression.

If we went down the latter path, then:

gen = ((yield future) for future in list_of_futures)

and:

gen = (future for future in list_of_futures)

would be two different ways of writing the same thing.

The pay-off for allowing it would be that you could write things like:

gen = (f(yield future) for future in list_of_futures)

as a shorthand equivalent to:

def gen(list_of_futures=list_of_futures):
for future in list_of_futures:
f(yield future)

(Right now, you instead get "yield f(yield future)" as the innermost
statement, which probably isn't what you wanted)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 559 - built-in noop()

[Nick Coghlan]

On 24 November 2017 at 01:49, Barry Warsaw  wrote:

> On Nov 22, 2017, at 19:32, Victor Stinner 
> wrote:
> >
> > Aha, contextlib.nullcontext() was just added, cool!
>
> So, if I rewrite PEP 559 in terms of decorators it won’t get rejected?
>

The conceptual delta between knowing how to call "noop()" and how to write
"def noop(): pass" is just a *teensy* bit smaller than that between knowing
how to use:

with nullcontext(value) as var:
...

and how to write:

@contextlib.contextmanager
def nullcontext(enter_result):
yield enter_result

or:

class nullcontext(object):
def __init__(self, enter_result):
self.enter_result = enter_result
def __enter__(self):
return self.enter_result
def __exit__(self, *args):
pass

So the deciding line for me was "Should people need to know how to write
their own context managers in order to have access to a null context
manager?", and I eventually decided the right answer was "No", since the
context management protocol is actually reasonably tricky conceptually, and
even the simplest version still requires knowing how to use decorators and
generators (as well as knowing which specific decorator to use).

The conceptual step between calling and writing functions is much smaller,
and defining your own re-usable functions is a more fundamental Python
skill than defining your own context managers.

And while I assume you were mostly joking, the idea of a
"@functools.stub_call(result=None)" decorator to temporarily replace an
otherwise expensive function call could be a genuinely interesting
primitive. `unittest.mock` and other testing libraries already have a bunch
of tools along those lines, so the principle at work there would be pattern
extraction based on things people already do for themselves.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python initialization and embedded Python

[Nick Coghlan]

On 24 November 2017 at 09:19, Victor Stinner 
wrote:

> Hi,
>
> We are close to the 3.7a3 release and the bug is not fixed yet. I
> propose to revert the changes on memory allocators right now, and take
> time to design a proper fix which will respect all constraints.
>
> https://github.com/python/cpython/pull/4532
>
> Today, someone came to me on IRC to complain that calling
> Py_DecodeLocale() does now crash on Python 3.7. He is doing tests to
> embed Python on Android. Later he asks me about
> PyImport_AppendInittab(), but I don't know this function. He told me
> that it does crash in PyMem_Realloc()... But PyImport_AppendInittab()
> must be called before Py_Initialize()...
>
> It confirms that Python is embedded and that the C API is used before
> Py_Initialize().
>
> We don't know yet exactly how the the C API is used, which functions
> are called before Py_Initialize().


We do note some of them explicitly at
https://docs.python.org/3/c-api/init.html (search for "before Py").

What we've been missing is a test case that ensures
https://docs.python.org/3/extending/embedding.html#very-high-level-embedding
actually works reliably (hence how we managed to break it by way of the
internal state management refactoring).

Once that core regression has been fixed, we can review the docs and the
test suite and come up with:

- a consolidated list of *all* the APIs that can safely be called before
Py_Initialize
- one or more new or updated test cases to ensure that any not yet tested
pre-initialization APIs actually work as intended


> Moreover, PEP 432 implementation is
> still incomplete, and calling _PyRuntime_Initialize() is just not
> possible, since it's a private API which is not exported...
>

Even after we reach the point of exposing the more fine-grained
initialisation API (which I'm now thinking we may be able to do for 3.8
given Eric & Victor's work on it for 3.7), we're still going to have to
ensure the existing configuration API keeps working as expected.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python initialization and embedded Python

[Glenn Linderman]

On 11/23/2017 5:31 PM, Nick Coghlan wrote:
- a consolidated list of *all* the APIs that can safely be called 
before Py_Initialize
So it is interesting to know that list, of course, but the ones that are 
to be supported and documented might be a smaller list. Or might not.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python initialization and embedded Python

[Nick Coghlan]

On 24 November 2017 at 12:21, Glenn Linderman  wrote:

> On 11/23/2017 5:31 PM, Nick Coghlan wrote:
>
> - a consolidated list of *all* the APIs that can safely be called before
> Py_Initialize
>
> So it is interesting to know that list, of course, but the ones that are
> to be supported and documented might be a smaller list. Or might not.
>

Ah, sorry - "safely" was a bit ambiguous there. By "safely" I meant
"CPython has a regression test that ensures that particular API will keep
working before Py_Initialize(), regardless of any changes we may make to
the way we handle interpreter initialization".

We've long had a lot of other APIs that happen to work well enough for
CPython itself to get away with using them during the startup process, but
the official position on those is "Don't count on these APIs working prior
to Py_Initialize() in the general case - we only get away with it because
we can adjust the exact order in which we do things in order to account for
any other changes that break it".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python initialization and embedded Python

[Serhiy Storchaka]

24.11.17 04:21, Glenn Linderman пише:

On 11/23/2017 5:31 PM, Nick Coghlan wrote:
- a consolidated list of *all* the APIs that can safely be called 
before Py_Initialize
So it is interesting to know that list, of course, but the ones that are 
to be supported and documented might be a smaller list. Or might not.


This is a small list, 11 functions.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com