Re: [Python-Dev] Accepting PEP 492 (async/await)

2015-05-06 Thread Paul Moore
On 6 May 2015 at 00:58, Guido van Rossum  wrote:
> I totally forgot to publicly congratulate Yury on this PEP. He's put a huge
> effort into writing the PEP and the implementation and managing the
> discussion, first on python-ideas, later on python-dev. Congrats, Yury! And
> thanks for your efforts. Godspeed.

Agreed, congratulations! There's been a lot of debate on this PEP, and
Yury has done a great job of responding where needed and keeping
things on track, which can't have been easy. Thanks for all the work.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-06 Thread Greg Ewing

Paul Moore wrote:


What about Greg Ewing's example?
http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Examples/Scheduler/scheduler.txt


That doesn't cover any of the higher level abstractions like tasks or
futures (at least not by those names or with those interfaces).


Because a minimal event loop doesn't *need* those.

In my little scheduler, a "task" is nothing more than
a yield-frommable object sitting on a queue of things
to be run. There is no need to wrap it in another
object.

And there's really no need for the concept of a
"future" at all, except maybe at the boundary
between generator-based async code and other things
that are based on callbacks. Even then, a "future"
is really just "an object that can be passed to
yield-from". There is no need for a concrete
Future class, it's just a protocol.


And I
don't see where the PEP 492 additions would fit in (OK, "replace yield
from with await" is part of it, but I don't see the rest).


That's really all there is to it. The rest is
concerned with catching certain kinds of mistakes,
and providing convenient syntax for some patterns
of using 'await'.


There's a lot of asyncio
that doesn't seem to me to be IO-related. Specifically the future and
task abstractions. I view those as relevant to "coroutine programming
in Python" because they are referenced in any discussion of coroutines
(you yield from a future, for example).


Only because they've been elevated to prominence
by asyncio and its documentation, which I regard
as unfortunate.

When Guido was designing asyncio, I tried very
hard to dissuade him from giving futures such a
central place in the model. I saw them as an
unnecessary concept that would only clutter up
people's thinking. Seeing all the confusion now,
I'm more convinced than ever that I was right. :-(


In some ways I wish there had been an "asyncio" library that covered
the areas that are fundamentally about IO multiplexing. And a separate
library (just "async", maybe, although that's now a bad idea as it
clashes with a keyword :-)) that covered generic event loop, task and
synchronisation areas.


As I said before, I don't think it's really
possible to factor an event loop into those kinds
of parts. You may be able to factor the *API* that
way, but any given implementation has to address
all the parts at once.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Accepting PEP 492 (async/await)

2015-05-06 Thread Andrew Svetlov
Congrats, Yury!

On Wed, May 6, 2015 at 11:07 AM, Paul Moore  wrote:
> On 6 May 2015 at 00:58, Guido van Rossum  wrote:
>> I totally forgot to publicly congratulate Yury on this PEP. He's put a huge
>> effort into writing the PEP and the implementation and managing the
>> discussion, first on python-ideas, later on python-dev. Congrats, Yury! And
>> thanks for your efforts. Godspeed.
>
> Agreed, congratulations! There's been a lot of debate on this PEP, and
> Yury has done a great job of responding where needed and keeping
> things on track, which can't have been easy. Thanks for all the work.
>
> Paul.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com



-- 
Thanks,
Andrew Svetlov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] Minimal async event loop and async utilities (Was: PEP 492: async/await in Python; version 4)

2015-05-06 Thread Paul Moore
On 6 May 2015 at 07:46, Greg Ewing  wrote:
> Another problem with the "core" idea is that
> you can't start with an event loop that "just does
> scheduling" and then add on other features such
> as I/O *from the outside*. There has to be some
> point at which everything comes together, which
> means choosing something like select() or
> poll() or I/O completion queues, and build that
> into the heart of your event loop. At that point
> it's no longer something with a simple core.

Looking at asyncio.queues, the only features it needs are:

1. asyncio.events.get_event_loop()
2. asyncio.futures.Future - creating a standalone Future
3. asyncio.locks.Event
4. @coroutine

locks.Event in turn only needs the other 3 items. And you can ignore
get_event_loop() as it's only used to get the default loop, you can
pass in your own.

And asyncio.futures only uses get_event_loop (and _format_callback)
from asyncio.events.

Futures require the loop to support:
1. call_soon
2. call_exception_handler
3. get_debug

So, to some extent (how far is something I'd need to code up a loop to
confirm) you can build the Futures and synchronisation mechanisms with
an event loop that supports only this "minimal interface".

Essentially, that's my goal - to allow people who want to write (say)
a Windows GUI event loop, or a Windows event loop based of
WaitForXXXObject, or a Tkinter loop, or whatever, to *not* have to
write their own implementation of synchronisation or future objects.

That may mean lifting the asyncio code and putting it into a separate
library, to make the separation between "asyncio-dependent" and
"general async" clearer. Or if asyncio's provisional status doesn't
last long enough to do that, we may end up with an asyncio
implementation and a separate (possibly 3rd party) "general"
implementation.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-06 Thread Wolfgang Langner
Hi Yury,


On 05.05.2015 20:25, Yury Selivanov wrote:

>>
>> We forget to address the major problems here. How can someone in a
>> "sync" script use this async stuff easy. How can async and sync stuff
>> cooperate and we don't need to rewrite the world for async stuff.
>> How can a normal user access the power of async stuff without rewriting
>> all his code. So he can use a simple asyc request library in his code.
>> How can a normal user learn and use all this in an easy way.
> 
> asyncio and twisted answered these questions ;) The answer is
> that you have to write async implementations.
> 
> gevent has a different answer, but greenlents/stackless is
> something that will never be merged in CPython and other
> implementations.
> 

I think monkeypatching and the gevent way is wrong. And explicit is better
than implicit.

I have to clear this. I meant how can we make this async stuff more accessible
to the average sync user. Sometime even without writing or knowing how
to write coroutines or other async stuff.

Let me explain it with a deeper example (Some of it is related to Python 2 and
twisted):

I had the same problem for a server application using twisted. Provide
easy interfaces to my users most not aware of async stuff.

My solution was to write my own decorator similar to twisted's
@inlineCallbacks. On top of it I added one more level to the decorator
and distinguish if it was called from the main thread (also running the
mainloop) and other threads. This object usable also as decorator
is called "task" and has some utility methods. And returns a "deferred".
(in asyncio this would be a Future)

Resulting in code like:

@task
def myAsyncFunction(webaddress):
  result = yield fetch(webaddress)
  # only to show sleep example
  yield task.sleep(0.1)
  task.Return(result)

Usable in a sync context (extra script thread):

def do():
  content = myAsyncFunction("http://xyz";)

or async context:

@task
def ado():
  content = yield myAsyncFunction("http://xyz";)


The task decorator has functionality to check if something is
called from the main thread (-> also a task and async)
or it is called from another thread (-> sync or script).

So this async interface is usable from both worlds. If someone
operates async he/she must only know the task decorator and when to yield.
If he/she uses it in sync mode nothing special has to be done.

To allow all this the server starts the async main loop in the main thread
and executes the script in an extra script thread. The user has every time his
own thread, also for rpc stuff. The only way to switch into the main loop
is to decorate a function as @task, every task is a coroutine and executed
in the main thread (also thread of main loop).

Benefit of all this:

- Easy to write a async task it is marked as one and special stuff belongs
  to the task object. (task.Return is needed because we are in Python 2)
- The normal user executes his stuff in his own thread and he/she
  can program in sync mode. No problem it is an extra thread and the main loop
  does not block.
- A network client or other stuff has to be written only once, most time this
  can be a @task in the async world. But this should not care the end user.
  We don't have to implement all twice once for async and once for the sync
  world. -> Less overhead

This is what I mean if I say we must address the bridging problem between the
worlds.
It think it is the wrong way to divide it in async and sync stuff and
duplicate all networking libraries in the sync and async ones.

For me the answer is to write one async netowrk library and use it in both,
a sync script and in an async main loop. With an easy interface and not
forcing the user to know this is an async library I have to do something 
special.

And in future go one step further and use all this combined with PyParallel
to solve the multiprocessing problem in Python.
(Script in extra thread, mainloop in main thread, executed and managed via
PyParallel avoiding the gil)
But this is only a vision/dream of mine.


>>
>> And for all this we still can't tell them "oh the async stuff solves
>> the multiprocessing problem of Python learn it and switch to version
>> 3.5". It does not and it is only most useful for networking stuff
>> nothing more.
> 
> "networking stuff", and in particular, web, is a huge
> part of current Python usage.  Please don't underestimate
> that.

I do not. But for most they want only use it as a client
and the main concern for most is "I want to get this web page"
and not "I will implement a web client have to do this async and get it".


Regards,

Wolfgang

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-06 Thread Paul Moore
On 6 May 2015 at 09:20, Greg Ewing  wrote:
>> That doesn't cover any of the higher level abstractions like tasks or
>> futures (at least not by those names or with those interfaces).
>
> Because a minimal event loop doesn't *need* those.

It doesn't *need* them, but as abstractions they allow easier building
of reusable higher-level libraries. You can write an event loop with
nothing but coroutines, but to build reusable libraries on top of it,
you need some common interfaces.

> In my little scheduler, a "task" is nothing more than
> a yield-frommable object sitting on a queue of things
> to be run. There is no need to wrap it in another
> object.
>
> And there's really no need for the concept of a
> "future" at all, except maybe at the boundary
> between generator-based async code and other things
> that are based on callbacks. Even then, a "future"
> is really just "an object that can be passed to
> yield-from". There is no need for a concrete
> Future class, it's just a protocol.

Agreed, you don't need a Future class, all you need is to agree what
reusable code is allowed to do with the core objects you are passing
around - that's how duck typing works. The objects *I* can see are
futures (in a PEP 492 world, "awaitables" which may or may not be
equivalent in terms of the operations you'd want to focus on) and the
event loop itself.

In your example, the event loop is implicit (as it's a singleton, you
use global functions rather than methods on the loop object) but
that's a minor detail.

>> And I
>> don't see where the PEP 492 additions would fit in (OK, "replace yield
>> from with await" is part of it, but I don't see the rest).
>
> That's really all there is to it. The rest is
> concerned with catching certain kinds of mistakes,
> and providing convenient syntax for some patterns
> of using 'await'.

So, "things you can wait on" have one operation - "wait for a result".
That's OK. You can create such things as coroutines, which is also
fine. You may want to create such things explicitly (equivalent to
generators vs __iter__) - maybe that's where __aiter__ comes in in PEP
492 and the Future class in asyncio. Again, all fine.

You also need operations like "schedule a thing to run", which is the
event loop "interface". Your sample has the following basic event loop
methods that I can see: run, schedule, unschedule, and
expire_timeslice (that last one may be an implementation detail, but
the other 3 seem pretty basic). PEP 492 has nothing to say on the
event loop side of things (something that became clear to me during
this discussion).

>> There's a lot of asyncio
>> that doesn't seem to me to be IO-related. Specifically the future and
>> task abstractions. I view those as relevant to "coroutine programming
>> in Python" because they are referenced in any discussion of coroutines
>> (you yield from a future, for example).
>
> Only because they've been elevated to prominence
> by asyncio and its documentation, which I regard
> as unfortunate.
>
> When Guido was designing asyncio, I tried very
> hard to dissuade him from giving futures such a
> central place in the model. I saw them as an
> unnecessary concept that would only clutter up
> people's thinking. Seeing all the confusion now,
> I'm more convinced than ever that I was right. :-(

Futures seem to me to be (modulo a few details) what "awaitables" are
in PEP 492. I can't see how you can meaningfully talk about event
loops in a Python context without having *some* term for "things you
wait for". Maybe Future wasn't a good name, and maybe the parallel
with concurrent.futures.Future wasn't helpful (I think both things
were fine, but you may not) but we had to have *some* way of talking
about them, and of constructing standalone awaitables. PEP 492 has
new, and hopefully better, ways, but I think that awaitables *have* to
be central to any model where you wait for things...

By the way, it feels to me like I'm now arguing in favour of PEP 492
with a reasonable understanding of what it "means". Assuming what I
said above isn't complete rubbish, thanks to everyone who's helped me
get to this point of understanding through this thread! (And if I
haven't understood, that's my fault, and still thanks to everyone for
their efforts :-))

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-06 Thread Greg Ewing

Guido van Rossum wrote:
the bytecode generated for 
await treats coroutine objects special, just like the bytecode generated 
for yield-from treats generator objects special. The special behavior 
they have in common is the presence of send() and throw() methods,


I don't think that's quit accurate. Yield-from treats
any object having send() and throw() methods the same
way it treats a generator -- there's nothing special
about the generator *type*. Presumably 'await' is the
same.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: What is the real goal?

2015-05-06 Thread Oscar Benjamin
On 5 May 2015 at 17:48, Yury Selivanov  wrote:
>
> I've updated the PEP with some fixes of the terminology:
> https://hg.python.org/peps/rev/f156b272f860

Yes that looks better.

> I still think that 'coroutine functions' and 'coroutines'
> is a better pair than 'async functions' and 'coroutines'.

Fair enough. The terminology in the PEP seems consistent now which is
more important than the exact terms used.


--
Oscar
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-06 Thread Greg Ewing

Paul Moore wrote:

I can't see how you can meaningfully talk about event
loops in a Python context without having *some* term for "things you
wait for".


PEP 3152 was my attempt at showing how you could do that.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 5

2015-05-06 Thread Greg Ewing

Terry Reedy wrote:

What I do not understand is how io events become event loop Event 
instances.


They don't become Events exactly, but you can register
a callback to be called when a file becomes ready for
reading or writing, see:

http://effbot.org/pyfaq/can-i-have-tk-events-handled-while-waiting-for-i-o.htm

That's probably enough of a hook to be able to get
asyncio-style file I/O working on top of Tkinter.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Accepting PEP 492 (async/await)

2015-05-06 Thread Larry Hastings

On 05/05/2015 04:53 PM, Guido van Rossum wrote:
I've given Yury clear instructions to focus on how to proceed -- he's 
to work with another core dev on getting the implementation ready in 
time for beta 1 (scheduled for May 24, but I think the target date 
should be May 19).


Released on Sunday May 24 means it'll be tagged on Saturday May 23. 
Please take care that it be checked in by then.


Your friendly neighborhood release manager,


//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Minimal async event loop and async utilities (Was: PEP 492: async/await in Python; version 4)

2015-05-06 Thread Guido van Rossum
On Wed, May 6, 2015 at 1:27 AM, Paul Moore  wrote:

> On 6 May 2015 at 07:46, Greg Ewing  wrote:
> > Another problem with the "core" idea is that
> > you can't start with an event loop that "just does
> > scheduling" and then add on other features such
> > as I/O *from the outside*. There has to be some
> > point at which everything comes together, which
> > means choosing something like select() or
> > poll() or I/O completion queues, and build that
> > into the heart of your event loop. At that point
> > it's no longer something with a simple core.
>
> Looking at asyncio.queues, the only features it needs are:
>
> 1. asyncio.events.get_event_loop()
> 2. asyncio.futures.Future - creating a standalone Future
> 3. asyncio.locks.Event
> 4. @coroutine
>
> locks.Event in turn only needs the other 3 items. And you can ignore
> get_event_loop() as it's only used to get the default loop, you can
> pass in your own.
>
> And asyncio.futures only uses get_event_loop (and _format_callback)
> from asyncio.events.
>
> Futures require the loop to support:
> 1. call_soon
> 2. call_exception_handler
> 3. get_debug
>
> So, to some extent (how far is something I'd need to code up a loop to
> confirm) you can build the Futures and synchronisation mechanisms with
> an event loop that supports only this "minimal interface".
>
> Essentially, that's my goal - to allow people who want to write (say)
> a Windows GUI event loop, or a Windows event loop based of
> WaitForXXXObject, or a Tkinter loop, or whatever, to *not* have to
> write their own implementation of synchronisation or future objects.
>
> That may mean lifting the asyncio code and putting it into a separate
> library, to make the separation between "asyncio-dependent" and
> "general async" clearer. Or if asyncio's provisional status doesn't
> last long enough to do that, we may end up with an asyncio
> implementation and a separate (possibly 3rd party) "general"
> implementation.
>

This is actually a great idea, and I encourage you to go forward with it.
The biggest piece missing from your inventory is probably Task, which is
needed to wrap a Future around a coroutine.

I expect you'll also want to build cancellation into your "base async
framework"; and the primitives to wait for multiple awaitables. The next
step would be some mechanism to implement call_later()/call_at() (but this
needs to be pluggable since for a "real" event loop it needs to be
implemented by the basic I/O selector).

If you can get this working it would be great to include this in the stdlib
as a separate "asynclib" library. The original asyncio library would then
be a specific implementation (using a subclass of asynclib.EventLoop) that
adds I/O, subprocesses, and integrates with the selectors module (or with
IOCP, on Windows).

I don't see any particular hurry to get this in before 3.5; the refactoring
of asyncio can be done later, in a backward compatible way. It would be a
good way to test the architecture of asyncio!

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] Ancient use of generators

2015-05-06 Thread Guido van Rossum
For those interested in tracking the history of generators and coroutines
in Python, I just found out that PEP 342
 (which introduced
send/throw/close and made "generators as coroutines" a mainstream Python
concept) harks back to PEP 288 ,
which was rejected. PEP 288 also proposed some changes to generators. The
interesting bit though is in the references: there are two links to old
articles by David Mertz that describe using generators in state machines
and other interesting and unconventional applications of generators. All
these well predated PEP 342, so yield was a statement and could not receive
a value from the function calling next() -- communication was through a
shared class instance.

http://gnosis.cx/publish/programming/charming_python_b5.txt
http://gnosis.cx/publish/programming/charming_python_b7.txt

Enjoy!

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 5

2015-05-06 Thread Yury Selivanov

Hi Ben,

On 2015-05-06 12:05 AM, Ben Darnell wrote:

On Tue, May 5, 2015 at 3:25 PM, Yury Selivanov 
wrote:


Yes, there is no other popular event loop for 3.4 other
than asyncio, that uses coroutines based on generators
(as far as I know).


Tornado supports Python 3.4 and uses generator-based coroutines. We use
`yield` instead of `yield from` for compatibility with Python 2. I have a
patch to support the new async/await syntax here:
https://github.com/bdarnell/tornado/commit/e3b71c3441e9f87a29a9b112901b7644b5b6edb8


I don't know how this happened, especially since I've used
Tornado myself!  It's amazing that Tornado will have support
of async/await when 3.5 is out!



Overall, I like the PEP. I've been reluctant to embrace `yield from` for
Tornado coroutines (Tornado's Futures do not implement `__iter__`) because
I'm worried about confusion between `yield` and `yield from`, but async and
await are explicit enough that that's not really a problem.

My one request would be that there be a type or ABC corresponding to
inspect.isawaitable(). Tornado uses functools.singledispatch to handle
interoperability with other coroutine frameworks, so it would be best if we
could distinguish awaitables from other objects in a way that is compatible
with singledispatch. The patch above simply registers types.GeneratorType
which isn't quite correct.



Sure. I'll add Awaitable and Coroutine ABCs.

Thanks,
Yury

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 5

2015-05-06 Thread Terry Reedy

On 5/5/2015 10:59 PM, Guido van Rossum wrote:

For this you should probably use an integration of asyncio (which can do
async subprocess output nicely) with Tkinter. Over in tulip-land there
is an demo of such an integration.


After redirection from googlecode tulip, I found
https://github.com/python/asyncio/tree/master/examples
None of the 4 *process*.py examples mention tkinter.

I also found "Create a Tkinter/Tulip integration"
https://github.com/python/asyncio/issues/21
with attachment tk_ayncio.zip
copied (with 'async' replacing 'tulip') to
https://bitbucket.org/haypo/asyncio_staging/src/bb76064d80b0a03bf3f7b13652e595dfe475c7f8/asyncio_tkinter/?at=default

None of the integration files mention subprocess, so I presume you are 
suggesting that I use a modification of one of the example subprocess 
coroutines with the integration framework.


If this works well, might it make sense to consider using an elaboration 
of examples/subprocess_shell.py to replace subprocess socket 
communication with pipe comminication?



On Tue, May 5, 2015 at 6:03 PM, Terry Reedy mailto:tjre...@udel.edu>> wrote:



My specific use case is to be able to run a program in a separate
process, but display the output in the gui process -- something like
this (in Idle, for instance).  (Apologies if this misuses the new
keywords.)

async def menu_handler()
 ow = OutputWindow(args)  # tk Widget
 proc = subprocess.Popen (or multiprocessing equivalent)
 out = (stdout from process)
 await for line in out:
 ow.write(line)
 finish()

I want the handler to not block event processing, and disappear
after finishing.  Might 492 make this possible someday?  Or would
having 'line in pipe' or just 'data in pipe' translated to a tk
event likely require a patch to tk?


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 5

2015-05-06 Thread Guido van Rossum
Sorry to send you on such a wild goose chase! I did mean the issue you
found #21). I just updated it with a link to a thread that has more
news:  
https://groups.google.com/forum/#!searchin/python-tulip/tkinter/python-tulip/TaSVW-pjWro/hCP6qS4eRnAJ

I wasn't able to verify the version by Luciano Ramalho. (And yes, extending
all this to working with a subprocess is left as an exercise. It's all
pretty academic IMO, given Tkinter's lack of popularity outside IDLE.)


On Wed, May 6, 2015 at 2:32 PM, Terry Reedy  wrote:

> On 5/5/2015 10:59 PM, Guido van Rossum wrote:
>
>> For this you should probably use an integration of asyncio (which can do
>> async subprocess output nicely) with Tkinter. Over in tulip-land there
>> is an demo of such an integration.
>>
>
> After redirection from googlecode tulip, I found
> https://github.com/python/asyncio/tree/master/examples
> None of the 4 *process*.py examples mention tkinter.
>
> I also found "Create a Tkinter/Tulip integration"
> https://github.com/python/asyncio/issues/21
> with attachment tk_ayncio.zip
> copied (with 'async' replacing 'tulip') to
>
> https://bitbucket.org/haypo/asyncio_staging/src/bb76064d80b0a03bf3f7b13652e595dfe475c7f8/asyncio_tkinter/?at=default
>
> None of the integration files mention subprocess, so I presume you are
> suggesting that I use a modification of one of the example subprocess
> coroutines with the integration framework.
>
> If this works well, might it make sense to consider using an elaboration
> of examples/subprocess_shell.py to replace subprocess socket communication
> with pipe comminication?
>
>  On Tue, May 5, 2015 at 6:03 PM, Terry Reedy > > wrote:
>>
>
>  My specific use case is to be able to run a program in a separate
>> process, but display the output in the gui process -- something like
>> this (in Idle, for instance).  (Apologies if this misuses the new
>> keywords.)
>>
>> async def menu_handler()
>>  ow = OutputWindow(args)  # tk Widget
>>  proc = subprocess.Popen (or multiprocessing equivalent)
>>  out = (stdout from process)
>>  await for line in out:
>>  ow.write(line)
>>  finish()
>>
>> I want the handler to not block event processing, and disappear
>> after finishing.  Might 492 make this possible someday?  Or would
>> having 'line in pipe' or just 'data in pipe' translated to a tk
>> event likely require a patch to tk?
>>
>
> --
> Terry Jan Reedy
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 4

2015-05-06 Thread Ben Darnell
On Tue, May 5, 2015 at 1:39 PM, Paul Moore  wrote:

>
> It would probably be helpful to have a concrete example of a basic
> event loop that did *nothing* but schedule tasks. No IO waiting or
> similar, just scheduling. I have a gut feeling that event loops are
> more than just asyncio, but without examples to point to it's hard to
> keep a focus on that fact. And even harder to isolate "what is an
> event loop mechanism" from "what is asyncio specific". For example,
> asyncio.BaseEventLoop has a create_connection method. That's
> *obviously* not a fundamental aspect of a generic event loop, But
> call_soon (presumably) is. Having a documented "basic event loop"
> interface would probably help emphasise the idea than event loops
> don't have to be asyncio. (Actually, what *is* the minimal event loop
> interface that is needed for the various task/future mechanisms to
> work, independently of asyncio? And what features of an event loop etc
> are needed for the PEP, if it's being used outside of asyncio?)
>

Twisted has a pretty good taxonomy of event loop methods, in the interfaces
at the bottom of this page:
http://twistedmatrix.com/documents/15.1.0/core/howto/reactor-basics.html
and the comparison matrix at
http://twistedmatrix.com/documents/15.1.0/core/howto/choosing-reactor.html

The asyncio event loops implement most of these (not the exact interfaces,
but the same functionality). Tornado implements FDSet, Time, and part of
Threads in the IOLoop itself, with the rest of the functionality coming
from separate classes. (You may wonder then why Twisted and asyncio put
everything in the core event loop? It's necessary to abstract over certain
platform differences, which is one big reason why Tornado has poor support
for Windows).

-Ben



>
> I guess the other canonical event loop use case is GUI system message
> dispatchers.
>
> >> You can argue that the syntax is needed to help
> >> make async more accessible - but if that's the case then the
> >> terminology debates and confusion are clear evidence that it's not
> >> succeeding in that goal.
> >
> > Perhaps, but arguing about the nitty-gritty details of something doesn't
> > automatically lead to a clearer understanding of the higher level
> concept.
> > Discussing how turning a steering wheel in a car might help you grasp how
> > cars turn, but it isn't a requirement to get "turn the wheel left to make
> > the car go left".
>
> Fair point. If only I could avoid driving into walls :-)
>
> >> Of course, that's based on my perception of
> >> one of the goals of the PEP as being "make coroutines and asyncio more
> >> accessible", If the actual goals are different, my conclusion is
> >> invalid.
> >
> > I think the goal is "make coroutines easier to use" and does not directly
> > relate to asyncio.
>
> OK. But in that case, some examples using a non-asyncio toy "just
> schedule tasks" event loop might help.
>
> >> Well, twisted always had defer_to_thread. Asyncio has run_in_executor,
> >> but that seems to be callback-based rather than coroutine-based?
> >
> > Yep.
>
> ... and so you can't use it with async/await?
>
> >> Many people use requests for their web access. There are good reasons
> >> for this. Are you saying that until someone steps up and writes an
> >> async implementation of requests, I have to make a choice - requests
> >> or asyncio?
> >
> > I believe so; you need something to implement __await__. This is true in
> any
> > language that implements co-routines.
> >
> >> Unfortunately, I can't see myself choosing asyncio in that
> >> situation. Which again means that asyncio becomes "something that the
> >> average user can't use". Which in turn further entrenches it as a
> >> specialist-only tool.
> >
> > You forgot to append "... yet" to that statement. Just because something
> > isn't available out of the box without some effort to support doesn't
> mean
> > it will never happen, else there would be absolutely no Python 3 users
> out
> > there.
>
> Fair point. Yuri mentioned aiohttp, as well. The one difference
> between this and Python 2/3, is that here you *have* to have two
> separate implementations. There's no equivalent of a "shared source"
> async and synchronous implementation of requests. So the typical
> "please support Python 3" issue that motivates projects to move
> forward doesn't exist in the same way. It's not to say that there
> won't be async versions of important libraries, it's just hard to see
> how the dynamics will work. I can't see myself raising an issue on
> cx_Oracle saying "please add asyncio support", and I don't know who
> else I would ask...
>
> > Co-routine-based asynchronous programming is new to Python, so as a
> > community we don't have it as something everyone learns over time. If you
> > don't come from an area that supports it then it will be foreign to you
> and
> > not make sense without someone giving you a good tutorial on it. But
> > considering C#, Dart, and Ecmascript 6 (will) have co-routine su

Re: [Python-Dev] PEP 492: async/await in Python; version 5

2015-05-06 Thread Terry Reedy

On 5/6/2015 5:39 PM, Guido van Rossum wrote:

Sorry to send you on such a wild goose chase! I did mean the issue you
found #21). I just updated it with a link to a thread that has more
news:
https://groups.google.com/forum/#!searchin/python-tulip/tkinter/python-tulip/TaSVW-pjWro/hCP6qS4eRnAJ

I wasn't able to verify the version by Luciano Ramalho. (And yes,
extending all this to working with a subprocess is left as an exercise.
It's all pretty academic IMO, given Tkinter's lack of popularity outside
IDLE.)


On Stackoverflow pyside has gotten 40 questions tagged in the last 30 
days, wxpython 70 in the last 30 days, pyqt 114 in 30 days, while 
tkinter has gotten 101 in the last week, which would project to about 
425 in the last 30 days.  So tkinter is being used at least by 
beginners. There have been a few tkinter and python-asyncio questions.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Clarification of PEP 476 "opting out" section

2015-05-06 Thread Nick Coghlan
On 30 Apr 2015 5:59 pm, "M.-A. Lemburg"  wrote:

> On 30.04.2015 02:33, Nick Coghlan wrote:
> > Hi folks,
> >
> > This is just a note to highlight the fact that I tweaked the "Opting
> > out" section in PEP 476 based on various discussions I've had over the
> > past few months: https://hg.python.org/peps/rev/dfd96ee9d6a8
> >
> > The notable changes:
> >
> > * the example monkeypatching code handles AttributeError when looking
> > up "ssl._create_unverified_context", in order to accommodate older
> > versions of Python that don't have PEP 476 implemented
> > * new paragraph making it clearer that while the intended use case for
> > the monkeypatching trick is as a workaround to handle environments
> > where you *know* HTTPS certificate verification won't work properly
> > (including explicit references to sitecustomize.py and Standard
> > Operating Environments for Python), there's also a secondary use case
> > in allowing applications to provide a system administrator controlled
> > setting to globally disable certificate verification (hence the change
> > to the example code)
> > * new paragraph making it explicit that even though we've improved
> > Python's default behaviour, particularly security sensitive
> > applications should still provide their own context rather than
> > relying on the defaults
>
> Can we please make the monkeypatch a regular part of Python's
> site.py which can enabled via an environment variable, say
> export PYTHONHTTPSVERIFY=0.
>
> See http://bugs.python.org/issue23857 for the discussion.
>
> Esp. for Python 2.7.9 the default verification from PEP 476
> is causing problems for admins who want to upgrade their
> Python installation without breaking applications using
> Python. They need an easy and official non-hackish way to
> opt-out from the PEP 476 default on a per application basis.

My current recommendation to the Red Hat Python team is to put
together a draft PEP 394 style informational PEP to define a
documented configuration file based approach that redistributors may
choose to implement in order to provide a smoother transition path for
their end users:
https://bugzilla.redhat.com/show_bug.cgi?id=1173041#c8

The problem I identified with reusing the environment variable based
approach that we used with hash randomisation is that "-E" turns the
upstream default behaviour back on. That was acceptable for hash
randomisation, but would be too limiting here (as the appropriate
value for the HTTPS certificate verification configuration setting
relates primarily to the general state of SSL/TLS certificate
management in the environment where Python is being deployed, moreso
than to the specific networked applications that are being run).

I actually do think it would be good to have such a feature as a
native part of Python 2.7 in order to provide a nicer "revert to the
pre-PEP-476 behaviour" experience for Python 2.7 users (leaving the
"there's no easy way to turn HTTPS certificate verification off
globally" state of affairs to Python 3), but I don't currently have
the time available to push for that against the "end users can't be
trusted not to turn certificate verification off when they should be
fixing their certificate management instead" perspective.

Cheers,
Nick.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Ancient use of generators

2015-05-06 Thread Nick Coghlan
David Beazley's tutorials on these topics are also excellent:

* Generator Tricks for Systems Programmers (PyCon 2008:
http://www.dabeaz.com/generators/)
* A Curious Course on Coroutines and Concurrency (PyCon 2009:
http://www.dabeaz.com/coroutines/)
* Generators: The Final Frontier (PyCon 2014:
http://www.dabeaz.com/finalgenerator/)

The first one focuses on iteration, the second expands to cover PEP
342 and sending values into coroutines, while the last expands to
cover *all* the different ways we use generator suspension points
these days (including in context managers and asynchronous I/O).

The async I/O section is particularly interesting because David
initially uses threads in order to delay getting into the complexities
of event loops, while still illustrating the concept of "waiting for
other things to happen".

(Note: once you get to Part 5 of the last tutorial, you're getting to
stuff that really pushes the boundaries of what generators can do,
using examples from domains that are complex in their own right. The
very last section also contains some wise words around the fact that
we're genuinely pushing out the boundaries of the language's
expressive capabilities, which does carry some non-trivial risks)

I do believe that last tutorial also does a good job of illustrating a
lot of the complexity that PEP 492 is intended to *hide* from
(future!) end users by simplifying the story to "use async & await for
cooperative multitasking, threads & processes for preemptive
multitasking, and yield & yield from for iteration", rather than
having the "cooperative multitasking" case continue to be a particular
way of using yield & yield from as it is in Python 3.4 (and a way of
use "yield" in earlier releases).

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PEP 492: async/await in Python; version 5

2015-05-06 Thread Guido van Rossum
Thanks for these stats! (Though I'm sure there's some bias because Tkinter
is in the stdlib and the others aren't. Stdlib status still counts a lot
for many people, pip notwithstanding. (But that's a different thread. :-))

FWIW I tried to get the Tkinter-asyncio demo mentioned in the above thread
to work but I couldn't (then again Tk on OS X is difficult). If someone
could turn this into a useful event loop that blends asyncio and Tkinter
that would be truly awesome!

On Wed, May 6, 2015 at 6:32 PM, Terry Reedy  wrote:

> On 5/6/2015 5:39 PM, Guido van Rossum wrote:
>
>> Sorry to send you on such a wild goose chase! I did mean the issue you
>> found #21). I just updated it with a link to a thread that has more
>> news:
>>
>> https://groups.google.com/forum/#!searchin/python-tulip/tkinter/python-tulip/TaSVW-pjWro/hCP6qS4eRnAJ
>> <
>> https://groups.google.com/forum/#%21searchin/python-tulip/tkinter/python-tulip/TaSVW-pjWro/hCP6qS4eRnAJ
>> >
>> I wasn't able to verify the version by Luciano Ramalho. (And yes,
>> extending all this to working with a subprocess is left as an exercise.
>> It's all pretty academic IMO, given Tkinter's lack of popularity outside
>> IDLE.)
>>
>
> On Stackoverflow pyside has gotten 40 questions tagged in the last 30
> days, wxpython 70 in the last 30 days, pyqt 114 in 30 days, while tkinter
> has gotten 101 in the last week, which would project to about 425 in the
> last 30 days.  So tkinter is being used at least by beginners. There have
> been a few tkinter and python-asyncio questions.
>
>
> --
> Terry Jan Reedy
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com