Re: [Python-Dev] Importance of "async" keyword

[Nick Coghlan]

On 26 Jun 2015 05:15, "Andrew Svetlov"  wrote:
>
> > Another issue that bothers me, is code reuse. Independent from whether
the
> > 'async def' makes sense or not, it would not allow us to reuse asyncio
> > functions as if they were normal functions and vice versa (if I
understood
> > that correctly). So, we would have to implement things twice for the
asyncio
> > world and the classic world. To me, it would be important to use one
> > function in either world where it suits me better. I am uncertain if
that
> > makes sense but right now it does to me.
>
>
> Yes, you cannot call async function from synchronous code. There are
> two worlds: classic and async.

Exactly, the sync/async split is inherent in the problem domain. Nobody
really likes this consequence of explicitly asynchronous programming
models, and this is a good article on why it's annoying:
http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/

However, it's also one of those cases like binary floating point: there are
good underlying reasons things are the way they are, but truly coming to
grips with them can take a long time, so in the meantime time it's
necessary to just accept it as "just one of those things" (like learning
negative or complex numbers for the first time). For explicitly
asynchronous programming, Glyph's post at
https://glyph.twistedmatrix.com/2014/02/unyielding.html is a decent
starting point on that journey.

Cheers,
Nick.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importance of "async" keyword

[Nick Coghlan]

On 26 Jun 2015 10:46, "Greg Ewing"  wrote:
>
> Sven R. Kunze wrote:
>> So, we would have to implement things twice for the asyncio world and
the classic world.
>
>
> Not exactly; it's possible to create a wrapper that takes an
> async function and runs it to completion, allowing it to be
> called from sync code. I can't remember offhand, but there's
> likely something like this already in asyncio.

https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.BaseEventLoop.run_until_complete

It's also possible to use a more comprehensive synchronous-to-asynchronous
adapter like gevent to call asynchronous code from synchronous code.

Going in the other direction (calling sync code from async) uses a thread
or process pool:
https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.BaseEventLoop.run_in_executor

Cheers,
Nick.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importance of "async" keyword

[Sven R. Kunze]

On 25.06.2015 21:11, Andrew Svetlov wrote:

Another issue that bothers me, is code reuse. Independent from whether the
'async def' makes sense or not, it would not allow us to reuse asyncio
functions as if they were normal functions and vice versa (if I understood
that correctly). So, we would have to implement things twice for the asyncio
world and the classic world. To me, it would be important to use one
function in either world where it suits me better. I am uncertain if that
makes sense but right now it does to me.


Yes, you cannot call async function from synchronous code. There are
two worlds: classic and async.


My point is: why does everyone assume that it has to be like that?


@Nick
Thanks for these links; nice reads and reflect exactly what I think 
about these topics. Btw. complex numbers basically works the same way 
(same API) as integers. I would like to see that for functions and 
awaitables as well.


Still, the general assumption is: "the sync/async split is inherent in 
the problem domain". Why? I do not see that inherent split.



@Greg
I do not like wrappers IF it is just because to have wrappers. We 
already have 2 types of wrappers: [normal function call] and [function 
call using await]. Both works like wrappers right in the place where you 
need them.



@Steven
> Where would the function suspend
> if there are zero suspension points?

It would not.

> How can you tell what the suspension
> points *in* the coroutine are from "await func()"?

Let me answer this with a question: How can you tell what the suspension 
points *in* the coroutine are from "async def"?


> Can you show a code snippet of your proposal?
By analogy to PEP 0492:

def complex_calc(a):
if a >= 0:
return await open(unicode(a)).read()
l = await complex_calc(a - 1)
r = complex_calc(a - 2)
return l + '; ' + r

def business():
return complex_calc(5)

def business_new()
return await complex_calc(10)


Maybe, I completely missed the point of the proposal, but this is the 
way I would expect it to work. Putting in an 'await' whenever I see fit 
and it just works.



Regards,
Sven

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importance of "async" keyword

[Ethan Furman]

On 06/26/2015 06:48 AM, Sven R. Kunze wrote:


def business():
 return complex_calc(5)

def business_new()
 return await complex_calc(10)



Maybe, I completely missed the point of the proposal, but this is the way I 
would expect it to work. Putting in an 'await' whenever I see fit and it just 
works.


Sadly, I have basically no experience in this area -- perhaps that's why Sven's 
arguments make sense to me.  ;)

As Nick said earlier: the caller always blocks; by extension (to my mind, at least) 
putting an `await` in front of something is saying, "it's okay if other tasks run 
while I'm blocking on this call."

Maybe we can get there someday.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importance of "async" keyword

[Chris Angelico]

On Sat, Jun 27, 2015 at 12:20 AM, Ethan Furman  wrote:
> As Nick said earlier: the caller always blocks; by extension (to my mind, at
> least) putting an `await` in front of something is saying, "it's okay if
> other tasks run while I'm blocking on this call."

Apologies if this is a really REALLY dumb question, but... How hard
would it be to then dispense with the await keyword, and simply
_always_ behave that way? Something like:

def data_from_socket():
# Other tasks may run while we wait for data
# The socket.read() function has yield points in it
data = socket.read(1024, 1)
return transmogrify(data)

def respond_to_socket():
while True:
data = data_from_socket()
# We can pretend that socket writes happen instantly,
# but if ever we can't write, it'll let other tasks wait while
# we're blocked on it.
socket.write("Got it, next please!")

Do these functions really need to be aware that there are yield points
in what they're calling?

I come from a background of thinking with threads, so I'm accustomed
to doing blocking I/O and assuming/expecting that other threads will
carry on while we wait. If asynchronous I/O can be made that
convenient, it'd be awesome.

But since it hasn't already been made that easy in every other
language, I expect there's some fundamental problem with this
approach, something that intrinsically requires every step in the
chain to know what's a (potential) block point.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importance of "async" keyword

[Paul Sokolovsky]

Hello,

On Sat, 27 Jun 2015 00:31:01 +1000
Chris Angelico  wrote:

> On Sat, Jun 27, 2015 at 12:20 AM, Ethan Furman 
> wrote:
> > As Nick said earlier: the caller always blocks; by extension (to my
> > mind, at least) putting an `await` in front of something is saying,
> > "it's okay if other tasks run while I'm blocking on this call."
> 
> Apologies if this is a really REALLY dumb question, but... How hard
> would it be to then dispense with the await keyword, and simply
> _always_ behave that way? Something like:
> 
> def data_from_socket():
> # Other tasks may run while we wait for data
> # The socket.read() function has yield points in it
> data = socket.read(1024, 1)
> return transmogrify(data)
> 
> def respond_to_socket():
> while True:
> data = data_from_socket()
> # We can pretend that socket writes happen instantly,
> # but if ever we can't write, it'll let other tasks wait while
> # we're blocked on it.
> socket.write("Got it, next please!")
> 
> Do these functions really need to be aware that there are yield points
> in what they're calling?
> 
> I come from a background of thinking with threads, so I'm accustomed
> to doing blocking I/O and assuming/expecting that other threads will
> carry on while we wait. If asynchronous I/O can be made that
> convenient, it'd be awesome.

Some say "convenient", others say "dangerous".

Consider:

buf = bytearray(MULTIMEGABYTE)

In one function you do:

socket.write(buf),

in another function (coroutine), you keep mutating buf.

So, currently in Python you know if you do:

socket.write(buf)

Then you know it will finish without interruptions for entire buffer.
And if you write:

await socket.write(buf)

then you know there may be interruption points inside socket.write(),
in particular something else may mutate it while it's being written.
With threads, you always have to assume this last scenario, and do
extra effort to protect against it (mutexes and stuff).


Note that it's original thinking of Guido, but seeing how it all
combines in asyncio (far from being nice consistent way), it's not
surprising that some people will keep not understanding how asyncio
works/how to use it, while other will grow disappointed by it. Here's
my latest disappointment wrt to asyncio usage in MicroPython:
http://bugs.python.org/issue24449


-- 
Best regards,
 Paul  mailto:pmis...@gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importance of "async" keyword

[Chris Angelico]

On Sat, Jun 27, 2015 at 12:51 AM, Paul Sokolovsky  wrote:
> Some say "convenient", others say "dangerous".
>
> Consider:
>
> buf = bytearray(MULTIMEGABYTE)
>
> In one function you do:
>
> socket.write(buf),
>
> in another function (coroutine), you keep mutating buf.
>
> So, currently in Python you know if you do:
>
> socket.write(buf)
>
> Then you know it will finish without interruptions for entire buffer.
> And if you write:
>
> await socket.write(buf)
>
> then you know there may be interruption points inside socket.write(),
> in particular something else may mutate it while it's being written.
> With threads, you always have to assume this last scenario, and do
> extra effort to protect against it (mutexes and stuff).

Hmm. I'm not sure how this is a problem; whether you use 'await' or
not, using a mutable object from two separate threads or coroutines is
going to have that effect.

The way I'm seeing it, coroutines are like cooperatively-switched
threads; you don't have to worry about certain operations being
interrupted (particularly low-level ones like refcount changes or list
growth), but any time you hit an 'await', you have to assume a context
switch. That's all very well, but I'm not sure it's that big a problem
to accept that any call could context-switch; atomicity is already a
concern in other cases, which is why we have principles like EAFP
rather than LBYL.

There's clearly some benefit to being able to assume that certain
operations are uninterruptible (because there's no 'await' anywhere in
them), but are they truly so? Signals can already interrupt something
_anywhere_:

>>> import signal
>>> def mutate(sig, frm):
... print("Mutating the global!")
... lst[5] = 0
...
>>> signal.signal(signal.SIGALRM, mutate)

>>> lst = [1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> def spin():
... while True:
... lst[0] += lst[5]
... lst[2] += lst[0]
... lst[4] += lst[6]
... lst[6] -= lst[2]
... if not lst[5]:
... print("Huh???")
... break
...
>>> signal.alarm(2)
0
>>> spin()
Mutating the global!
Huh???

Any time you're using mutable globals, you have to assume that they
can be mutated out from under you. Coroutines don't change this, so is
there anything gained by knowing where you can't be interrupted by one
specific possible context switch?

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importance of "async" keyword

[Yury Selivanov]

On 2015-06-26 10:31 AM, Chris Angelico wrote:

On Sat, Jun 27, 2015 at 12:20 AM, Ethan Furman  wrote:

>As Nick said earlier: the caller always blocks; by extension (to my mind, at
>least) putting an `await` in front of something is saying, "it's okay if
>other tasks run while I'm blocking on this call."

Apologies if this is a really REALLY dumb question, but... How hard
would it be to then dispense with the await keyword, and simply
_always_  behave that way? Something like:


Chris, Sven, if you want this behavior, gevent & Stackless Python
are your friends.  This approach, however, won't ever be merged in
CPython (this was discussed plenty of times both on python-ideas
and python-dev).

There is also a great essay about explicit vs implicit suspension
points: https://glyph.twistedmatrix.com/2014/02/unyielding.html

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importance of "async" keyword

[R. David Murray]

On Sat, 27 Jun 2015 01:10:33 +1000, Chris Angelico  wrote:
> The way I'm seeing it, coroutines are like cooperatively-switched
> threads; you don't have to worry about certain operations being
> interrupted (particularly low-level ones like refcount changes or list
> growth), but any time you hit an 'await', you have to assume a context
> switch. That's all very well, but I'm not sure it's that big a problem
> to accept that any call could context-switch; atomicity is already a
> concern in other cases, which is why we have principles like EAFP
> rather than LBYL.

Read Glyph's article, it explains why:

https://glyph.twistedmatrix.com/2014/02/unyielding.html

> There's clearly some benefit to being able to assume that certain
> operations are uninterruptible (because there's no 'await' anywhere in
> them), but are they truly so? Signals can already interrupt something
> _anywhere_:

Yes, and you could have an out of memory error anywhere in your program
as well.  (Don't do things in your signal handlers, set a flag.)  But
that doesn't change the stuff Glyph talks about (and Guido talks about)
about *reasoning* about your code.

I did my best to avoid using threads, and never invested the time and
effort in Twisted.  But I love programming with asyncio for highly
concurrent applications.  It fits in my brain :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2015-06-19 - 2015-06-26)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open4908 (+16)
  closed 31369 (+27)
  total  36277 (+43)

Open issues with patches: 2254 


Issues opened (29)
==

#15014: smtplib: add support for arbitrary auth methods
http://bugs.python.org/issue15014  reopened by barry

#24475: The docs never define what a pool "task" is
http://bugs.python.org/issue24475  opened by Zahari.Dim

#24477: In argparse subparser's option goes to parent parser
http://bugs.python.org/issue24477  opened by py.user

#24479: Support LMMS project files in mimetypes.guess_type
http://bugs.python.org/issue24479  opened by Andreas Nilsson

#24481: hotshot pack_string Heap Buffer Overflow
http://bugs.python.org/issue24481  opened by JohnLeitch

#24482: itertools.tee causes segfault in a multithreading environment,
http://bugs.python.org/issue24482  opened by Dmitry Odzerikho

#24483: Avoid repeated hash calculation in C implementation of functoo
http://bugs.python.org/issue24483  opened by serhiy.storchaka

#24484: multiprocessing cleanup occasionally throws exception
http://bugs.python.org/issue24484  opened by Jorge Herskovic

#24485: Function source inspection fails on closures
http://bugs.python.org/issue24485  opened by malthe

#24488: ConfigParser.getboolean fails on boolean options
http://bugs.python.org/issue24488  opened by andreas-h

#24492: using custom objects as modules: AttributeErrors new in 3.5
http://bugs.python.org/issue24492  opened by arigo

#24493: subprocess with env=os.environ doesn't preserve environment va
http://bugs.python.org/issue24493  opened by The Compiler

#24494: Can't specify encoding with fileinput and inplace=True
http://bugs.python.org/issue24494  opened by lilydjwg

#24498: Shoudl ptags and eptags be removed from repo?
http://bugs.python.org/issue24498  opened by r.david.murray

#24499: Python Installer text piles up during installation process
http://bugs.python.org/issue24499  opened by Zach “The Quantum Mechanic” W

#24500: provide context manager to redirect C output
http://bugs.python.org/issue24500  opened by Zahari.Dim

#24501: configure does not find (n)curses in /usr/local/libs
http://bugs.python.org/issue24501  opened by petepdx

#24502: OS X 2.7 package has zeros for version numbers in sub-packages
http://bugs.python.org/issue24502  opened by Jim Zajkowski

#24503: csv.writer fails when within csv.reader
http://bugs.python.org/issue24503  opened by ezzieyguywuf

#24505: shutil.which wrong result on Windows
http://bugs.python.org/issue24505  opened by bobjalex

#24506: make fails with gcc 4.9 due to fatal warning of unused variabl
http://bugs.python.org/issue24506  opened by krichter

#24507: CRLF issues
http://bugs.python.org/issue24507  opened by RusiMody

#24508: Backport 3.5's Windows build project files to 2.7
http://bugs.python.org/issue24508  opened by zach.ware

#24510: Make _PyCoro_GetAwaitableIter a public API
http://bugs.python.org/issue24510  opened by yselivanov

#24511: Add methods for async protocols
http://bugs.python.org/issue24511  opened by yselivanov

#24512: multiprocessing should log a warning when forking multithreade
http://bugs.python.org/issue24512  opened by trcarden

#24514: tarfile fails to extract archive (handled fine by gnu tar and 
http://bugs.python.org/issue24514  opened by pombreda

#24515: docstring of isinstance
http://bugs.python.org/issue24515  opened by Luc Saffre

#24516: SSL create_default_socket purpose insufficiently documented
http://bugs.python.org/issue24516  opened by messa



Most recent 15 issues with no replies (15)
==

#24512: multiprocessing should log a warning when forking multithreade
http://bugs.python.org/issue24512

#24499: Python Installer text piles up during installation process
http://bugs.python.org/issue24499

#24498: Shoudl ptags and eptags be removed from repo?
http://bugs.python.org/issue24498

#24481: hotshot pack_string Heap Buffer Overflow
http://bugs.python.org/issue24481

#24477: In argparse subparser's option goes to parent parser
http://bugs.python.org/issue24477

#24475: The docs never define what a pool "task" is
http://bugs.python.org/issue24475

#24466: extend_path explanation in documentation is ambiguous
http://bugs.python.org/issue24466

#24424: xml.dom.minidom: performance issue with Node.insertBefore()
http://bugs.python.org/issue24424

#24417: Type-specific documentation for __format__ methods
http://bugs.python.org/issue24417

#24415: SIGINT always reset to SIG_DFL by Py_Finalize()
http://bugs.python.org/issue24415

#24414: MACOSX_DEPLOYMENT_TARGET set incorrectly by configure
http://bugs.python.org/issue24414

#24407: Use after free in PyDict_merge
http://bugs.python.org/issue24407

#24381: Got warning when compiling ffi.c on Mac
http://bugs.python.org/issue24381

#243

Re: [Python-Dev] Importance of "async" keyword

[Chris Angelico]

On Sat, Jun 27, 2015 at 2:07 AM, R. David Murray  wrote:
> On Sat, 27 Jun 2015 01:10:33 +1000, Chris Angelico  wrote:
>> The way I'm seeing it, coroutines are like cooperatively-switched
>> threads; you don't have to worry about certain operations being
>> interrupted (particularly low-level ones like refcount changes or list
>> growth), but any time you hit an 'await', you have to assume a context
>> switch. That's all very well, but I'm not sure it's that big a problem
>> to accept that any call could context-switch; atomicity is already a
>> concern in other cases, which is why we have principles like EAFP
>> rather than LBYL.
>
> Read Glyph's article, it explains why:
>
> https://glyph.twistedmatrix.com/2014/02/unyielding.html

Makes some fair points, but IMO it emphasizes what I'm saying about
atomicity. If you really need it, establish it by something other than
just having code that naively progresses through. That's what
databasing is for, after all. Threading and signals force you to think
about concurrency; other models *may* allow you to pretend it doesn't
exist.

Anyway, that answers my question about why the explicit "this can let
other things run" marker. Thanks all.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importance of "async" keyword

[Nick Coghlan]

On 26 June 2015 at 23:48, Sven R. Kunze  wrote:
> @Nick
> Thanks for these links; nice reads and reflect exactly what I think about
> these topics. Btw. complex numbers basically works the same way (same API)
> as integers.

Not using complex numbers in Python - coming to grips with what they
mean physically.

> I would like to see that for functions and awaitables as well.

This is akin to asking for unification of classes and modules -
they're both namespaces so they have a lot of similarities, but the
ways in which they're different are by design.

Similarly, subroutines and coroutines are not the same thing:
https://en.wikipedia.org/wiki/Coroutine#Comparison_with_subroutines

They're *related*, but they're still not the same thing.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importance of "async" keyword

[Steve Dower]

On 06/26/2015 06:48 AM, Sven R. Kunze wrote:

> def business():
>  return complex_calc(5)
>
> def business_new()
>  return await complex_calc(10)

> Maybe, I completely missed the point of the proposal, but this is the way I 
> would expect it to work. Putting in an 'await' whenever I see fit and it just 
> works.

Assuming "async def business_new" (to avoid the syntax error), there's no 
difference between those functions or the one they're calling:

* "complex_calc" returns an awaitable object that, after you've awaited it, 
will result in an int.
* "business" returns the return value of "complex_calc", which is an awaitable 
object that, after you've awaited it, will result in an int.
* "business_new" returns an awaitable object that, after you've awaited it, 
will result in an int.

In all three of these cases, the result is the same. The fact that the 
awaitable object returned from any of them is implemented by a coroutine isn't 
important (in the same way that an iterable object may be implemented by a 
generator, but it's really irrelevant).

The value of the await syntax is when you're doing something interesting, 
a.k.a. your function is more than delegating directly to another function:

async def business_async():
tasks = [complex_calc_async(i) for i in range(10)]
results = [await t for t in tasks]
await write_to_disk_async(filename, results)
return sum(results)

Now it's actually useful to be able to await when we choose. Each call to 
complex_calc_async() could be starting a thread and then suspending until the 
thread is complete, so we actually start all 10 threads running here before 
blocking, and then collect the results in the order we started them (and not 
the order they complete, though I think asyncio has a function to do that). 
Without the explicit await this would be impossible.

The async def also lets us create coroutines consistently even if they don't 
await anything:

if has_version:
async def get_version_async():
return VERSION
else:
async def get_version_async():
return (await get_major_version_async(), await 
get_minor_version_async())

async def show_version():
print(await get_version_async())

If, like generators, regular functions became coroutines only in the presence 
of an await, we'd have to do convoluted code to produce the fast 
get_version_async(), or else make the caller worry about whether they can skip 
the await. (Also consider calls that cache their result - a coroutine MUST be 
awaited, but it doesn't have to await anything if it already has the result).

(Aside: the "_async" suffix is based on the convention used in C#, and it's 
certainly one that I'll be using throughout my async Python code and 
encouraging in any code that I review. It's the most obvious way to let callers 
know whether they need to await the function result or not.)

Cheers,
Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importance of "async" keyword

[Nick Coghlan]

On 27 June 2015 at 00:31, Chris Angelico  wrote:
> I come from a background of thinking with threads, so I'm accustomed
> to doing blocking I/O and assuming/expecting that other threads will
> carry on while we wait. If asynchronous I/O can be made that
> convenient, it'd be awesome.

Folks, it's worth reading through both
https://glyph.twistedmatrix.com/2014/02/unyielding.html and
http://python-notes.curiousefficiency.org/en/latest/pep_ideas/async_programming.html

It *is* possible to make a language where *everything* is run as a
coroutine, and get rid of the subroutine/coroutine distinction that
way - a subroutine would *literally* be executed as a coroutine that
never waited for anything. The "doesn't suspend" case could then be
handled as an implicit optimisation, rather than as a distinct type,
and the *only* way to do IO would be asynchronously through an event
loop, rather than through blocking API calls.

But that language wouldn't be Python. Python's core execution model is
a synchronous procedural dynamically typed one, with everything beyond
that, whether object oriented programming, functional programming,
asynchronous programming, gradual typing, etc, being a way of managing
the kinds of complexity that arise when managing more state, or more
complicated algorithms, or more interactivity, or a larger development
team.

> But since it hasn't already been made that easy in every other
> language, I expect there's some fundamental problem with this
> approach, something that intrinsically requires every step in the
> chain to know what's a (potential) block point.

As Glyph explains, implicitly switched coroutines are just shared
memory threading under another name - when any function call can cause
you to lose control of the flow of execution, you have to consider
your locking model in much the same way as you do for preemptive
threading.

For Python 2, this "lightweight threads" model is available as
https://pypi.python.org/pypi/gevent, and there also appears to have
been some good recent progress on Python 3 compatibility:
https://github.com/gevent/gevent/issues/38

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importance of "async" keyword

[Ethan Furman]

On 06/26/2015 08:47 AM, Steve Dower wrote:

On 06/26/2015 06:48 AM, Sven R. Kunze wrote:


def business():
  return complex_calc(5)

def business_new()
  return await complex_calc(10)



Assuming "async def business_new" (to avoid the syntax error), there's no 
difference between those functions or the one they're calling:

* "complex_calc" returns an awaitable object that, after you've awaited it, 
will result in an int.
* "business" returns the return value of "complex_calc", which is an awaitable 
object that, after you've awaited it, will result in an int.
* "business_new" returns an awaitable object that, after you've awaited it, 
will result in an int.

In all three of these cases, the result is the same. The fact that the 
awaitable object returned from any of them is implemented by a coroutine isn't 
important (in the same way that an iterable object may be implemented by a 
generator, but it's really irrelevant).


What?  Shouldn't 'business_new' return the int?  It did await, after all.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importance of "async" keyword

[Ron Adam]

On 06/26/2015 10:31 AM, Chris Angelico wrote:

Apologies if this is a really REALLY dumb question, but... How hard
would it be to then dispense with the await keyword, and simply
_always_  behave that way? Something like:

def data_from_socket():
 # Other tasks may run while we wait for data
 # The socket.read() function has yield points in it
 data = socket.read(1024, 1)
 return transmogrify(data)

def respond_to_socket():
 while True:
 data = data_from_socket()
 # We can pretend that socket writes happen instantly,
 # but if ever we can't write, it'll let other tasks wait while
 # we're blocked on it.
 socket.write("Got it, next please!")

Do these functions really need to be aware that there are yield points
in what they're calling?


I think "yield points" is a concept that needs to be spelled out a bit 
clearer in the PEP 492.


It seems that those points are defined by other means outside of a function 
defined with "async def".  From the PEP...


   * It is a SyntaxError to have yield or yield from expressions
 in an async function.

So somewhere in an async function, it needs to "await something" with a 
yield in it that isn't an async function.


This seems to be a bit counter intuitive to me.  Or am I missing something?

Regards,
   Ron




















___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importance of "async" keyword

[Yury Selivanov]

On 2015-06-26 1:40 PM, Ethan Furman wrote:

On 06/26/2015 08:47 AM, Steve Dower wrote:

On 06/26/2015 06:48 AM, Sven R. Kunze wrote:


def business():
  return complex_calc(5)

def business_new()
  return await complex_calc(10)


Assuming "async def business_new" (to avoid the syntax error), 
there's no difference between those functions or the one they're 
calling:


* "complex_calc" returns an awaitable object that, after you've 
awaited it, will result in an int.
* "business" returns the return value of "complex_calc", which is an 
awaitable object that, after you've awaited it, will result in an int.
* "business_new" returns an awaitable object that, after you've 
awaited it, will result in an int.


In all three of these cases, the result is the same. The fact that 
the awaitable object returned from any of them is implemented by a 
coroutine isn't important (in the same way that an iterable object 
may be implemented by a generator, but it's really irrelevant).


What?  Shouldn't 'business_new' return the int?  It did await, after all. 


"business_new" should be defined with an 'async' keyword, that's where 
all the confusion came from:


  async def business_new():
 return await complex_calc(10)

Now, "business_new()" returns a coroutine (which will resolve to the 
result of "complex_calc" awaitable), "await business_new()" will return 
an int.



Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importance of "async" keyword

[Steve Dower]

Ethan Furman wrote:
> On 06/26/2015 08:47 AM, Steve Dower wrote:
>> On 06/26/2015 06:48 AM, Sven R. Kunze wrote:
>>
>>> def business():
>>> return complex_calc(5)
>>>
>>> def business_new()
>>> return await complex_calc(10)
> 
>> Assuming "async def business_new" (to avoid the syntax error), there's no
>> difference between those functions or the one they're calling:
>>
>> * "complex_calc" returns an awaitable object that, after you've awaited it,
>> will result in an int.
>> * "business" returns the return value of "complex_calc", which is an 
>> awaitable
>> object that, after you've awaited it, will result in an int.
>> * "business_new" returns an awaitable object that, after you've awaited it,
>> will result in an int.
>>
>> In all three of these cases, the result is the same. The fact that the
>> awaitable object returned from any of them is implemented by a coroutine 
>> isn't
>> important (in the same way that an iterable object may be implemented by a
>> generator, but it's really irrelevant).
> 
> What? Shouldn't 'business_new' return the int? It did await, after all.

I assumed "async def business_new()", rather than some imaginary "await in a 
non-async function will block because I love to create deadlocks in my code" 
feature.

Note that blocking prevents *all* coroutines from making progress, unlike 
threading. When you await all the way to an event loop, it defers the rest of 
the coroutine until a signal (via a callback) is raised and continues running 
other coroutines.

Cheers,
Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importance of "async" keyword

[Nick Coghlan]

On 27 June 2015 at 04:06, Ron Adam  wrote:
> It seems that those points are defined by other means outside of a function
> defined with "async def".  From the PEP...
>
>* It is a SyntaxError to have yield or yield from expressions
>  in an async function.
>
> So somewhere in an async function, it needs to "await something" with a
> yield in it that isn't an async function.

This isn't the case - it can be async functions and C level coroutines
all the way down. Using a generator or other iterable instead requires
adaptation to the awaitable protocol (which makes it possible to tap
into all the work that has been done for the generator-based
coroutines used previously, rather than having to rewrite it to use
native coroutines).

This isn't very clear in the currently released beta as we made some
decisions to simplify the original implementation that we thought
would also be OK from an API design perspective, but turned out to
pose significant problems once folks actually started trying to
integrate native coroutines with other async systems beyond asyncio.

Yury fixed those underlying object model limitations as part of
addressing Ben Darnell's report of problems attempting to integrate
native coroutine support into Tornado
(https://hg.python.org/cpython/rev/7a0a1a4ac639).

Yury had already updated the PEP to account for those changes, but
I've now also added a specific note regarding the API design change in
response to beta feedback: https://hg.python.org/peps/rev/0c963fa25db8

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importance of "async" keyword

[Jim J. Jewett]

On Fri Jun 26 16:51:13 CEST 2015, Paul Sokolovsky wrote:

> So, currently in Python you know if you do:

>socket.write(buf)

> Then you know it will finish without interruptions for entire buffer.

How do you know that?

Are you assuming that socket.write is a builtin, rather than a
python method?  (Not even a python wrapper around a builtin?)

Even if that were true, it would only mean that the call itself
is processed within a single bytecode ... there is no guarantee
that the write method won't release the GIL or call back into
python (and thereby allow a thread switch) as part of its own
logic.

> And if you write:

>await socket.write(buf)

> then you know there may be interruption points inside socket.write(),
> in particular something else may mutate it while it's being written.

I would consider that external mutation to be bad form ... at least
as bad as violating the expectation of an atomic socket.write() up
above.

So either way, nothing bad SHOULD happen, but it might anyhow.  I'm
not seeing what the async-coloring actually bought you...

-jJ

--

If there are still threading problems with my replies, please
email me with details, so that I can try to resolve them.  -jJ
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com