Asynchronous processing is more efficient -- surely not?

[Steven D'Aprano]

So, I'm, still trying to wrap my brain around async processing, and I 
started reading this tutorial:

http://stackabuse.com/python-async-await-tutorial/

and the very first paragraph broke my brain.

"Asynchronous programming has been gaining a lot of traction in the past 
few years, and for good reason. Although it can be more difficult than 
the traditional linear style, it is also much more efficient."

I can agree with the first part of the first sentence (gaining a lot of 
traction), and the first part of the second sentence (more difficult than 
the traditional style), but the second part? Asynchronous processing is 
*more efficient*?

I'm no expert, but it seems to me that this has surely got to be crazy 
talk. Whatever task you're doing, processing it asynchronously doesn't 
reduce the amount of work. For example, if you want to download ten 
files, you still have to download all ten files, and they're not any 
smaller or the network connection any faster because you're using async.

On the contrary, surely it is *less efficient*: there's the amount of 
work you need to do, the *useful* work (downloading those files) PLUS a 
bunch of book-keeping work (most, but not all, done for you in the async 
framework or language) needed to swap jobs.

I can see that the advantage of async is to *increase responsiveness*. 
You aren't waiting a long time for each file to download (or the page to 
render, or whatever task it is that you are doing) before being able to 
start the next one. Quoting the article:

"With asynchronous programming, you allow your code to handle other tasks 
while waiting for these other resources to respond."

Yes, this exactly. And if you're writing a web app, or certain kinds of 
desktop apps, these seems sensible. I don't want my browser to come to a 
complete halt just because some resource is taking a few seconds to 
respond.

But honestly, from everything I've seen, outside of browser apps (and I'm 
not even sure about them), async processing looks to me like the wrong 
solution to, well, everything. It combines the worst of sequential and 
parallel processing with the benefits of neither. It's like cooperative 
multi-tasking, only without the actual multi-tasking and even less 
convenience. Its harder to use than threading or multiprocessing, but 
without their ability to run in parallel. It lacks the isolation of 
multiprocessing, and the preemptive task swapping of threads/processes.

Its as hard to wrap your brain around as parallel processing in general, 
but with even worse performance than sequential processing.

Am I totally wrong?


-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous processing is more efficient -- surely not?

[INADA Naoki]

I agree with you.

Async IO is more efficient than threading for **waiting** I/O.
When there are thousands of idle connections, async I/O is best idea.

On the other hand, async I/O uses more system calls for busy I/O.

For example, when building chat application which handles thousands
WebSocket connection and uses MySQL as backend, I recommend to
use asyncio for WebSocket, but threadpool for transaction.

Regards,
-- 
https://mail.python.org/mailman/listinfo/python-list

OSError: [Errno -9981] Input overflowed

[asphjt--- via Python-list]

Here is my code:

import pyaudio
tim=1
chunk = 8
FORMAT = pyaudio.paInt16
CHANNELS = 1
RATE = 5
p = pyaudio.PyAudio()
s = p.open(format = FORMAT,channels = CHANNELS,rate = RATE,input = 
True,output=True,frames_per_buffer = chunk)
d=[]
print((RATE // chunk) * tim)
for i in range(0, (RATE // chunk * tim)):
data = s.read(chunk)
d.append(data)
s.close()
p.terminate()


There is no problem, but if I want to use the variable data during the loop 
for, I have this error:

Traceback (most recent call last):
  File "", line 14, in 
data = s.read(chunk)
  File "c:\users\tissot\miniconda3\lib\site-packages\pyaudio.py", line 608, in 
read
return pa.read_stream(self._stream, num_frames, exception_on_overflow)
OSError: [Errno -9981] Input overflowed


for example if i want to print data with this code:

import pyaudio
tim=1
chunk = 8
FORMAT = pyaudio.paInt16
CHANNELS = 1
RATE = 5
p = pyaudio.PyAudio()
s = p.open(format = FORMAT,channels = CHANNELS,rate = RATE,input = 
True,output=True,frames_per_buffer = chunk)
d=[]
print((RATE // chunk) * tim)
for i in range(0, (RATE // chunk * tim)):
data = s.read(chunk)
d.append(data)
print(data)# here is the problem
s.close()
p.terminate()

In reality I would like to send it to my Arduino card as and when 
recording,(using: ser = serial.Serial('COM4', 115200)   ser.write(data) ) but I 
have the same error as when I display it on the screen.
If anyone knows an answer to this question, thank you in advance.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous processing is more efficient -- surely not?

[Jan Erik Moström]

On 4 Apr 2018, at 9:27, Steven D'Aprano wrote:

Its as hard to wrap your brain around as parallel processing in 
general,

but with even worse performance than sequential processing.

Am I totally wrong?


I would say that it all depends on what kind of stuff you're doing. I'm 
no scheduling expert but our high performance centre run their jobs in 
batch mode, every job is allowed to finish before it removed from the 
processor (assuming they keep inside their time quota) but they are 
hight compute intensive jobs with little I/O.


On the other hand, a desktop computer probably have "reactive jobs" with 
a high ratio of I/O which would make it desirable to have threads/async 
processing.


Other systems are probably between these two cases. So saying that it's 
always good is probably not true but sometimes it can be a win.


Also, assuming you have threads/processes and a multicore system the 
scheduling might take advantage of this giving higher performance ... 
I'm haven't thought about how co-routines would work here.


(another plus for threads is that *I* think that the logical 
division/coding of the system becomes easier if threads and IPC is used 
... but I don't have any data to support this - and yes I know that 
threads/processes/synchronization open up its own can of worms)


= jem
--
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous processing is more efficient -- surely not?

[Paul Moore]

On 4 April 2018 at 08:27, Steven D'Aprano
 wrote:
> "Asynchronous programming has been gaining a lot of traction in the past
> few years, and for good reason. Although it can be more difficult than
> the traditional linear style, it is also much more efficient."
>
> I can agree with the first part of the first sentence (gaining a lot of
> traction), and the first part of the second sentence (more difficult than
> the traditional style), but the second part? Asynchronous processing is
> *more efficient*?

I'd need to know what "efficient" meant. Obviously you're never going
to get more than 100% utilisation of a single core with async (because
of the GIL) but I can easily imagine async making more complete use of
that core by having less time spent waiting for I/O. Whether you
describe that as "more efficient" use of the CPU, or something else, I
don't know.

Honestly, that paragraph reads more like sales blurb than anything
else, so I'd be inclined to take it with a pinch of salt anyway. IMO,
async has proved useful for handling certain types of IO bound
workloads with lower overheads[1] than traditional multi-threaded or
multi-process designs. Whether it's a good fit for any particular
application is something you'd have to test, as with anything else.

Paul

[1] I found it really hard to avoid saying "more efficiently" there.
Not sure what that implies other than that the phrase means whatever
you want it to mean!!!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous processing is more efficient -- surely not?

[Chris Angelico]

On Wed, Apr 4, 2018 at 5:27 PM, Steven D'Aprano
 wrote:
> So, I'm, still trying to wrap my brain around async processing, and I
> started reading this tutorial:
>
> http://stackabuse.com/python-async-await-tutorial/
>
> and the very first paragraph broke my brain.
>
> "Asynchronous programming has been gaining a lot of traction in the past
> few years, and for good reason. Although it can be more difficult than
> the traditional linear style, it is also much more efficient."
>
> I can agree with the first part of the first sentence (gaining a lot of
> traction), and the first part of the second sentence (more difficult than
> the traditional style), but the second part? Asynchronous processing is
> *more efficient*?

The only way it could be "more efficient" is if you assume that
"sequential processing" == "well, let's parallelize it by going for
multiple processes, each independently started". This is an
assumption, but it's not wholly unreasonable - if you write your code
in a purely sequential way (say, youtube-dl) and want to do more than
one of these at a time, the easiest way is to pop open another
terminal and fire up another copy of the program, with an entire copy
of everything. But that isn't inherent to the design of sequential
processing. It's a cost of a naive form of parallelism, one that could
be alleviated by threading too.

IMO asyncio should be compared to threads, not to sequential
processing. *First* imagine reworking your code to use threads; then
consider cutting out the actual OS-managed threads and using a
cooperative single-threaded system instead. You depend on
application-level scheduling, which means that gethostbyname() can
utterly break you if you aren't careful, and you lose about thirty or
forty years of collective experience in developing operating systems
with intelligent schedulers, but in return, you gain the ability to
manage a hundred thousand concurrent tasks in a single process. You
also lose the ability to context-swap in the middle of a logical
operation, but in return, you lose the ability to context-swap in the
middle of a logical operation. For people who've grown up with actual
real threads, this latter is not an advantage; for people who've grown
up casually mutating global state, that's huge.

IN THEORY, it would be possible to create a lock-free implementation
of Python that is incapable of using multiple threads, but still able
to use asyncio. In practice, since CPython has a highly efficient
locking system (the GIL), the advantage is unlikely to be compelling.

So, no, generally async processing will NOT be more efficient than
sequential processing. It has other advantages, but not that.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous processing is more efficient -- surely not?

[Richard Damon]

On 4/4/18 3:27 AM, Steven D'Aprano wrote:
> So, I'm, still trying to wrap my brain around async processing, and I 
> started reading this tutorial:
>
> http://stackabuse.com/python-async-await-tutorial/
>
> and the very first paragraph broke my brain.
>
> "Asynchronous programming has been gaining a lot of traction in the past 
> few years, and for good reason. Although it can be more difficult than 
> the traditional linear style, it is also much more efficient."
>
> I can agree with the first part of the first sentence (gaining a lot of 
> traction), and the first part of the second sentence (more difficult than 
> the traditional style), but the second part? Asynchronous processing is 
> *more efficient*?
>
> I'm no expert, but it seems to me that this has surely got to be crazy 
> talk. Whatever task you're doing, processing it asynchronously doesn't 
> reduce the amount of work. For example, if you want to download ten 
> files, you still have to download all ten files, and they're not any 
> smaller or the network connection any faster because you're using async.
>
> On the contrary, surely it is *less efficient*: there's the amount of 
> work you need to do, the *useful* work (downloading those files) PLUS a 
> bunch of book-keeping work (most, but not all, done for you in the async 
> framework or language) needed to swap jobs.
>
> I can see that the advantage of async is to *increase responsiveness*. 
> You aren't waiting a long time for each file to download (or the page to 
> render, or whatever task it is that you are doing) before being able to 
> start the next one. Quoting the article:
>
> "With asynchronous programming, you allow your code to handle other tasks 
> while waiting for these other resources to respond."
>
> Yes, this exactly. And if you're writing a web app, or certain kinds of 
> desktop apps, these seems sensible. I don't want my browser to come to a 
> complete halt just because some resource is taking a few seconds to 
> respond.
>
> But honestly, from everything I've seen, outside of browser apps (and I'm 
> not even sure about them), async processing looks to me like the wrong 
> solution to, well, everything. It combines the worst of sequential and 
> parallel processing with the benefits of neither. It's like cooperative 
> multi-tasking, only without the actual multi-tasking and even less 
> convenience. Its harder to use than threading or multiprocessing, but 
> without their ability to run in parallel. It lacks the isolation of 
> multiprocessing, and the preemptive task swapping of threads/processes.
>
> Its as hard to wrap your brain around as parallel processing in general, 
> but with even worse performance than sequential processing.
>
> Am I totally wrong?
>
>
Asynchronous processing will use a bit more of some processing resources
to handle the multi-processing, but it can be more efficient at fully
using many of the resources that are available.

Take your file download example. When you are downloading a file, your
processor sends out a request for a chuck of the data, then waits for
the response. The computer on the other end gets that requests, sends a
packet, and then waits. You get that packet, send and acknowledgement,
and then wait, the other computer gets that acknowledgement and sends
more data, and then waits, and so on. Even if your pipe to the Internet
is the limiting factor, there is a fair amount of dead time in this
operation, so starting another download to fill more of that pipe can
decrease the total time to get all the data downloaded.

Yes, if your single path of execution can fully use the critical
resources, then adding asynchronous processing won't help, but rarely
does it. Very few large machines today a single-threaded, but most have
multiple cores and often even those cores have the ability to handle
multiple threads at once. Thus there normally are extra resources that
the asynchronous processing can better use up, so even processor usage
can be improved in many cases. In most cases I am familiar with, the
type of asynchronous programming you are talking about is to move I/O
bound operations into a second execution path, allowing your main path
to focus on keeping the CPU busy.

-- 
Richard Damon

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous processing is more efficient -- surely not?

[Chris Angelico]

On Wed, Apr 4, 2018 at 8:42 PM, Paul Moore  wrote:
> IMO,
> async has proved useful for handling certain types of IO bound
> workloads with lower overheads[1] than traditional multi-threaded or
> multi-process designs. Whether it's a good fit for any particular
> application is something you'd have to test, as with anything else.

This I would agree with. There are certain types of tasks that really
lend themselves spectacularly well to async I/O models - mainly those
that are fundamentally reactive and can have inordinate numbers of
connected users. Take a chat room, for example. The client establishes
a connection to a server (maybe IRC, maybe a WebSocket, whatever), and
the server just sits there doing nothing until some client sends a
message. Then the server processes the message, does whatever it
thinks is right, and sends messages out to one or more clients. It's
essential that multiple concurrent clients be supported (otherwise
you're chatting with yourself), so how do the different models hold
up?

1) Multiple independent processes. Abysmal; they have to communicate
with each other, so this would require a measure of persistence (eg a
database) and some means of signalling other processes. A pain to
write, and horribly inefficient. You probably would need two threads
per process (one to read, one to write).

2) The multiprocessing module. Better than the above because you can
notify other processes using a convenient API, but you still need an
entire process for every connected client. Probably you'll top out at
a few hundred clients, even if they're all quiet. Still need two
threads per process.

3) Native OS threads using the threading module. Vast improvement; the
coding work would be pretty much the same as for multiprocessing, but
instead of a process, you need a thread. Probably would top out at a
few thousand clients, maybe a few tens of thousands, as long as
they're all fairly quiet. Since all state is now shared, you now need
only one thread per process (its reading thread), and writing is done
in the thread that triggered it. Everything that's global is now
stored in just one place.

4) Asynchronous I/O with an event loop. Small improvement over the
above; now that there's no OS threads involved, you're now limited
only by available memory. You could have tens of millions of connected
clients as long as they're all quiet. Concurrency is now 100% in the
programmer's hands (with explicit yield points), instead of having
automatic yield points any time a thread blocks for any reason; this
restricts parallelism to the points where actual I/O is happening. One
DNS lookup can bring you down, but 100K connected sockets would be no
problem at all.

Async I/O certainly has its place, but the performance advantages
don't really kick in until you're scaling to ridiculous levels of
dormant clients. (If you have a large number of *active* clients, your
actual work is going to be more significant, and you'll need to spend
more time in the CPU, making multiple processes look more attractive.)
Its biggest advantages are in _code simplicity_, not performance. (And
only for people who can't wrap their head around threads; if you're
fluent in threads, the simplicity is comparable, so there's less
advantage.)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous processing is more efficient -- surely not?

[Chris Angelico]

On Wed, Apr 4, 2018 at 9:02 PM, Richard Damon  wrote:
> Asynchronous processing will use a bit more of some processing resources
> to handle the multi-processing, but it can be more efficient at fully
> using many of the resources that are available.
>
> Take your file download example. When you are downloading a file, your
> processor sends out a request for a chuck of the data, then waits for
> the response. The computer on the other end gets that requests, sends a
> packet, and then waits. You get that packet, send and acknowledgement,
> and then wait, the other computer gets that acknowledgement and sends
> more data, and then waits, and so on. Even if your pipe to the Internet
> is the limiting factor, there is a fair amount of dead time in this
> operation, so starting another download to fill more of that pipe can
> decrease the total time to get all the data downloaded.

Assuming that you're downloading this via a TCP/IP socket (eg from a
web server), the acknowledgements are going to be handled by the OS
kernel, not your process. Plus, TCP allows acknowledgements to stack,
so you're not really waiting for each other's acks very much. A single
socket is entirely capable of saturating one computer's uplink. I once
proved to my employer that a particular host had gigabit internet by
renting a dozen EC2 instances with 100Mbit uplinks and having each of
them transfer data to the same host concurrently - via one socket
connection each.

Much more interesting is operating a high-bandwidth server (let's say,
a web application) that is responding to requests from myriad
low-bandwidth clients. Traditional servers such as Apache's prefork
mode would follow a model like this:

while "more sockets":
newsock = mainsock.accept()
fork_to_subprocess(handler, newsock)

def handler(sock):
while "need headers":
sock.receive_headers()
while "need body":
sock.receive_body()
generate_response()
while "response still sending":
sock.send_response()

A threaded model does the same thing, but instead of forking a
subprocess, it spawns a thread. The handler is pretty simple and
straight-forward; it reads from the socket until it has everything it
needs, then it sends off a response. Both reading and writing can and
will block, and generating the response is the only part that's really
CPU-bound.

"Parallelism" here means two things: how many active clients can you
support (throughput), and how many dormant clients can you support
(saturation). In a forked model, you spend a lot of resources spinning
up processes (you can reduce this with process pools and such, at the
expense of code complexity and a slower spin-down when idle); in a
threaded model, you spend far less, but you're still paying a
significant price per connection, and saturation can be a problem. The
beauty of async I/O is that saturation becomes almost completely
insignificant; the cost is that throughput is capped at a single
thread's capabilities.

In theory, you could use async I/O with multiple threads pumping the
same set of events. I'm not sure if anyone has ever actually done
this, as it combines the complexities of both models, but it would
maximize both throughput and saturation levels - dormant clients cost
very little, and you're able to use multiple CPU cores. More commonly,
you could run a thread pool, doling out clients to whichever thread is
least busy, and then having each thread run an independent event loop,
which would be fine in the average case. But that's still more
complicated; you still have to think about threads.

> Yes, if your single path of execution can fully use the critical
> resources, then adding asynchronous processing won't help, but rarely
> does it. Very few large machines today a single-threaded, but most have
> multiple cores and often even those cores have the ability to handle
> multiple threads at once. Thus there normally are extra resources that
> the asynchronous processing can better use up, so even processor usage
> can be improved in many cases.

I'm not sure what tasks would allow you to reduce processor usage this
way. Got an example?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous processing is more efficient -- surely not?

[Julien Salort]

Le 04/04/2018 à 09:27, Steven D'Aprano a écrit :


Yes, this exactly. And if you're writing a web app, or certain kinds of
desktop apps, these seems sensible. I don't want my browser to come to a
complete halt just because some resource is taking a few seconds to
respond.

But honestly, from everything I've seen, outside of browser apps (and I'm
not even sure about them), async processing looks to me like the wrong
solution to, well, everything.
For what it's worth, I can give you an example that I have been playing 
with lately, and it has nothing to do with browser app.
I have been doing scientific image acquisition with Python for some 
time, and was quite happy with my program.


But then, I suddently needed to fetch from several cameras at once, and 
also query another device at another interval.
Of course, it is possible to do that in a simple loop without asyncio, 
but I thought I would give asyncio a try, and I was amazed at how easy 
it makes this kind of task. And then it is simple to remove/add a 
camera, etc.
My only concern is that I tend to have code duplication between the 
async version of the program and the sync version of the program.


In this case, the script spends most of its time waiting for a frame to 
be available on the cameras, and the time interval to query the other 
device. The fetching and processing of the frames take negligible time. 
The library that I use is not asynchronous, but it was very easy to 
run_in_executor the C function that blocks until a frame is available on 
a given camera.


Compared to a non-async version where I would have had to use that C 
function with some short timeout and iterate over all cameras, I think 
the async version is more efficient. The await will just be as long as 
it takes for the WaitFrame C function to run... When the C function 
ends, the asynchronous task resumes... There is no need to adjust some 
timeout by hand.


Bests,

Julien
--
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous processing is more efficient -- surely not?

[Chris Angelico]

On Wed, Apr 4, 2018 at 10:16 PM, Julien Salort  wrote:
> In this case, the script spends most of its time waiting for a frame to be
> available on the cameras, and the time interval to query the other device.
> The fetching and processing of the frames take negligible time. The library
> that I use is not asynchronous, but it was very easy to run_in_executor the
> C function that blocks until a frame is available on a given camera.
>
> Compared to a non-async version where I would have had to use that C
> function with some short timeout and iterate over all cameras, I think the
> async version is more efficient. The await will just be as long as it takes
> for the WaitFrame C function to run... When the C function ends, the
> asynchronous task resumes... There is no need to adjust some timeout by
> hand.

Can you give an example? Let's say we have a simple blocking C function:

int get_data() {
sleep(2);
return 42;
}

How do you use run_in_executor to turn this asynchronous, and how
would this compare to creating one thread for each camera? AIUI,
run_in_executor uses a thread pool under the surface, so all you're
doing is using threads via an asyncio interface. Comparing to a
timeout implementation is unfair; a threaded implementation is more
comparable.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous processing is more efficient -- surely not?

[Matěj Cepl]

On 2018-04-04, 07:27 GMT, Steven D'Aprano wrote:
> I'm no expert, but it seems to me that this has surely got to 
> be crazy talk. Whatever task you're doing, processing it 
> asynchronously doesn't reduce the amount of work. For example, 
> if you want to download ten files, you still have to download 
> all ten files, and they're not any smaller or the network 
> connection any faster because you're using async.

I agree that the formulation is unfortunate, but your argument 
seems to lie only on semantics. Anyway, this 
https://hackernoon.com/asynchronous-python-45df84b82434 seems to 
be a better explanation of cooperative green threads and 
eventually also asyncio.

Best,

Matěj
-- 
https://matej.ceplovi.cz/blog/, Jabber: mc...@ceplovi.cz
GPG Finger: 3C76 A027 CA45 AD70 98B5  BC1D 7920 5802 880B C9D8
  
[...] sleep is no substitute for caffeine.
   -- Robert Storey in review of Debian
  (when describing re-compilation of kernel :-)


pgpRacL75wPhc.pgp
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous processing is more efficient -- surely not?

[Julien Salort]

Le 04/04/2018 à 14:45, Chris Angelico a écrit :


Can you give an example? Let's say we have a simple blocking C function:
int get_data() {
 sleep(2);
 return 42;
}


I am not saying that I understand 100% and that this is the best way, 
but it works for me:


% cat get_data.c
#include 

int get_data() {
    sleep(2);
    return 42;
}

% clang -shared -o libgetdata.dylib get_data.c

% cat get_data_async.py
import ctypes
import asyncio

mylib = ctypes.cdll.LoadLibrary('libgetdata.dylib')

loop = asyncio.get_event_loop()


async def get_data():
    result = await loop.run_in_executor(None, mylib.get_data)
    print('C function returned', result)
    return result


async def another_task():
    for i in range(5):
    print(i)
    await asyncio.sleep(1)


loop.run_until_complete(asyncio.gather(get_data(), another_task()))
loop.close()

% python get_data_async.py
0
1
C function returned 42
2
3
4


How do you use run_in_executor to turn this asynchronous, and how
would this compare to creating one thread for each camera?
This is exactely like creating a thread. Except that I have to do so 
only for blocking calls and without having to bother myself with threads 
or thread pools.

AIUI,
run_in_executor uses a thread pool under the surface, so all you're
doing is using threads via an asyncio interface. Comparing to a
timeout implementation is unfair; a threaded implementation is more
comparable.

Agreed.
It is just that it looks very simple to me. But I have never really done 
any asynchronous programming before. So, maybe using threads is just as 
simple, I don't know. What I find nice with asyncio is that it 
integrates easily with already written Python code, i.e. converting 
synchronous code to asynchronous code is relatively straightforward. 
Again, my problem is that it leads to code duplication. But that 
probably means that I should separate the logic into separate functions 
more.


Bests,

Julien
--
https://mail.python.org/mailman/listinfo/python-list

Re: julian 0.14 library

[Thomas Jollans]

On 2018-04-04 05:44, Chris Angelico wrote:
> On Wed, Apr 4, 2018 at 12:24 PM, sum abiut  wrote:
>> Hi,
>> Has anyone try this https://pypi.python.org/pypi/julian/0.14
>>
>> i got this error trying to import julian
>>
> import julian
>> Traceback (most recent call last):
>>   File "", line 1, in 
>>   File "/usr/local/lib/python2.7/dist-packages/julian/__init__.py", line 1,
>> in 
>> from julian.julian import to_jd, from_jd
>>   File "/usr/local/lib/python2.7/dist-packages/julian/julian.py", line 5
>> def __to_format(jd: float, fmt: str) -> float:
>>   ^
>> SyntaxError: invalid syntax
>>
> 
> Looks like that package requires Python 3, but was uploaded to PyPI
> without any version tags. You could try running it in Python 3.x, but
> there's no way to know which ".x" versions are going to work.

the cheeseshop description says 3.2+
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous processing is more efficient -- surely not?

[Chris Angelico]

On Thu, Apr 5, 2018 at 1:48 AM, Julien Salort  wrote:
> Le 04/04/2018 à 14:45, Chris Angelico a écrit :
>> How do you use run_in_executor to turn this asynchronous, and how
>> would this compare to creating one thread for each camera?
>
> This is exactely like creating a thread. Except that I have to do so only
> for blocking calls and without having to bother myself with threads or
> thread pools.
> It is just that it looks very simple to me. But I have never really done any
> asynchronous programming before. So, maybe using threads is just as simple,
> I don't know. What I find nice with asyncio is that it integrates easily
> with already written Python code, i.e. converting synchronous code to
> asynchronous code is relatively straightforward. Again, my problem is that
> it leads to code duplication. But that probably means that I should separate
> the logic into separate functions more.

Okay, that's fair. So you're basically using asyncio as a means of
managing threads, without actually using threads. I do recommend
looking into the threading module and getting a feel for actual
threaded programming, but if you don't, so be it.

In any case, asyncio is functioning as a wrapper around a thread pool;
so it's never going to be more efficient or more effective than an
actual thread pool, but might be easier to work with in your code.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: julian 0.14 library

[sum abiut]

I got the error  below, tryinig in on python 3.2.

import julian
Traceback (most recent call last):
  File "", line 1, in 
ImportError: No module named julian


On Thu, Apr 5, 2018 at 3:21 AM, Thomas Jollans  wrote:

> On 2018-04-04 05:44, Chris Angelico wrote:
> > On Wed, Apr 4, 2018 at 12:24 PM, sum abiut  wrote:
> >> Hi,
> >> Has anyone try this https://pypi.python.org/pypi/julian/0.14
> >>
> >> i got this error trying to import julian
> >>
> > import julian
> >> Traceback (most recent call last):
> >>   File "", line 1, in 
> >>   File "/usr/local/lib/python2.7/dist-packages/julian/__init__.py",
> line 1,
> >> in 
> >> from julian.julian import to_jd, from_jd
> >>   File "/usr/local/lib/python2.7/dist-packages/julian/julian.py", line
> 5
> >> def __to_format(jd: float, fmt: str) -> float:
> >>   ^
> >> SyntaxError: invalid syntax
> >>
> >
> > Looks like that package requires Python 3, but was uploaded to PyPI
> > without any version tags. You could try running it in Python 3.x, but
> > there's no way to know which ".x" versions are going to work.
>
> the cheeseshop description says 3.2+
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: julian 0.14 library

[Dale Marvin via Python-list]

>> On 2018-04-04 05:44, Chris Angelico wrote:
>>> On Wed, Apr 4, 2018 at 12:24 PM, sum abiut  wrote:
 Hi,
 Has anyone try this https://pypi.python.org/pypi/julian/0.14

 i got this error trying to import julian

>>> import julian
 Traceback (most recent call last):
File "", line 1, in 
File "/usr/local/lib/python2.7/dist-packages/julian/__init__.py",
>> line 1,
 in 
  from julian.julian import to_jd, from_jd
File "/usr/local/lib/python2.7/dist-packages/julian/julian.py", 
line

>> 5
  def __to_format(jd: float, fmt: str) -> float:
^
 SyntaxError: invalid syntax

>>>
>>> Looks like that package requires Python 3, but was uploaded to PyPI
>>> without any version tags. You could try running it in Python 3.x, but
>>> there's no way to know which ".x" versions are going to work.
>>
>> the cheeseshop description says 3.2+
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>>
On 4/4/18 3:35 PM, sum abiut wrote:

I got the error  below, tryinig in on python 3.2.

import julian
Traceback (most recent call last):
   File "", line 1, in 
ImportError: No module named julian


On Thu, Apr 5, 2018 at 3:21 AM, Thomas Jollans  wrote:



Did you pip install julian into your python 3.2 installation?
You may need type pip3 install julian depending upon how your python was 
installed.


Dale
--
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous processing is more efficient -- surely not?

[Terry Reedy]

On 4/4/2018 3:27 AM, Steven D'Aprano wrote:

So, I'm, still trying to wrap my brain around async processing, and I
started reading this tutorial:

http://stackabuse.com/python-async-await-tutorial/

and the very first paragraph broke my brain.

"Asynchronous programming has been gaining a lot of traction in the past
few years, and for good reason. Although it can be more difficult than
the traditional linear style, it is also much more efficient."


For multiple byte streams, async is more efficient in terms of bytes 
transmitted per unit time than transmitting each stream one at a time 
until completed.  Indeed, multiple infinite streams can be interleaved.


[snip]


But honestly, from everything I've seen, outside of browser apps (and I'm
not even sure about them), async processing looks to me like the wrong
solution to, well, everything.


The async module is event-driven programming applied to transmission 
between cpu and disk, other processes, or other systems.  The tkinter 
module does event-driven programming applied to transmission between cpu 
and humans (key and mouse input, screen output).  Both tkinter and async 
have time events.  One can even drive tkinter GUIs with the async loop 
instead of the tk loop.  (And this allows use of 'async' and 'await' 
with tkinter programs.)  Event-loop GUI applications and many games are 
'asynchronous processing' in the generic sense.  But these are a few 
decades old, not just a few years old.


 It combines the worst of sequential and

parallel processing with the benefits of neither. It's like cooperative
multi-tasking, only without the actual multi-tasking


I am not sure what you mean by 'multi-tasking' here.


and even less
convenience. Its harder to use than threading or multiprocessing, but


I can't imaging doing a gui with either, but maybe async is less 
convenient to use than async.



without their ability to run in parallel. It lacks the isolation of
multiprocessing, and the preemptive task swapping of threads/processes.

Its as hard to wrap your brain around as parallel processing in general,
but with even worse performance than sequential processing.


Do you have trouble with interactive games and GUIs?  The async module 
is the same idea applied (mainly) to socket events.  Indeed, key, mouse, 
and display events can be transmitted through sockets as byte streams. 
(I believe that this is how unix X-servers operate).



Am I totally wrong?


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Is pdb suitable for debugging asyncio module?

[jfong]

I have a module below and run it under pdb, the result seems not easy to xplain.
(Note: the sleep periods are in reverse order)
---
# asyncio_as_completed.py

import asyncio

@asyncio.coroutine
def phase(i):
print('in phase {}'.format(i))
yield from asyncio.sleep(0.5 - (0.1 * i))
print('done with phase {}'.format(i))
return 'phase {} result'.format(i)

@asyncio.coroutine
def main(num_phases):
print('starting main')
phases = [
phase(i)
for i in range(num_phases)
]
print('waiting for phases to complete')
results = []
for next_to_complete in asyncio.as_completed(phases):
answer = yield from next_to_complete  # <--set breakpoint here
print('received answer {!r}'.format(answer))
results.append(answer)
print('results: {!r}'.format(results))
return results

event_loop = asyncio.get_event_loop()
try:
event_loop.run_until_complete(main(3))
finally:
event_loop.close()
---

D:\Works\Python>py -m pdb asyncio_as_completed.py
> d:\works\python\asyncio_as_completed.py(3)()
-> import asyncio
(Pdb) n
> d:\works\python\asyncio_as_completed.py(5)()
-> @asyncio.coroutine
(Pdb) b 22
Breakpoint 1 at d:\works\python\asyncio_as_completed.py:22
(Pdb) c
starting main
waiting for phases to complete
> d:\works\python\asyncio_as_completed.py(22)main()
-> answer = yield from next_to_complete

(Pdb) n
in phase 1
in phase 0
in phase 2
Internal StopIteration# <---
> d:\works\python\asyncio_as_completed.py(8)phase()
-> yield from asyncio.sleep(0.5 - (0.1 * i))
(Pdb) n
> d:\works\python\asyncio_as_completed.py(9)phase()
-> print('done with phase {}'.format(i))
(Pdb) n
done with phase 2
> d:\works\python\asyncio_as_completed.py(10)phase()
-> return 'phase {} result'.format(i)

(Pdb) n
Internal StopIteration: phase 2 result
> d:\works\python\asyncio_as_completed.py(22)main()
-> answer = yield from next_to_complete
(Pdb) n
> d:\works\python\asyncio_as_completed.py(23)main()
-> print('received answer {!r}'.format(answer))
(Pdb) n
received answer 'phase 2 result'# <---
> d:\works\python\asyncio_as_completed.py(24)main()
-> results.append(answer)
(Pdb) n
> d:\works\python\asyncio_as_completed.py(21)main()
-> for next_to_complete in asyncio.as_completed(phases):
(Pdb) n
> d:\works\python\asyncio_as_completed.py(22)main()
-> answer = yield from next_to_complete

(Pdb) n
Internal StopIteration# <---
> d:\works\python\asyncio_as_completed.py(8)phase()
-> yield from asyncio.sleep(0.5 - (0.1 * i))
(Pdb) n
> d:\works\python\asyncio_as_completed.py(9)phase()
-> print('done with phase {}'.format(i))
(Pdb) n
done with phase 1
> d:\works\python\asyncio_as_completed.py(10)phase()
-> return 'phase {} result'.format(i)

(Pdb) n
Internal StopIteration# <---
> d:\works\python\asyncio_as_completed.py(8)phase()
-> yield from asyncio.sleep(0.5 - (0.1 * i))
(Pdb) n
> d:\works\python\asyncio_as_completed.py(9)phase()
-> print('done with phase {}'.format(i))
(Pdb) n
done with phase 0
> d:\works\python\asyncio_as_completed.py(10)phase()
-> return 'phase {} result'.format(i)

(Pdb) n
Internal StopIteration: phase 1 result# <---
> d:\works\python\asyncio_as_completed.py(22)main()
-> answer = yield from next_to_complete
(Pdb) n
> d:\works\python\asyncio_as_completed.py(23)main()
-> print('received answer {!r}'.format(answer))
(Pdb) n
received answer 'phase 1 result'
> d:\works\python\asyncio_as_completed.py(24)main()
-> results.append(answer)
(Pdb) n
> d:\works\python\asyncio_as_completed.py(21)main()
-> for next_to_complete in asyncio.as_completed(phases):
(Pdb) n
> d:\works\python\asyncio_as_completed.py(22)main()
-> answer = yield from next_to_complete

(Pdb) n
> d:\works\python\asyncio_as_completed.py(23)main()
-> print('received answer {!r}'.format(answer))
(Pdb) n
received answer 'phase 0 result'
> d:\works\python\asyncio_as_completed.py(24)main()
-> results.append(answer)
(Pdb) n
> d:\works\python\asyncio_as_completed.py(21)main()
-> for next_to_complete in asyncio.as_completed(phases):

(Pdb) n
> d:\works\python\asyncio_as_completed.py(25)main()
-> print('results: {!r}'.format(results))
(Pdb) n
results: ['phase 2 result', 'phase 1 result', 'phase 0 result']
> d:\works\python\asyncio_as_completed.py(26)main()
-> return results
(Pdb) n
The program finished and will be restarted
> d:\works\python\asyncio_as_completed.py(3)()
-> import asyncio
(Pdb)


I saw three "Internal StopIteration" lines which match the three "phase" tasks, 
it's OK. But there are only two "Internal StopIteration: phase x result" lines, 
the "phase 0" was missed. Why is that?

Best Regards,
Jach Fong
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous processing is more efficient -- surely not?

[Paul Rudin]

Steven D'Aprano  writes:

> So, I'm, still trying to wrap my brain around async processing, and I 
> started reading this tutorial:
>
> http://stackabuse.com/python-async-await-tutorial/
>
> and the very first paragraph broke my brain.
>
> "Asynchronous programming has been gaining a lot of traction in the past 
> few years, and for good reason. Although it can be more difficult than 
> the traditional linear style, it is also much more efficient."
>
> I can agree with the first part of the first sentence (gaining a lot of 
> traction), and the first part of the second sentence (more difficult than 
> the traditional style), but the second part? Asynchronous processing is 
> *more efficient*?



It really depends on your definition of "efficient". Using async
generally introduces some overhead, so there's a cost. However it also
allows for the possibility of making better use of your compute
resources by doing useful work rather than idle-waiting for network
interactions to complete. 

As with many things - it's a useful tool and can be used to your
advantage, but you can also shoot yourself in the foot if used
inappropriately.
-- 
https://mail.python.org/mailman/listinfo/python-list