Re: Asynchronous programming

[Paul Rubin]

Christian Gollwitzer  writes:
> I'm convinced that it is possible to integrate Tcl's event loop with
> asyncio's loop without regular update polling. This might require a
> patch to Tkinter at the C level. For example, an easy way is to put
> Tcl/Tk in it's own thread. ...

I did something like that a while back, putting tkinter in its own
thread but also using its .after() method to wake up the tkinter loop
every 50 msec or something like that.  I didn't notice any significant
cpu load or UI lag from that, and it was on a fairly slow machine (300
mhz ARM, I think).  The tkinter thread received callbacks from tkinter
and passed them on to the application's main dispatch loop through a
normal Python queue.  So there was no need to patch the tkinter C code.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python slang

[Marko Rauhamaa]

Lawrence D’Oliveiro :

> On Thursday, August 11, 2016 at 7:18:28 AM UTC+12, Marko Rauhamaa wrote:
>
>> Oh, there's of course COBOL:
>> 
>>ADD X TO Y GIVING RESULT
>
> I think COBOL has the same usage of “=” as BASIC, as assignment or
> equality comparison, depending on context.

There's the COMPUTE verb that works a bit like that. However, the

   ADD X TO Y [GIVING Z]
   MULTIPLY X BY Y [GIVING Z]
   DIVIDE X BY Y [GIVING Z]

is more traditional, I would guess.

Also, for simple assignments, there's MOVE:

   MOVE 'INDIA' TO WS-COUNTRY.

   http://www.tutorialspoint.com/cobol/cobol_basic_verbs.htm>

My uncle made a career out of COBOL programming. He described his work
as consisting mostly of MOVE statements.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Marko Rauhamaa]

Chris Angelico :

> Hmm. I'm not sure about that last bit. In order to properly support
> asynchronous I/O and the concurrency and reentrancy that that implies,
> you basically need all the same disciplines that you would for
> threaded programming

Correct. Python's asyncio closely mirrors threads in its API and thought
model. An async is like a thread except that you know the control will
only be relinquished at an "await" statement. Thus, you probably need
far less locking.

One visible difference between asyncs and threads is the fact that
threads don't need a special "await" construct. Instead, ordinary
function calls support context switching deep down in the Python VM and
the operating system. Each "await" explicitly marks a state in a state
machine. Whenever you turn a nonblocking function into a blocking one,
you will need to change the function call to an await statement.

Asyncs have a couple of distinct advantages over threads:

 * You can await multiple events in a single statement allowing for
   multiplexing. Threads can only wait for a single event.

 * Asyncs can be canceled; in general, threads can't.

> But maybe I'm too comfortable with threads. It's entirely possible; my
> first taste of reentrancy was interrupt handling in real mode 80x86
> assembly language, and in comparison to that, threads are pretty
> benign!

I steer clear of threads if I can because:

 * No multiplexing.

 * No cancellation.

 * Deadlocks and/or race conditions are virtually guaranteed.

 * Error symptoms are highly nonlocal making debugging very tricky.

Instead, I strongly prefer asynchronous programming and multiprocessing.

However, I'm afraid Python is following an unfortunate trend with
asyncio. Deep down concurrent programming involves finite state
machines. State machines can be expressed in many ways; traditionally,
you have SDL diagrams, state/event matrices etc. Asyncio marks states
with inline "await" keywords. Toy examples look really enticing and
readable compared with the more traditional implementation methods.
However, real-world code can quickly become spaghetti as the dozens of
cells of the state/event matrix are implemented.

My favorite asynchronous development model is the "callback hell," where
each cell of the state/event matrix is represented by a method (or at
least a switch case in C) and each state has an explicit name in the
source code. The resulting code is still hard to read and hard to get
right, but that complexity is unavoidable because reality just is that
complex.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Chris Angelico]

On Thu, Aug 11, 2016 at 5:55 PM, Marko Rauhamaa  wrote:
> Chris Angelico :
>
>> Hmm. I'm not sure about that last bit. In order to properly support
>> asynchronous I/O and the concurrency and reentrancy that that implies,
>> you basically need all the same disciplines that you would for
>> threaded programming
>
> Correct. Python's asyncio closely mirrors threads in its API and thought
> model. An async is like a thread except that you know the control will
> only be relinquished at an "await" statement. Thus, you probably need
> far less locking.

I've never needed ANY explicit locking in a threaded Python or Pike
program, and only rarely in C. But then, I'm one of those people who
finds threaded programming relatively easy.

> Asyncs have a couple of distinct advantages over threads:
>
>  * You can await multiple events in a single statement allowing for
>multiplexing. Threads can only wait for a single event.

Err, that's kinda the point of select() and friends. There are some
types of event that you can't select() on, but those also tend not to
work perfectly with async programming either - because, guess what,
event loops for async programs are usually built on top of select.

>  * Asyncs can be canceled; in general, threads can't.

Sadly, it's not that simple. Some async actions can be cancelled,
others cannot. At best, what you're saying is that *more* things can
be cancelled if they were started asynchronously.

>> But maybe I'm too comfortable with threads. It's entirely possible; my
>> first taste of reentrancy was interrupt handling in real mode 80x86
>> assembly language, and in comparison to that, threads are pretty
>> benign!
>
> I steer clear of threads if I can because:
>
>  * No multiplexing.
>
>  * No cancellation.

See above

>  * Deadlocks and/or race conditions are virtually guaranteed.

Definitely not. There are EASY ways to avoid deadlocks, and race
conditions are only a problem when you have some other bug. Yes,
threading bugs can be harder to debug; but only when you've violated
the other principles.

>  * Error symptoms are highly nonlocal making debugging very tricky.

But at least you have consistent and simple tracebacks. Callback model
code generally lacks that. (That's one of the advantages of Python's
generator-based async model - although it's not perfect.)

> Instead, I strongly prefer asynchronous programming and multiprocessing.

Multiprocessing is *exactly* as complicated as multithreading, with
the additional overhead of having to pass state around explicitly. And
if you're not getting the overhead of passing state around, you
wouldn't have the complexity of shared state in threading, ergo both
models would be identical.

> My favorite asynchronous development model is the "callback hell," where
> each cell of the state/event matrix is represented by a method (or at
> least a switch case in C) and each state has an explicit name in the
> source code. The resulting code is still hard to read and hard to get
> right, but that complexity is unavoidable because reality just is that
> complex.

Wow. Are you for real, or are you trolling us? You actually *enjoy*
callback hell?

Give me yield-based asyncio any day.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Steven D'Aprano]

On Thursday 11 August 2016 16:21, Christian Gollwitzer wrote:

> In typical GUI code, there are usually not that many places qhere ou
> have sequential code. A simple exmaple might be a counter. Using asyncio
> and a properly integrated GUI toolkit, you could write it as (pseudo-code)
> 
> async def counter():
> for i in range(10):
> button.settext("click me %d"%i)
> await button.click()
> button.disable()
> messageBox("You reached the end!")
> 
> Writing that same thing in callback code to button.on_click() is
> obviously less fun and feels inverted.

I don't know how you would write it in callback fashion, but the above seems 
inside out to me. It feels like COMEFROM instead of GOTO, if you understand the 
analogy. Rather than 

execute code in a linear fashion
until you reach a yield (GOTO), then transfer control *out*

it's 

execute code in a linear fashion
control transferred *in* from somewhere else (COMEFROM)


This is how I would expect to write something like that in an event-driven way:

def mouseUp(self):
# assume self.counter is initially set to 0
self.settext("Click Me %d" % self.counter)
self.counter += 1
if self.counter == 10:
self.disable()
messageBox("You have reached the end!")


which feels natural to me. Processing of each click is independent of any other 
click, and there's no non-linear transfer of control. The user clicks, the 
mouseUp method is called, it runs, and it is done. Whatever persistent state 
there is is part of the button itself.

I don't know whether you would call that a callback. I suppose it could be, in 
the sense that you might say:

button.set_mouseup_function(mouseUp)

but I'm used to thinking of it as a property of the button.



-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Chris Angelico]

On Thu, Aug 11, 2016 at 7:45 PM, Steven D'Aprano
 wrote:
> I don't know whether you would call that a callback. I suppose it could be, in
> the sense that you might say:
>
> button.set_mouseup_function(mouseUp)
>
> but I'm used to thinking of it as a property of the button.

"Callback" simply means "function that someone else calls". In fact,
most dunder methods could be described as callbacks.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Paul Rudin]

Chris Angelico  writes:

> On Thu, Aug 11, 2016 at 7:45 PM, Steven D'Aprano
>  wrote:
>> I don't know whether you would call that a callback. I suppose it could be, 
>> in
>> the sense that you might say:
>>
>> button.set_mouseup_function(mouseUp)
>>
>> but I'm used to thinking of it as a property of the button.
>
> "Callback" simply means "function that someone else calls". In fact,
> most dunder methods could be described as callbacks.
>


A callback is normally a callable you pass to some other function with
the expectation that it will be invoked in certain circumstances.


That's not really the same thing as something someone else calls: any
function that's part of a published api is intended to be called by
someone else; but that doesn't really make it a callback.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Marko Rauhamaa]

Chris Angelico :

> On Thu, Aug 11, 2016 at 5:55 PM, Marko Rauhamaa  wrote:
>> My favorite asynchronous development model is the "callback hell," where
>> each cell of the state/event matrix is represented by a method (or at
>> least a switch case in C) and each state has an explicit name in the
>> source code. The resulting code is still hard to read and hard to get
>> right, but that complexity is unavoidable because reality just is that
>> complex.
>
> Wow. Are you for real, or are you trolling us? You actually *enjoy*
> callback hell?
>
> Give me yield-based asyncio any day.

To me, the key is that the code should closely mirror the classic FSM
formalisms:

  http://images.slideplayer.com/14/4228155/slides/slide_21.jpg>

  http://www.detewe.ru/q931/image/10.gif>

  https://people.ece.cornell.edu/land/courses/ece4760/FinalPr
  ojects/s2011/pss83/Website/Time%20System/instruction_matrix.PNG>


It *would* be possible to write asyncio code the same way where you
would represent states as asyncs and express state transitions as:

await self.ESTABLISHING()

Alas, that would lead to unlimited async recursion, which, I'm pretty
sure, would overflow Python's execution stack.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Steven D'Aprano]

On Thu, 11 Aug 2016 03:34 pm, Paul Rudin wrote:

> Steven D'Aprano  writes:
> 
>>
>> Is there a good beginner's tutorial introducing the basics of
>> asynchronous programming? Starting with, why and where would you use it?
> 
> You could do worse than watch Dave Beazley's pycon talk:
> https://www.youtube.com/watch?v=lYe8W04ERnY

Instructions unclear, poked myself in the eye with a sharp stick.

I will, thanks for the reference, but in general I find that videos are good
for giving the illusion of understanding. It all seems so very clear at the
time, and then twenty minutes later it's all "Now did he say to rotate the
shifter or shift the rotator? How far into the talk was that bit again?"

Between programmers taking screenshots of code instead of copying and
pasting text, and the popularity of video over written examples, I'm
feeling like Master Li at the end of "The Story Of The Stone".

Bonus points to anyone who gets the reference without having to google it.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Steven D'Aprano]

On Thu, 11 Aug 2016 07:33 pm, Chris Angelico wrote:

> Yes,
> threading bugs can be harder to debug; but only when you've violated
> the other principles.


Especially the principle "Avoid threaded programming".

Some people, when faced with a problem, Now they think "I know, I'll have
use two threading." problems.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Is it ‘allowed’ to get parameters like this

[Cecil Westerhof]

It has been a while since I worked with Python. I wanted to get some
stats about the idle time of my computer, so that was a good moment to
pick up Python again. ;-)


As I understood it getopt is the way to get the parameters for your
script. But at the moment it looks a bit like overkill, my script does
not have much parameters as parameter_error shows:
def parameter_error():
print('Error: {0} [ --5 | --10 | --20 | --25 ] --all | --this-month | 
--this-year | --today'
  .format(basename(sys.argv[0])))
sys.exit(1)

So at the moment this is my init function:
def init():
global conn
global cursor
global vmstat_params

arg_length = len(sys.argv)
if   (arg_length == 1):
period   = '--today'
slice_length = 20
elif (arg_length == 2):
period   = sys.argv[1]
slice_length = 20
elif (arg_length == 3):
period = sys.argv[2]
if (sys.argv[1] in ['--5', '--10', '--20', '--25']):
slice_length = int(sys.argv[1][2:])
else:
parameter_error()
else:
parameter_error()
conn= connect(expanduser('~/Databases/general.sqlite'))
cursor  = conn.cursor()
all_data= '%'
today   = cursor.execute('SELECT CURRENT_DATE').fetchone()[0]
this_month  = today[0:8] + '%'
this_year   = today[0:5] + '%'
if   (period == '--today'):
vmstat_params = [today,  slice_length]
elif (period == '--this-month'):
vmstat_params = [this_month, slice_length]
elif (period == '--this-year'):
vmstat_params = [this_year,  slice_length]
elif (period == '--all'):
vmstat_params = [all_data,   slice_length]
else:
parameter_error()

Is this acceptable, or is it a big no-no?

By the way: the reason I fetch today from the database is that I work
with UTC dates.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Terry Reedy]

On 8/11/2016 2:34 AM, Christian Gollwitzer wrote:

Am 11.08.16 um 06:38 schrieb Terry Reedy:

You might be able to glean something from the succession of files I
uploaded to
https://bugs.python.org/issue27546
Integrate tkinter and asyncio (and async)

I started with just mixing tk and asyncio callbacks.  After some
struggle with similar question as you ask above, I ended up with more or
less callback free code using async def and async for. Once I got over
the hump, I rather like it.


My next experiment, not uploaded, is adapting tk mainloop so it can be 
used instead of asyncio.loop.runforever as the driver for async def and 
async for and await.  There is either a simple bug I overlooked or I am 
missing some detail of how the future and task interact.



Interesting discussion! Tcl itself acquired coroutines some years ago
(in 8.6) and people in this community are very excited with async based
programming.


When I read something like "Python finally acquired an event loop in 
3.4" I wonder where people have been.  The tk event loop has been in 
Python perhaps for 2 decades, and one actually could call select every 
xx milleseconds.  The main problem is that tcl/tk does not do file 
events on Windows.  The asyncio people did tackle that, but ended up 
with two loops, Selector and Proactor, neither of which are as complete 
as Selector on *nix.  Tcl/tk could try to borrow than work.



I'm convinced that it is possible to integrate Tcl's event loop with
asyncio's loop without regular update polling. This might require a
patch to Tkinter at the C level. For example, an easy way is to put
Tcl/Tk in it's own thread. The Tcl event loop can be woken up from a
different thread by the TclThreadQueueEvent() and TclTheradAlert() API
calls. Basically, you would spawn a new thread (from the C level), when
loading Tkinter, which loads Tk and sits in a Tcl event loop ("vwait
forever"). tk.eval() and tk.call() would be redefined to post events
using the aforementioned functions. When a Tcl callback arrives, it
would send an event to asyncio in the main thread - no idea how this
works, though. In the end, it would also allow to use Tkinter from
different Python threads, because all calls would be rooted to the one
instance only.


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Marko Rauhamaa]

Steven D'Aprano :

> Instructions unclear, poked myself in the eye with a sharp stick.

I have updated my Dining Philosophers example for Python 3.5:

   http://pacujo.net/~marko/philosophers.py>

It demonstrates how to get an event loop and start a number of asyncs
concurrently.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is it ‘allowed’ to get parameters like this

[Peter Otten]

Cecil Westerhof wrote:

> It has been a while since I worked with Python. I wanted to get some
> stats about the idle time of my computer, so that was a good moment to
> pick up Python again. ;-)
> 
> 
> As I understood it getopt is the way to get the parameters for your
> script. But at the moment it looks a bit like overkill, my script does
> not have much parameters as parameter_error shows:
> def parameter_error():
> print('Error: {0} [ --5 | --10 | --20 | --25 ] --all | --this-month |
> --this-year | --today'
>   .format(basename(sys.argv[0])))
> sys.exit(1)
> 
> So at the moment this is my init function:
> def init():
> global conn
> global cursor
> global vmstat_params
> 
> arg_length = len(sys.argv)
> if   (arg_length == 1):
> period   = '--today'
> slice_length = 20
> elif (arg_length == 2):
> period   = sys.argv[1]
> slice_length = 20
> elif (arg_length == 3):
> period = sys.argv[2]
> if (sys.argv[1] in ['--5', '--10', '--20', '--25']):
> slice_length = int(sys.argv[1][2:])
> else:
> parameter_error()
> else:
> parameter_error()



Contrast that with the following cli based on argparse:

$ cat demo.py
#!/usr/bin/env python3

def init():
import argparse

parser = argparse.ArgumentParser()
parser.add_argument(
"period", nargs="?",
choices="today this-month this-year all".split(),
default="today",
help="Period covered")
parser.add_argument(
"slice_length", type=int, nargs="?",
choices=[5, 10, 20, 25],
default=20,
help="Whatever...")

print(parser.parse_args())

init()
$ ./demo.py -h
usage: demo.py [-h] [{today,this-month,this-year,all}] [{5,10,20,25}]

positional arguments:
  {today,this-month,this-year,all}
Period covered
  {5,10,20,25}  Whatever...

optional arguments:
  -h, --helpshow this help message and exit
$ ./demo.py all
Namespace(period='all', slice_length=20)
$ ./demo.py none
usage: demo.py [-h] [{today,this-month,this-year,all}] [{5,10,20,25}]
demo.py: error: argument period: invalid choice: 'none' (choose from 
'today', 'this-month', 'this-year', 'all')
$ ./demo.py all 5
Namespace(period='all', slice_length=5)
$ ./demo.py all 7
usage: demo.py [-h] [{today,this-month,this-year,all}] [{5,10,20,25}]
demo.py: error: argument slice_length: invalid choice: 7 (choose from 5, 10, 
20, 25)

The interface is not exactly the same, but the code that defines it is 
declarative rather than trying to cover all possible combinations of input 
and should be easier to maintain in the long run.

Thus the answer to your question is no, the Python Police will have to 
arrest you if you stick with your original approach ;)

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Steven D'Aprano]

On Thu, 11 Aug 2016 03:06 pm, Paul Rubin wrote:

> The basic characteristic of asynchronous programming is that it involves
> changing all your usual blocking i/o calls to non-blocking ones, so your
> program can keep running as soon as your request is started.

That's the bit that confuses me. I understand about parallel processing, but
that's not what this is about.

Say I want to download data from a network, and it will take a long time. If
I can do the read in parallel to something else, that makes sense:

  begin downloading in another thread/process
  make a coffee
  process download


But what's the point in doing it asynchronously if I have to just wait for
it to complete?

  begin downloading in an async thread
  twiddle thumbs, doing nothing
  process download

Take Chris' "yield from" example, rewritten for await:


def asynchronous(id):
trn = await conn.begin_transaction()
await trn.execute("select name from people where id=%d", (id,))
name, = await trn.fetchone()
await trn.execute("update people set last_seen=now() where id=%d",(id,))
await trn.commit()
return name

Since I'm spending all my time waiting for the other things to return
control, why don't I just write it synchronously?

def synchronous(id):
trn = conn.begin_transaction()
trn.execute("select name from people where id=%d", (id,))
name, = trn.fetchone()
trn.execute("update people set last_seen=now() where id=%d", (id,))
trn.commit()
return name



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Marko Rauhamaa]

Steven D'Aprano :

> Say I want to download data from a network, and it will take a long
> time. If I can do the read in parallel to something else, that makes
> sense:
>
>   begin downloading in another thread/process
>   make a coffee
>   process download
>
> But what's the point in doing it asynchronously if I have to just wait for
> it to complete?

You can do many things concurrently, as the philosophers in my previous
example.

You will need two asyncs, one for downloading, another one for making
coffee, just like you would with threads. Then, run both asyncs
concurrently with asyncio.wait().


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Advice on optimizing a Python data driven rules engine

[Malcolm Greene]

Background: I'm building a  rules engine for transforming rows of data being 
returned by csv DictReader, eg. each row of data is a dict of column name to 
value mappings. My rules are a list of rule objects whose attributes get 
referenced by rule specific methods. Each rule has an associated method which 
gets called based on rule name.

Looking for some advice on how to optimize the BOILERPLATE portions of
the following type of code. There's an awful lot of dot dereferencing
going on. One thought was to pass in the values being dereferenced as
parameters and return value. But this would just move the dereferencing
to another point in my program and would add the overhead of parameter
passage. Is there a technique where I could store a reference to these
values that would make their access more efficient?

def action_strip(self):
# BOILERPLATE: lookup value being transformed (possible to get a
# direct 'pointer' to this dic entry???)
value = self.data[self.rule.target_column]

# BOILERPLATE: get rule's hard coded parameter
match = self.rule.value.strip()

# perform a generic transformation action
while value.startswith(match):
value = value.replace(match, '', 1).strip()
while value.endswith(match):
value = value[0:len(value) - len(match)].strip()

# BOILERPLATE: update the value we just transformed
self.data[self.rule.target_column] = value

Thank you,
Malcolm
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Steven D'Aprano]

On Thu, 11 Aug 2016 02:41 pm, Chris Angelico wrote:

> Consider these three ways of doing a database transaction:
> 
> def synchronous(id):
> trn = conn.begin_transaction()
> trn.execute("select name from people where id=%d", (id,))
> name, = trn.fetchone()
> trn.execute("update people set last_seen=now() where id=%d", (id,))
> trn.commit()
> return name

That makes perfect sense. Good old fashioned synchronous programming.


> def callbacks_1(cb, id):
> conn.begin_transaction(callbacks_2, cb, id)
> def callbacks_2(trn, cb, id):
> trn.execute("select name from people where id=%d", (id,),
> callbacks_3, cb, id)
> def callbacks_3(trn, cb, id):
> trn.fetchone(callbacks_4, cb, id)
> def callbacks_4(trn, data, cb, id):
> name, = data
> trn.execute("update people set last_seen=now() where id=%d",
> (id,), callbacks_5, cb, name)
> def callbacks_5(trn, cb, name):
> trn.commit(callbacks_6, cb, name)
> def callbacks_6(trn, cb, name):
> cb(name)

Now you're surely pulling my leg. Your conn.begin_transaction has a
completely different signature! (No arguments in the first case, three in
this case.)


> def asynchronous(id):
> trn = yield from conn.begin_transaction()
> yield from trn.execute("select name from people where id=%d", (id,))
> name, = yield from trn.fetchone()
> yield from trn.execute("update people set last_seen=now() where
>id=%d", (id,))
> yield from trn.commit()
> return name

That ... looks wrong. You're taking something which looks like a procedure
in the first case (trn.execute), so it probably returns None, and yielding
over it. Even it that's not wrong, and it actually returned something which
you ignored in the first case, it looks like you're mixing two distinct
ways of using generators:

- Generator as iterator ("yield x" or "yield from subiterator"); 
  something which *sends* values out for the purpose of iteration.

- Generator as coroutine ("y = yield x"); something which *receives*
  values from the called using the send() method.

Another way of putting it is, "a bogus generator that generates and receives
values (not a recommended coding style)". (David Beazly, per url below.)

When I say coroutine here, I mean in the *old* sense, as described by David
Beasley's A Curious Course on Coroutines and Concurrency:

http://www.dabeaz.com/coroutines/

So there's something deeply suspicious to me about the coding style.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is it ‘allowed’ to get parameters like this

[Steven D'Aprano]

On Thu, 11 Aug 2016 11:23 pm, Cecil Westerhof wrote:

> It has been a while since I worked with Python. I wanted to get some
> stats about the idle time of my computer, so that was a good moment to
> pick up Python again. ;-)
> 
> 
> As I understood it getopt is the way to get the parameters for your
> script. 

getopt is mainly for C programmers from Unix or Linux systems who are used
to using the Unix getopt function for handling parameters. There are two
alternatives in the standard library:

optparse (depreciated, for legacy use only)
argparse

https://docs.python.org/3/library/getopt.html
https://docs.python.org/3/library/optparse.html
https://docs.python.org/3/library/argparse.html

and many third-party options, including DocOpt:

https://pypi.python.org/pypi/docopt


[...]
> Is this acceptable, or is it a big no-no?

Despite what Peter says, the Python police won't come after you. Your
punishment for writing hard-to-maintain code is that you have to maintain
your own hard-to-maintain code.

If you have only a few command line arguments, and they don't change often,
the effort required to learn the "right" way to process them may be
significantly harder than the effort required to do it the "wrong" way.
That's called "technical debt", and in an on-going software project it can
hurt you very, very much. Technical debt can cause software projects to
fail.

But for personal use, where your software tends to be small and simple,
don't feel guilty about it. You'll know when it's time to learn one of the
other argument parsers: when you find yourself cursing the fact that you
have to edit your own argument-parsing code, that's the time to bite the
bullet and learn them.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is it ‘allowed’ to get parameters like this

[Cecil Westerhof]

On Thursday 11 Aug 2016 16:29 CEST, Peter Otten wrote:

> Cecil Westerhof wrote:
>
>> It has been a while since I worked with Python. I wanted to get
>> some stats about the idle time of my computer, so that was a good
>> moment to pick up Python again. ;-)
>>
>>
>> As I understood it getopt is the way to get the parameters for your
>> script. But at the moment it looks a bit like overkill, my script
>> does not have much parameters as parameter_error shows: def
>> parameter_error(): print('Error: {0} [ --5 | --10 | --20 | --25 ]
>> --all | --this-month | --this-year | --today'
>> .format(basename(sys.argv[0]))) sys.exit(1)
>>
>> So at the moment this is my init function:
>> def init():
>> global conn
>> global cursor
>> global vmstat_params
>>
>> arg_length = len(sys.argv)
>> if   (arg_length == 1):
>> period   = '--today'
>> slice_length = 20
>> elif (arg_length == 2):
>> period   = sys.argv[1]
>> slice_length = 20
>> elif (arg_length == 3):
>> period = sys.argv[2]
>> if (sys.argv[1] in ['--5', '--10', '--20', '--25']):
>> slice_length = int(sys.argv[1][2:])
>> else:
>> parameter_error()
>> else:
>> parameter_error()
>
> 
>
> Contrast that with the following cli based on argparse:
>
> $ cat demo.py
> #!/usr/bin/env python3
>
> def init():
> import argparse
>
> parser = argparse.ArgumentParser()
> parser.add_argument(
> "period", nargs="?",
> choices="today this-month this-year all".split(),
> default="today",
> help="Period covered")
> parser.add_argument(
> "slice_length", type=int, nargs="?",
> choices=[5, 10, 20, 25],
> default=20,
> help="Whatever...")
>
> print(parser.parse_args())
>
> init() $ ./demo.py -h usage: demo.py [-h]
> [{today,this-month,this-year,all}] [{5,10,20,25}]
>
> positional arguments:
> {today,this-month,this-year,all}
> Period covered
> {5,10,20,25}  Whatever...
>
> optional arguments: -h, --help show this help message and exit $
> ./demo.py all Namespace(period='all', slice_length=20) $ ./demo.py
> none usage: demo.py [-h] [{today,this-month,this-year,all}]
> [{5,10,20,25}] demo.py: error: argument period: invalid choice:
> 'none' (choose from 'today', 'this-month', 'this-year', 'all') $
> ./demo.py all 5 Namespace(period='all', slice_length=5) $ ./demo.py
> all 7 usage: demo.py [-h] [{today,this-month,this-year,all}]
> [{5,10,20,25}] demo.py: error: argument slice_length: invalid
> choice: 7 (choose from 5, 10, 20, 25)
>
> The interface is not exactly the same, but the code that defines it
> is declarative rather than trying to cover all possible combinations
> of input and should be easier to maintain in the long run.

That is very neat indeed. I am used of options starting with --, but I
am not married to that. I am going to look into this.


> Thus the answer to your question is no, the Python Police will have
> to arrest you if you stick with your original approach ;)

Luckily I am going to change the code. :-D

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Advice on optimizing a Python data driven rules engine

[Michael Selik]

On Thu, Aug 11, 2016 at 10:57 AM Malcolm Greene  wrote:

> Background: I'm building a  rules engine for transforming rows of data
> being returned by csv DictReader, eg. each row of data is a dict of column
> name to value mappings. My rules are a list of rule objects whose
> attributes get referenced by rule specific methods. Each rule has an
> associated method which gets called based on rule name.
>

Could you share the API you've designed for this tool? An example of usage
will clarify what you intend.

One technique for applying a transformation function to each record
returned by a csv DictReader:

import csv
import math
from io import StringIO

f = StringIO('''\
a,b,c
1,2,3
4,5,6
''')

def pass_(value):
return value

transforms = {
'a': lambda v: math.sin(int(v)),
'b': lambda v: math.cos(int(v)),
}

for row in csv.DictReader(f):
print({k: transforms.get(k, pass_)(v) for k, v in row.items()})



Using __missing__ for your transform dict might be more elegant than .get
with a pass_ as default.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Michael Selik]

On Thu, Aug 11, 2016 at 11:01 AM Steven D'Aprano 
wrote:

> That ... looks wrong. You're taking something which looks like a procedure
> in the first case (trn.execute), so it probably returns None, and yielding
> over it. Even it that's not wrong, and it actually returned something which
> you ignored in the first case
>

It's a standard, perhaps a mistaken standard, but nonetheless database
cursors tend to have that feature: execute returns the mutated self. I
agree that execute looks like it should return None instead. The return
self pattern feels Rubyish to me (or Rubic?).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Michael Selik]

On Thu, Aug 11, 2016 at 11:46 AM Michael Selik 
wrote:

> On Thu, Aug 11, 2016 at 11:01 AM Steven D'Aprano <
> steve+pyt...@pearwood.info> wrote:
>
>> That ... looks wrong. You're taking something which looks like a procedure
>> in the first case (trn.execute), so it probably returns None, and yielding
>> over it. Even it that's not wrong, and it actually returned something
>> which
>> you ignored in the first case
>>
>
> It's a standard, perhaps a mistaken standard, but nonetheless database
> cursors tend to have that feature: execute returns the mutated self. I
> agree that execute looks like it should return None instead. The return
> self pattern feels Rubyish to me (or Rubic?).
>

Contradicting myself:
yield from c.execute(query).fetchall()# looks good
yield from c.execute(query)   # looks bad
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Jussi Piitulainen]

Michael Selik  writes:

> On Thu, Aug 11, 2016 at 11:46 AM Michael Selik 
> wrote:
>
>> On Thu, Aug 11, 2016 at 11:01 AM Steven D'Aprano <
>> steve+pyt...@pearwood.info> wrote:
>>
>>> That ... looks wrong. You're taking something which looks like a procedure
>>> in the first case (trn.execute), so it probably returns None, and yielding
>>> over it. Even it that's not wrong, and it actually returned something
>>> which
>>> you ignored in the first case
>>>
>>
>> It's a standard, perhaps a mistaken standard, but nonetheless database
>> cursors tend to have that feature: execute returns the mutated self. I
>> agree that execute looks like it should return None instead. The return
>> self pattern feels Rubyish to me (or Rubic?).
>>
>
> Contradicting myself:
> yield from c.execute(query).fetchall()# looks good
> yield from c.execute(query)   # looks bad

The first looks like it allocates a list to hold all the results at once
only to yield each result one by one.

How about:
c.execute(query)
yield from c
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Advice on optimizing a Python data driven rules engine

[Chris Angelico]

On Fri, Aug 12, 2016 at 12:56 AM, Malcolm Greene  wrote:
> Looking for some advice on how to optimize the BOILERPLATE portions of
> the following type of code. There's an awful lot of dot dereferencing
> going on. One thought was to pass in the values being dereferenced as
> parameters and return value. But this would just move the dereferencing
> to another point in my program and would add the overhead of parameter
> passage. Is there a technique where I could store a reference to these
> values that would make their access more efficient?

Until you find that the dereferencing of dots is actually hurting you,
don't change anything. Most likely, it won't be a performance problem,
so the only question is: Does this hurt code readability? And
optimizing for code readability by adding levels of indirection is
often a bad idea.

What you MAY be able to do, though, if (as I'm guessing) the
BOILERPLATE sections are repeated across lots of functions, is to
refactor the functions to put those bits somewhere else. If nothing
else, you could make a function decorator that surrounds the code with
the appropriate boilerplate. But that's still indirection, and only
worthwhile if there's a lot of duplication. Remember, every time you
read this code, you'll need to go look elsewhere for the other half of
the guts.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is it ‘allowed’ to get parameters like this

[Cecil Westerhof]

On Thursday 11 Aug 2016 17:15 CEST, Cecil Westerhof wrote:

> On Thursday 11 Aug 2016 16:29 CEST, Peter Otten wrote:
>
>> Cecil Westerhof wrote:
>>
>>> It has been a while since I worked with Python. I wanted to get
>>> some stats about the idle time of my computer, so that was a good
>>> moment to pick up Python again. ;-)
>>>
>>>
>>> As I understood it getopt is the way to get the parameters for
>>> your script. But at the moment it looks a bit like overkill, my
>>> script does not have much parameters as parameter_error shows: def
>>> parameter_error(): print('Error: {0} [ --5 | --10 | --20 | --25 ]
>>> --all | --this-month | --this-year | --today'
>>> .format(basename(sys.argv[0]))) sys.exit(1)
>>>
>>> So at the moment this is my init function:
>>> def init():
>>> global conn
>>> global cursor
>>> global vmstat_params
>>>
>>> arg_length = len(sys.argv)
>>> if   (arg_length == 1):
>>> period   = '--today'
>>> slice_length = 20
>>> elif (arg_length == 2):
>>> period   = sys.argv[1]
>>> slice_length = 20
>>> elif (arg_length == 3):
>>> period = sys.argv[2]
>>> if (sys.argv[1] in ['--5', '--10', '--20', '--25']):
>>> slice_length = int(sys.argv[1][2:])
>>> else:
>>> parameter_error()
>>> else:
>>> parameter_error()
>>
>> 
>>
>> Contrast that with the following cli based on argparse:
>>
>> $ cat demo.py
>> #!/usr/bin/env python3
>>
>> def init():
>> import argparse
>>
>> parser = argparse.ArgumentParser()
>> parser.add_argument(
>> "period", nargs="?",
>> choices="today this-month this-year all".split(),
>> default="today",
>> help="Period covered")
>> parser.add_argument(
>> "slice_length", type=int, nargs="?",
>> choices=[5, 10, 20, 25],
>> default=20,
>> help="Whatever...")
>>
>> print(parser.parse_args())
>>
>> init() $ ./demo.py -h usage: demo.py [-h]
>> [{today,this-month,this-year,all}] [{5,10,20,25}]
>>
>> positional arguments:
>> {today,this-month,this-year,all}
>> Period covered
>> {5,10,20,25}  Whatever...
>>
>> optional arguments: -h, --help show this help message and exit $
>> ./demo.py all Namespace(period='all', slice_length=20) $ ./demo.py
>> none usage: demo.py [-h] [{today,this-month,this-year,all}]
>> [{5,10,20,25}] demo.py: error: argument period: invalid choice:
>> 'none' (choose from 'today', 'this-month', 'this-year', 'all') $
>> ./demo.py all 5 Namespace(period='all', slice_length=5) $ ./demo.py
>> all 7 usage: demo.py [-h] [{today,this-month,this-year,all}]
>> [{5,10,20,25}] demo.py: error: argument slice_length: invalid
>> choice: 7 (choose from 5, 10, 20, 25)
>>
>> The interface is not exactly the same, but the code that defines it
>> is declarative rather than trying to cover all possible
>> combinations of input and should be easier to maintain in the long
>> run.
>
> That is very neat indeed. I am used of options starting with --, but
> I am not married to that. I am going to look into this.

I rewrote it to:
def init():
global conn
global cursor
global vmstat_params

parser = ArgumentParser()
parser.add_argument(
'period', nargs = '?',
choices = 'all this-month this-year today'.split(),
default = 'today',
help='Period covered')
parser.add_argument(
'slice_length', type = int, nargs ='?',
choices = [5, 10, 20, 25],
default = 20,
help = 'Length of a percentage slice')
args= parser.parse_args()
period  = args.period
slice_length= args.slice_length
conn= connect(expanduser('~/Databases/general.sqlite'))
cursor  = conn.cursor()
all_data= '%'
today   = cursor.execute('SELECT CURRENT_DATE').fetchone()[0]
this_month  = today[0:8] + '%'
this_year   = today[0:5] + '%'
if   (period == 'today'):
vmstat_params = [today,  slice_length]
elif (period == 'this-month'):
vmstat_params = [this_month, slice_length]
elif (period == 'this-year'):
vmstat_params = [this_year,  slice_length]
elif (period == 'all'):
vmstat_params = [all_data,   slice_length]
else:
print('Unknown period: {0}'.format(period))
sys.exit(1)

I kept the else part for the case that an argument is added that is
not put into the if/elif construction.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Paul Rudin]

Steven D'Aprano  writes:

>
> But what's the point in doing it asynchronously if I have to just wait for
> it to complete?
>
>   begin downloading in an async thread
>   twiddle thumbs, doing nothing
>   process download

If you have nothing else to do, then there's no point.

But suppose you're implementing a web server. A request comes in - in
order to respond you need some data from a database. So you do the
database bit async. Whilst you're waiting for that you can get on with
processing other incoming requests.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Steven D'Aprano]

On Thu, 11 Aug 2016 03:34 pm, Paul Rudin wrote:

> Steven D'Aprano  writes:
> 
>>
>> Is there a good beginner's tutorial introducing the basics of
>> asynchronous programming? Starting with, why and where would you use it?
> 
> You could do worse than watch Dave Beazley's pycon talk:
> https://www.youtube.com/watch?v=lYe8W04ERnY


Thanks. I'm not sure that it helped. I kind of gibbered a bit when Dave said
that threaded code was good because you can easily reason about it. Most of
his talk is about how asyncio shouldn't be in the standard library, and the
rest didn't really answer my questions.

What I'm taking from this is that asynchronous programming is mindboggling
difficult to reason about or understand, and its the best thing ever.

*half-a-wink*


Having-flashbacks-to-when-I-was-first-introduced-to-OOP-ly y'rs,



-- 
Steve
If anyone mentions monads, I'm out of here.

-- 
https://mail.python.org/mailman/listinfo/python-list

Print function not working

[Atri Mahapatra]

I have installed IDLE 3.5.1 and wrote the following  to check if print is 
working. When it runs, I do not see anything is printed:

class Base: #{
def __init__( self ): #{
print("Hello, world: \n\n");

#}

#}


if ( __name__ == " __main__"): #{
root = Base();
#}

Can anyone please point out the reason?

Thanks,
Atri
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Chris Angelico]

On Fri, Aug 12, 2016 at 12:55 AM, Steven D'Aprano
 wrote:
> On Thu, 11 Aug 2016 02:41 pm, Chris Angelico wrote:
>
>> Consider these three ways of doing a database transaction:
>>
>> def synchronous(id):
>> trn = conn.begin_transaction()
>> trn.execute("select name from people where id=%d", (id,))
>> name, = trn.fetchone()
>> trn.execute("update people set last_seen=now() where id=%d", (id,))
>> trn.commit()
>> return name
>
> That makes perfect sense. Good old fashioned synchronous programming.

Let's assume in this case that we started with:

conn = synchronous_database_connection()

>> def callbacks_1(cb, id):
>> conn.begin_transaction(callbacks_2, cb, id)
>> def callbacks_2(trn, cb, id):
>> trn.execute("select name from people where id=%d", (id,),
>> callbacks_3, cb, id)
>> def callbacks_3(trn, cb, id):
>> trn.fetchone(callbacks_4, cb, id)
>> def callbacks_4(trn, data, cb, id):
>> name, = data
>> trn.execute("update people set last_seen=now() where id=%d",
>> (id,), callbacks_5, cb, name)
>> def callbacks_5(trn, cb, name):
>> trn.commit(callbacks_6, cb, name)
>> def callbacks_6(trn, cb, name):
>> cb(name)
>
> Now you're surely pulling my leg. Your conn.begin_transaction has a
> completely different signature! (No arguments in the first case, three in
> this case.)

Let's assume that this one started with:

conn = callback_database_connection()

It's doing the same job as the 'conn' in the first example, but it's a
completely different API to cater to the fact that it has to handle
callbacks. You could use this API for synchronous calls by doing
something like this:

def begin_transaction(callback, *args):
real_conn.begin_transaction()
callback(*args)

The asynchronous version would end up saving callback and args
somewhere, triggering the operation, and having code somewhere that
processes the response. Supposing we're talking to PostgreSQL over a
socket (TCP or Unix domain), the response handler would be triggered
any time that socket becomes readable (ie via select() on the socket),
and it would decode the response, figure out which transaction is
being responded to (if there are multiple in flight), and send the
response on its way. Most likely the transaction would have some kind
of "current in-flight query" attribute (and would reject reentrant
calls - see, any form of async programming has to cope with
reentrancy), so that's where the callback would be stored.

>> def asynchronous(id):
>> trn = yield from conn.begin_transaction()
>> yield from trn.execute("select name from people where id=%d", (id,))
>> name, = yield from trn.fetchone()
>> yield from trn.execute("update people set last_seen=now() where
>>id=%d", (id,))
>> yield from trn.commit()
>> return name
>
> That ... looks wrong. You're taking something which looks like a procedure
> in the first case (trn.execute), so it probably returns None, and yielding
> over it. Even it that's not wrong, and it actually returned something which
> you ignored in the first case, it looks like you're mixing two distinct
> ways of using generators:
>
> - Generator as iterator ("yield x" or "yield from subiterator");
>   something which *sends* values out for the purpose of iteration.
>
> - Generator as coroutine ("y = yield x"); something which *receives*
>   values from the called using the send() method.

Yeah, generators as coroutines are a bit weird. That's another good
reason for using the new async and await "keywords" (not technically
keywords yet), as it doesn't look as weird. But ultimately, it's doing
the same thing - the methods would look something like this:

def begin_transaction():
# trigger the "begin transaction" query
yield Awaitable("waiting for transaction...")
# verify that the query was successful

The event loop attempts to step the "asynchronous" generator. It
yields from begin_transaction, which yields an Awaitable. The event
loop thus receives, from the generator, an object to be placed on the
queue. It's that simple.

Here's a very VERY simple, but complete, example of yield-based coroutines.

# Partially borrowed from example in Python docs:
# https://docs.python.org/3/library/selectors.html#examples
import selectors
import socket
import time

sel = selectors.DefaultSelector()
def eventloop():
while "loop forever":
for key, mask in sel.select():
sel.unregister(key.fileobj)
run_task(key.data)

def run_task(gen):
try:
waitfor = next(gen)
sel.register(waitfor, selectors.EVENT_READ, gen)
except StopIteration:
pass

def mainsock():
sock = socket.socket()
sock.bind(('localhost', 1234))
sock.listen(100)
sock.setblocking(False)
print("Listening on port 1234.")
while "moar sockets":
yield sock
conn, addr = sock.accept()  # Should be ready
print('accepted', conn, 'from', 

Print function not working

[Atri Mahapatra]

I have installed IDLE 3.5.1 and wrote the following print command to see if it 
is working. When I ran the code nothing is printed. Can you please point the 
reason- anything wrong with the code?



class Base: #{
def __init__( self ): #{
print("Hello, world: \n\n");

#}

#}


if ( __name__ == " __main__"): #{
root = Base();
#}

Thanks,
Atri
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Print function not working

[Chris Angelico]

On Fri, Aug 12, 2016 at 3:03 AM, Atri Mahapatra
 wrote:
> I have installed IDLE 3.5.1 and wrote the following  to check if print is 
> working. When it runs, I do not see anything is printed:
>
> class Base: #{
> def __init__( self ): #{
> print("Hello, world: \n\n");
>
> #}
>
> #}
>
>
> if ( __name__ == " __main__"): #{
> root = Base();
> #}
>
> Can anyone please point out the reason?
>
> Thanks,
> Atri

Is name equal to main? That is, are you running this code as a top-level script?

Also: You do not need to write Python code as if it were C or Java.
All that extra punctuation is just putting unnecessary stress on the
world's punctuation mines, which are already overworked. :)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Print function not working

[Peter Otten]

Atri Mahapatra wrote:

> I have installed IDLE 3.5.1 and wrote the following  to check if print is
> working. When it runs, I do not see anything is printed:
> 
> class Base: #{
> def __init__( self ): #{
> print("Hello, world: \n\n");
> 
> #}
> 
> #}
> 
> 
> if ( __name__ == " __main__"): #{
> root = Base();
> #}
> 
> Can anyone please point out the reason?

Hard to tell. Either there slipped an extra space into the string literal in

> if ( __name__ == " __main__"): #{

or the interpreter got really annoyed about you trying to write C/Java/PHP 
rather than proper Python...

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Print function not working

[MRAB]

On 2016-08-11 18:18, Chris Angelico wrote:

On Fri, Aug 12, 2016 at 3:03 AM, Atri Mahapatra
 wrote:

I have installed IDLE 3.5.1 and wrote the following  to check if print is 
working. When it runs, I do not see anything is printed:

class Base: #{
def __init__( self ): #{
print("Hello, world: \n\n");

#}

#}


if ( __name__ == " __main__"): #{
root = Base();
#}

Can anyone please point out the reason?

Thanks,
Atri


Is name equal to main? That is, are you running this code as a top-level script?

It's not checking whether the name is "__main__", but whether it's " 
__main__" (note the space after the first quote).



Also: You do not need to write Python code as if it were C or Java.
All that extra punctuation is just putting unnecessary stress on the
world's punctuation mines, which are already overworked. :)


+1

--
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Paul Rubin]

Steven D'Aprano  writes:
> But what's the point in doing it asynchronously if I have to just wait for
> it to complete?
>   begin downloading in an async thread
>   twiddle thumbs, doing nothing
>   process download

Suppose the remote server is overloaded so it sends files much slower
than your internet connection can receive them.  And you want to
download 10 different files from 10 different such servers.  If you do
them synchronously (wait for each download to finish before starting the
next) it takes much longer than necessary.  What you want is to download
the 10 files simultaneously, using 10 different network connections.
The download procedure has to speak some network protocol over a socket
for each file.

How do you deal with the concurrency?  There are many possibilities:

   - use 10 different computers
   - open 10 windows on 1 computer, and start a download in each window
 (i.e. this means use multiple processes).
   - single, multi-threaded download client (oh nooo!  the thread
 monsters will eat you if you try to do that)
   - single threaded client with asynchronous i/o, so it can have
 10 network requests "in the air" simultaneously, using select() or
 epoll() to handle each piece of incoming data as soon as it arrives.

The multi-thread and multi-process approaches are conceptually simple
since each connection appears to be synchronous and blocking.  They are
both actually async under the covers, but the async i/o and dispatch is
abstracted away by the OS, so the user program doesn't have to worry
about it.

The in-client async approach is generally the most efficient (OS
processes and threads are expensive), but imposes complexity on the
client protocol by making it juggle what each connection is doing,
i.e. where it is in the network protocol at any moment.

A lot of ways have developed over the years to organize client-side
async programs and keep them from getting too confusing:

  - explicit state machines (a struct with a state tag for each
connection, and a big event loop with a switch statement),
frequently seen in C programs
  - Chained callbacks ("callback hell") seen in node.js
  - Callbacks on objects ("reactor pattern"), used in Twisted Matrix
  - explicit cooperative multitasking (used in RTOS's, classic Forth,
etc.)
  - lightweight processes or threads handled by the language runtime
(GHC, Erlang).  This means the user program thinks it's doing
blocking i/o but it's really not.
  - coroutines (Lua and now Python's asyncio)
  - continuation-based hackery (various Haskell enumeratee libraries)
  - probably more that I don't know about or am forgetting.

I like the Erlang/GHC approach best, but it basically means building a
miniature OS into the language runtime and making sure all the
user-visible i/o calls actually use this "OS" instead of actual system
calls to the underlying kernel.  The Erlang and GHC implementations are
quite complicated while Python is basically a fairly simple interpreter
wrapped around the standard C libraries.

In Python, the async discussion is basically between various forms of
callbacks, and a few different forms of coroutines.  I think that the
conception of best practices is still not completely settled.

I ignore all this and use threads and take the performance hit.  I find
it simpler and I haven't been eaten by any thread monsters yet (though
there's always a first time).  I figure if I need high performance,
Python isn't the way to do it in the first place: Python is more about
convenience and productivity than performance.  I've had 1000 or so
Python threads on a midsized EC2 instance and it's worked ok.

If you really want to do crazy fast async i/o and you use C++, check out
  http://www.seastar-project.org/

I haven't tried it yet but want to.

Here's a cool paper about the current GHC I/O system:
http://haskell.cs.yale.edu/wp-content/uploads/2013/08/hask035-voellmy.pdf
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Print function not working

[Michael Selik]

On Thu, Aug 11, 2016 at 1:38 PM MRAB  wrote:

> On 2016-08-11 18:18, Chris Angelico wrote:
> > On Fri, Aug 12, 2016 at 3:03 AM, Atri Mahapatra
> >  wrote:
> >> I have installed IDLE 3.5.1 and wrote the following  to check if print
> is working. When it runs, I do not see anything is printed:
> >>
> >> class Base: #{
> >> def __init__( self ): #{
> >> print("Hello, world: \n\n");
> >>
> >> #}
> >>
> >> #}
> >>
> >>
> >> if ( __name__ == " __main__"): #{
> >> root = Base();
> >> #}
> >>
> >> Can anyone please point out the reason?
> >>
> >> Thanks,
> >> Atri
> >
> > Is name equal to main? That is, are you running this code as a top-level
> script?
> >
> It's not checking whether the name is "__main__", but whether it's "
> __main__" (note the space after the first quote).
>
> > Also: You do not need to write Python code as if it were C or Java.
> > All that extra punctuation is just putting unnecessary stress on the
> > world's punctuation mines, which are already overworked. :)
> >
> +1
>

When debugging something like this, one technique is to remove bits of code
to reduce the complexity. Remove something, run the code, check the
results, then repeat if it's still failing.

If you're not sure how class initializers work, maybe convert to a regular
function and try again.

def foo():
print('hello')
if __name__ == '__main__':
foo()

If that still doesn't work, maybe get rid of the function.

if __name__ == '__main__':
print('hello')

And if that still doesn't work, maybe get rid of the if-statement.

print('hello')

I think you'll find that the single line program "print('hello')" works
just fine.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What's the best way to minimize the need of run time checks?

[Russell Owen]

On 8/10/16 3:44 PM, Juan Pablo Romero Méndez wrote:

As to why I asked that, there are several reasons: I have a very concrete
need right now to find pragmatic ways to increase code quality, reduce
number of defects, etc. in a Python code base. But also I want to
understand better the mind set and culture of Python's community.


I am late to this thread, so my apologies for duplicated answers, but I 
have two concrete suggestions:
- Unit tests. These are a hassle to write, but pay huge dividends in 
robustness of your existing code and making it safer to modify the code 
later. There are also tools to measure test coverage which are worth 
considering. I don't think it is possible to write robust code in any 
language (even compiled languages) without a good test suite.
- Always run a linter such as flake8. Most source code editors can be 
configured to do this automatically. This will not catch everything that 
a compiler would catch in a compiled language, but it will catch many 
common errors.


-- Russell

--
https://mail.python.org/mailman/listinfo/python-list

Re: ctypes And The WACAH Principle

[Roel Schroeven]

Lawrence D’Oliveiro schreef op 2016-08-10 03:45:

... so WACAH very much applies here.


Pardon my ignorance, but what is the WACAH Principle?


--
The saddest aspect of life right now is that science gathers knowledge
faster than society gathers wisdom.
  -- Isaac Asimov

Roel Schroeven

--
https://mail.python.org/mailman/listinfo/python-list

Re: ctypes And The WACAH Principle

[MRAB]

On 2016-08-11 20:41, Roel Schroeven wrote:

Lawrence D’Oliveiro schreef op 2016-08-10 03:45:

... so WACAH very much applies here.


Pardon my ignorance, but what is the WACAH Principle?

I believe he means "€œWe're All Consenting Adults Here". (I don't think 
I've ever seen that acronym before.)

--
https://mail.python.org/mailman/listinfo/python-list

A strange list concatenation result

[Mok-Kong Shen]

def test(list1,list2):
  list1+=[4,5,6]
  list2=list2+[4,5,6]
  print("inside ",list1,list2)
  return

# With

list1=list2=[1,2,3]
test(list1,list2)
print("outside",list1,list2)

# I got the following:
# inside  [1, 2, 3, 4, 5, 6] [1, 2, 3, 4, 5, 6, 4, 5, 6]
# outside [1, 2, 3, 4, 5, 6] [1, 2, 3, 4, 5, 6]

# With

list1=[1,2,3]
list2=[1,2,3]
test(list1,list2)
print("outside",list1,list2)

# I got the following:
# inside  [1, 2, 3, 4, 5, 6] [1, 2, 3, 4, 5, 6]
# outside [1, 2, 3, 4, 5, 6] [1, 2, 3]
--
https://mail.python.org/mailman/listinfo/python-list

Re: A strange list concatenation result

[Gary Herron]

On 08/11/2016 03:06 PM, Mok-Kong Shen wrote:


def test(list1,list2):
  list1+=[4,5,6]
  list2=list2+[4,5,6]
  print("inside ",list1,list2)
  return

# With

list1=list2=[1,2,3]
test(list1,list2)
print("outside",list1,list2)

# I got the following:
# inside  [1, 2, 3, 4, 5, 6] [1, 2, 3, 4, 5, 6, 4, 5, 6]
# outside [1, 2, 3, 4, 5, 6] [1, 2, 3, 4, 5, 6]

# With

list1=[1,2,3]
list2=[1,2,3]
test(list1,list2)
print("outside",list1,list2)

# I got the following:
# inside  [1, 2, 3, 4, 5, 6] [1, 2, 3, 4, 5, 6]
# outside [1, 2, 3, 4, 5, 6] [1, 2, 3]



I think the following shows the same issue in a much simpler fashion:

In this (and your first) example, there is only one list, although it 
has two names to reference it.


>>> list1 = list2 = [1,2,3]
>>> list1 += [4,5,6]
>>> print(list1, list2)
[1, 2, 3, 4, 5, 6] [1, 2, 3, 4, 5, 6]


In this next example, there are two separate lists:

>>> list1 = [1,2,3]
>>> list2 = [1,2,3]
>>> list1 += [4,5,6]
>>> print(list1, list2)
[1, 2, 3, 4, 5, 6] [1, 2, 3]


Does that help?


Gary Herron


--
Dr. Gary Herron
Professor of Computer Science
DigiPen Institute of Technology
(425) 895-4418

--
https://mail.python.org/mailman/listinfo/python-list

Re: Generate reports in Python

[Lawrence D’Oliveiro]

On Thursday, August 11, 2016 at 8:30:42 PM UTC+12, wxjm...@gmail.com wrote:
> I do not see to much how a library can beat that approach, unless it
> duplicates what TeX does.

TEX may be fine for more free-form text, but often reports take a more 
structured layout than that--fixed columns etc.

For example, I generate my client invoices using odfpy 
, so that I can do any necessary tweaking using 
LibreOffice Writer before outputting a PDF file.
-- 
https://mail.python.org/mailman/listinfo/python-list

OT Winmx works again

[Seymore4Head]

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A strange list concatenation result

[Steven D'Aprano]

Note that your subject line is wrong. You are not doing list concatenation.

Unfortunately, for technical and optimization reasons, the += assignment
operator for lists is in-place, which means that it is NOT the same as
ordinary list concatenation + operator. It is equivalent to calling the
list.extend method.

If you want to do concatenation, creating NEW lists on every operation, you
have to write it the expanded way:

list1 = list1 + list2
# creates a new list from the concatenation of list1 and list2


Your code starts like this:

list1 = list2 = [1, 2, 3]

That is NOT two different lists with equal contents, but ONE list with two
names. You can see this with this:


py> list1 = list2 = [1, 2, 3]
py> list1.append(4)
py> list2
[1, 2, 3, 4]
py> list1 is list2  # Are they the same object?
True


So what happens when you do an in-place extend of a list to itself? You
should be able to guess from the example of append:

py> list1.extend(list2)
py> list2
[1, 2, 3, 4, 1, 2, 3, 4]

list1 will be the same, because list1 and list2 are the same list. Using +=
will give the same results.

So you have two gotchas here:

a = b = [1, 2, 3]

gives you the same list with two names. You could make a copy:

a = [1, 2, 3]
b = a[:]  # using slicing

or:

import copy
a = [1, 2, 3]
b = copy.copy(a)

The second gotcha is that += is not always equivalent to + and in particular
for lists it is equivalent to calling the extend() method.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Steven D'Aprano]

Thanks to everyone who has answered, I think I'm slowly starting to get it
now. Let's see if we can come up with a toy example that doesn't involve
low-level socket programming :-)

Let me simulate a slow function call:


import random, time

def work(id):
print("starting with id", id)
workload = random.randint(5, 15)
for i in range(workload):
time.sleep(0.2)  # pretend to do some real work
print("processing id", id)  # let the user see some progress
print("done with id", id)
return 10 + id


pending = [1, 2, 3, 4]

for i, n in enumerate(pending):
pending[i] = work(n)


How do I write work() so that it cooperatively multi-tasks with other ...
threads? processes? what the hell do we call these things? What does this
example become in the asynchronous world?

In this case, all the work is pure computation, so I don't expect to save
any time by doing this, because the work is still all being done in the
same process/thread, not in parallel. It may even take a little bit longer,
due to the overhead of switching from one to another.

(I presume that actual disk or network I/O may be better, because the OS
will run the I/O in parallel if possible, but I don't expect that in this
case.)

Am I getting close?



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Chris Angelico]

On Fri, Aug 12, 2016 at 10:08 AM, Steven D'Aprano
 wrote:
> Let me simulate a slow function call:
>
>
> import random, time
>
> def work(id):
> print("starting with id", id)
> workload = random.randint(5, 15)
> for i in range(workload):
> time.sleep(0.2)  # pretend to do some real work
> print("processing id", id)  # let the user see some progress
> print("done with id", id)
> return 10 + id
>
>
> pending = [1, 2, 3, 4]
>
> for i, n in enumerate(pending):
> pending[i] = work(n)
>
>
> How do I write work() so that it cooperatively multi-tasks with other ...
> threads? processes? what the hell do we call these things? What does this
> example become in the asynchronous world?
>
> In this case, all the work is pure computation, so I don't expect to save
> any time by doing this, because the work is still all being done in the
> same process/thread, not in parallel. It may even take a little bit longer,
> due to the overhead of switching from one to another.
>
> (I presume that actual disk or network I/O may be better, because the OS
> will run the I/O in parallel if possible, but I don't expect that in this
> case.)
>
> Am I getting close?

Well, let's see. I can quickly tweak my select() demo to support time delays.

# Partially borrowed from example in Python docs:
# https://docs.python.org/3/library/selectors.html#examples
import selectors
import socket
import time

sel = selectors.DefaultSelector()
sleepers = {} # In a non-toy, this would probably be a heap, not a dict
def eventloop():
while "loop forever":
t = time.time()
for gen, tm in list(sleepers.items()):
if tm <= t:
del sleepers[gen]
run_task(gen)
delay = min(sleepers.values(), default=t+3600) - t
if delay < 0: continue
for key, mask in sel.select(timeout=delay):
sel.unregister(key.fileobj)
run_task(key.data)

def run_task(gen):
try:
waitfor = next(gen)
if isinstance(waitfor, float):
sleepers[gen] = waitfor
else:
sel.register(waitfor, selectors.EVENT_READ, gen)
except StopIteration:
pass

def sleep(tm):
yield time.time() + tm

def mainsock():
sock = socket.socket()
sock.bind(('localhost', 1234))
sock.listen(100)
sock.setblocking(False)
print("Listening on port 1234.")
while "moar sockets":
yield sock
conn, addr = sock.accept()  # Should be ready
print('accepted', conn, 'from', addr)
conn.setblocking(False)
run_task(client(conn))

def client(conn):
while "moar data":
yield conn
data = conn.recv(1000)  # Should be ready
if not data: break
print("Got data")
# At this point, you'd do something smart with the data.
# But we don't. We just echo back, after a delay.
yield from sleep(3)
conn.send(data)  # Hope it won't block
if b"quit" in data: break
print('closing', conn)
conn.close()

if __name__ == '__main__':
run_task(mainsock())
eventloop()



So, if it used this tiny event loop, your code would look like this:

def work(id):
print("starting with id", id)
workload = random.randint(5, 15)
for i in range(workload):
yield from sleep(0.2)  # pretend to do some real work
print("processing id", id)  # let the user see some progress
print("done with id", id)
return 10 + id

for n in [1, 2, 3, 4]:
run_task(work(n))
eventloop()

But crucially, this depends on having some kind of waitable work. That
generally means either non-blocking I/O (of some sort - a lot of
things in a Unix system end up being reads and writes to some
file-like thing, eg a socket, pipe, or device), or a time delay, or
maybe waiting on a signal. If the 'work' is a blocking CPU-bound
operation, this will never be able to multiplex.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is it ‘allowed’ to get parameters like this

[Lawrence D’Oliveiro]

On Friday, August 12, 2016 at 1:28:04 AM UTC+12, Cecil Westerhof wrote:

> As I understood it getopt is the way to get the parameters for your
> script. But at the moment it looks a bit like overkill, my script does
> not have much parameters as parameter_error shows:

That’s fine. But your command-line needs are so simple, I wouldn’t bother with 
the “--” business either. Just take the interval as a single argument. Here’s 
how I would do it:

if len(sys.argv) == 2 :
periodspec = sys.argv[1]
elif len(sys.argv) == 1 :
periodspec = "all"
else :
raise RuntimeError("usage:\n{} [period]" % sys.argv[0])
#end if

today = time.strftime("%Y-%m-%d") # or whatever
  # original did “select current_date” from SQLite
if re.fullmatch(r"^\d+$", periodspec) != None :
period = int(periodspec)
else :
period = \
{
"today" : lambda : today + "%",
"this-month" : lambda : today[0:8] + "%",
"this-year" : lambda : today[0:5] + "%",
"all" : lambda : "%",
}.get(periodspec, lambda : None)()
if period == None :
raise RuntimeError("{}: unrecognized period spec “%s”" % periodspec)
#end if
#end if
sys.stderr.write("period = %s\n" % repr(period)) # debug
# ...
# vmstat_params = (period, slice_length)
# ... etc

So you would call that like

./demo 5
./demo today
./demo

etc. And as a bonus, it will accept any integer interval, not just your 
original limited set. And you see how I wrote the table of symbolic intervals? 
That makes it easy to add new entries.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Lawrence D’Oliveiro]

On Friday, August 12, 2016 at 2:25:05 AM UTC+12, Terry Reedy wrote:

> When I read something like "Python finally acquired an event loop in 
> 3.4" I wonder where people have been.  The tk event loop has been in 
> Python perhaps for 2 decades...

As was pointed out to me just a few days ago, that’s Tcl, not Python.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python 3.5.2 lounch: api-ms-win-crt-runtime-l1-1-0.dll is missing ?

[Lawrence D’Oliveiro]

On Tuesday, August 9, 2016 at 6:46:52 AM UTC+12, Joaquin Alzola wrote:

> This email is confidential and may be subject to privilege. If you are not
> the intended recipient, please do not copy or disclose its content but
> contact the sender immediately upon receipt.

Consider yourself contacted!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Call for Assistance

[Lawrence D’Oliveiro]

On Wednesday, August 10, 2016 at 3:42:39 AM UTC+12, Reto Brunner wrote:
> What on earth isn't "free" enough about
> 
> You are free to:
> Share — copy and redistribute the material in any medium or format

No you are not. A court has ruled 

 quite clearly that “non-commercial” means PERSONAL USE ONLY, no redistribution 
or republication.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Call for Assistance

[Lawrence D’Oliveiro]

On Wednesday, August 10, 2016 at 1:09:31 PM UTC+12, Charles Ross wrote:
> Well, I’ve been convinced. The license for the book is now Creative
> Commons Attribution-ShareAlike.

Great. :)

That’s the one to use if you are worried about companies who want something for 
nothing coming in and ripping you off.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Paul Rubin]

Steven D'Aprano  writes:
> How do I write work() so that it cooperatively multi-tasks with other ...
> threads? processes? what the hell do we call these things? What does this
> example become in the asynchronous world?

If it's heavily computational then you have to yield to the scheduler
frequently so that other tasks have a chance to run.  So if your 0.2
second computation involves 100 iterations of something, you could yield
once per iteration (every 2 msec).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Call for Assistance

[Paul Rubin]

Charles Ross  writes:
> Well, I’ve been convinced. The license for the book is now Creative
> Commons Attribution-ShareAlike.

That means you can post it on wikibooks.org and let people edit it
directly, if you want.  Wikibooks is nowhere near as crazy as wikipedia.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asynchronous programming

[Paul Rudin]

Steven D'Aprano  writes:

> Thanks to everyone who has answered, I think I'm slowly starting to get it
> now. Let's see if we can come up with a toy example that doesn't involve
> low-level socket programming :-)
>
> Let me simulate a slow function call:
>
>
> import random, time
>
> def work(id):
> print("starting with id", id)
> workload = random.randint(5, 15)
> for i in range(workload):
> time.sleep(0.2)  # pretend to do some real work
> print("processing id", id)  # let the user see some progress
> print("done with id", id)
> return 10 + id
>
>
> pending = [1, 2, 3, 4]
>
> for i, n in enumerate(pending):
> pending[i] = work(n)
>
>
> How do I write work() so that it cooperatively multi-tasks with other ...
> threads? processes? what the hell do we call these things? What does this
> example become in the asynchronous world?
>

They're not separate processes or threads, just think tasks that suspend
and restart cooperatively. You need some kind of event loop to
orchestrate running the tasks. Asyncio provides one.

You can do your example like this:

import asyncio, random

async def work(id):
print("starting with id", id)
workload = random.randint(5, 15)
for i in range(workload):
await asyncio.sleep(0.2)  # pretend to do some real work
print("processing id", id)  # let the user see some progress
print("done with id", id)
return 10 + id

loop = asyncio.get_event_loop()

pending = [1, 2, 3, 4]

jobs = [asyncio.ensure_future(work(n)) for n in pending]
 
loop.run_until_complete(asyncio.gather(*jobs))

loop.close()


> In this case, all the work is pure computation, so I don't expect to save
> any time by doing this, because the work is still all being done in the
> same process/thread, not in parallel. It may even take a little bit longer,
> due to the overhead of switching from one to another.
>

Yeah - you're not gaining anything here. All (?) the interesting use
cases involve waiting for something (e.g. disk access, network
communication, separate process) before you can proceed with your
computation.



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Call for Assistance

[Andreas Röhler]

On 12.08.2016 06:48, Lawrence D’Oliveiro wrote:

On Wednesday, August 10, 2016 at 3:42:39 AM UTC+12, Reto Brunner wrote:

What on earth isn't "free" enough about

You are free to:
Share — copy and redistribute the material in any medium or format

No you are not. A court has ruled 

 quite clearly that “non-commercial” means PERSONAL USE ONLY, no redistribution or 
republication.


The court ruled at the special case. The institution in question is paid 
by all germans due to a kind of fea imposed - "Rundfunkbeitrag". 
Everyone owning or renting an object in Germany has to pay to them - 
without regard if they uses that media or not. If you have to pay to 
someone, you would not consider that relation non-commercial.

--
https://mail.python.org/mailman/listinfo/python-list