Re: Multiple python versions, one dev environment???

[Akira]

Joep van Delft  wrote:
> Hello! 
> 
> The condensed version of the question would probably be: How does one
> deal with multiple interpreters and one package where you want to try
> some changes? 

You could use tox to test a package using different Python versions.


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Updating new e-books!!

[akira]

Hello!! once again i'm updating my e-books!!
Check this out!!
http://freebooks2007.blogspot.com

thank youu!! have a nice day! ^^

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: User Interface Suggestions? (newbie)

[Akira Li]

Beverly Howard  writes:

>...snip...
> A primary question would be, "What are options for building a display
> that would update displayed values without scrolling?"

To rewrite only the last character, you could use '\b':

  import os
  import itertools
  import time
  for c in map(str.encode, itertools.cycle('\-/|')):
  os.write(1, b'\b'+c)
  time.sleep(.3)

To rewrite only the last line, you could use '\r':

  for i in range(1, 101):
  time.sleep(.1)
  print('\r{:4.0%}'.format(i/100), flush=True, end='')

To control the full screen in the terminal e.g., to change the position,
you could use *blessings* module [1]:

  from blessings import Terminal  # $ pip install blessings

  line = 'Line in the middle of the terminal.'
  term = Terminal()
  with term.hidden_cursor(), term.fullscreen():
  x = (term.width - len(line)) // 2
  y = (term.height - 1) // 2
  with term.location(x, y):
  print(term.bold_white_on_black(line))
  
  with term.location(0, term.height - 1):
  input('press  to exit..')
  
For interactive command-line applications from a simple prompt to full
screen terminal applications, you could use *prompt_toolkit* module [2].

> My first goal is to build a program that would interface with a
> Raspberry Pi to control a heating process.
>

For flexibility, you could split your program into a server and a client
that controls it (text, GUI, web).

To control the Raspberry Pi from your mobile device, you could try Kivy
[3] or just create something quick in Pythonista 3 on iOS [4].  To create
a GUI on desktop, you could try Tkinter, Gtk, Qt frameworks [5,6].

For the web part on the server, you could try *flask*, *bottle* modules
[7], to return json data to a client using http protocol:

  from bottle import route # $ pip install bottle

  @route('/data')
  def data():
  return dict(data=[1,2,3])

On the (text/GUI) client in Python:

  import requests # $ pip install requests

  data = requests.get('https://your.server.example.com/data').json()

Or in the browser using jQuery [8]:

  $.getJSON('https://your.server.example.com/data', function(data) {
  //use data here
  });

To make the interaction more responsive, you could use WebSocket protocol e.g., 
via
Flask-SocketIO on the server [9].

[1]: https://pypi.python.org/pypi/blessings/
[2]: https://python-prompt-toolkit.readthedocs.io/
[3]: https://kivy.org
[4]: http://omz-software.com/pythonista/
[5]: http://www.tkdocs.com/tutorial/onepage.html
[6]: http://doc.qt.io/qt-4.8/gallery-gtk.html
[7]: http://bottlepy.org/docs/stable/
[8]: http://api.jquery.com/jquery.getjson/
[9]: https://flask-socketio.readthedocs.io

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Interacting with Subprocesses

[Akira Li]

Dick Holmes  writes:

> I am attempting to write a Python program that will interact with
> a (non-Python) process. The programs will run under MinGW. The
> process can use stdin/stdout commands and responses and can work 
> with pipes. The problem I'm having is that I can't find any
> way in Python to have a continuing dialog with the process. I
> have tried Popen communicate, but that protocol seems to be
> limited to a single message/response pair, and the response
> is not returned to the message originator until the process
> terminates. Unfortunately I don't have access to the process'
> source code so I can't change the communication medium.
>
> Is there some feature that will allow me to initiate the process
> and execute multiple message/response pairs between the Python
> program and the process during a single execution of the process?
>

Pass stdin=PIPE, stdout=PIPE and use p.stdin, p.stdout file objects to
write input, read output from the child process.

Beware, there could be buffering issues or the child process may change
its behavior some other way when the standard input/output streams are
redirected. See
http://pexpect.readthedocs.io/en/stable/FAQ.html#whynotpipe

btw, If pexpect module works in MingGW environment (if pty is
available); you could try it to communicate with the process
interactively.

You might also find the list of Stackoverflow question related to the
subprocess module useful http://stackoverflow.com/tags/subprocess/info


Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Interacting with Subprocesses

[Akira Li]

Terry Reedy  writes:

> On 5/4/2016 2:41 PM, Dick Holmes wrote:
>> I am attempting to write a Python program that will interact with
>> a (non-Python) process. The programs will run under MinGW. The
>> process can use stdin/stdout commands and responses and can work
>> with pipes. The problem I'm having is that I can't find any
>> way in Python to have a continuing dialog with the process. I
>> have tried Popen communicate, but that protocol seems to be
>> limited to a single message/response pair, and the response
>> is not returned to the message originator until the process
>> terminates. Unfortunately I don't have access to the process'
>> source code so I can't change the communication medium.
>>
>> Is there some feature that will allow me to initiate the process
>> and execute multiple message/response pairs between the Python
>> program and the process during a single execution of the process?
>
> I have been told that multiprocessing works better for this.  Not sure
> is true.

OP wants to interact with a *non-Python* process—*multiprocessing*
module won't help here.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Saving a file "in the background" -- How?

[Akira Li]

Virgil Stokes  writes:

> While running a python program I need to save some of the data that is
> being created. I would like to save the data to a file on a disk
> according to a periodical schedule  (e.g. every 10
> minutes). Initially, the amount of data is small (< 1 MB) but after
> sometime the amount of data can be >10MB. If a problem occurs during
> data creation, then the user should be able to start over from the
> last successfully saved data.
>
> For my particular application, no other file is being saved and the
> data should always replace (not be appended to) the previous data
> saved. It is important that  the data be saved without any obvious
> distraction to the user who is busy creating more data. That is, I
> would like to save the data "in the background".
>
> What is a good method to perform this task using Python 2.7.8 on a
> Win32 platform?

There are several requirements:

- save data asynchroniously -- "without any obvious distraction to the
  user"
- save data durably -- avoid corrupting previously saved data or
  writing only partial new data e.g., in case of a power failure
- do it periodically -- handle drift/overlap gracefully in a documented
  way 

A simple way to do asynchronios I/O on Python 2.7.8 on a Win32 platform
is to use threads:

  t = threading.Thread(target=backup_periodically, kwargs=dict(period=600))
  t.daemon = True # stop if the program exits
  t.start()

where backup_periodically() backups data every period seconds: 

  import time
  
  def backup_periodically(period, timer=time.time, sleep=time.sleep):
  start = timer()
  while True:
  try:
  backup()
  except Exception: # log exceptions and continue
  logging.exception() 
  # lock with the timer 
  sleep(period - (timer() - start) % period) 

To avoid drift over time of backup times, the sleep is locked with the
timer using the modulo operation. If backup() takes longer than *period*
seconds (unlikely for 10MB per 10 minutes) then the step may be
skipped. 

backup() makes sure that the data is saved and can be restore at any
time. 

  def backup():
  with atomic_open('backup', 'w') as file: 
  file.write(get_data())

where atomic_open() [1] tries to overcome multiple issues with saving
data reliably:

- write to a temporary file so that the old data is always available
- rename the file when all new data is written, handle cases such as:
  * "antivirus opens old file thus preventing me from replacing it"

either the operation succeeds and 'backup' contains new data or it fails
and 'backup' contains untouched ready-to-restore old data -- nothing in
between. 

[1]: https://github.com/mitsuhiko/python-atomicfile/blob/master/atomicfile.py

I don't know how ready atomicfile.py but you should be aware of the
issues it is trying to solve if you want a reliable backup solution.


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Emulating py2exe for python version 3 and above

[Akira Li]

Ian Dickinson  writes:

> Can i emulate py2exe for python version 3 and above i also use pygame any
> suggestions for a basic staring script would be greatly appreciated
>
>

py2exe supports Python 3.3+ Actually, default installers from pypi
support only Python 3.

I see unofficial pygame's Windows installers for Python 3 at
http://www.lfd.uci.edu/~gohlke/pythonlibs/#pygame 


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Challenge: optimizing isqrt

[Akira Li]

Steven D'Aprano  writes:

> There is an algorithm for calculating the integer square root of any
> positive integer using only integer operations:
>
> def isqrt(n):
> if n < 0: raise ValueError
> if n == 0:
> return 0
> bits = n.bit_length()
> a, b = divmod(bits, 2)
> x = 2**(a+b)
> while True:
> y = (x + n//x)//2
> if y >= x:
> return x
> x = y
>
> This returns the integer part of the square root of n, that is, the greatest
> whole number less than or equal to the square root of n:
>
> py> isqrt(35)
> 5
> py> isqrt(36)
> 6
>
>
> That makes it equivalent to int(math.sqrt(n)), which also happens to be
> much, much faster, at least for small values of n. However, for large
> values of n, using floating point intermediate calculations fail:
>
> py> import math
> py> int(math.sqrt(2**3000))
> Traceback (most recent call last):
>   File "", line 1, in 
> OverflowError: long int too large to convert to float
>
> Another problem is that, above a certain size, the resolution of floats is
> larger than 1, so you can't convert every int into a float without loss:
>
> py> float(2**90-1) == 2**90-1
> False
>
> which means that using math.sqrt is not correct:
>
> py> isqrt(2**90-1)
> 35184372088831
> py> int(math.sqrt(2**90-1))  # Off by one.
> 35184372088832
>
>
> So, the challenge is to identify when it is safe to optimise isqrt(n) as
> int(math.sqrt(n)):
>
> Q1: What is the largest value of M, such that 
>
> all(isqrt(i) == int(math.sqrt(n)) for n in range(M))
>
> returns True?
>
> I have done a brute force test, and in nine hours confirmed that M is at
> least 7627926244. That took nine hours, and I expect that a brute force
> test of every int representable as a float would take *months* of
> processing time.
>
> Q2: For values above M, is there a way of identifying which values of n are
> okay to use the optimized version?
>
> Q3: What is the largest value of n beyond which you can never use the float
> optimization?
>
>
> You can assume that Python floats are IEEE-754 C doubles, and that
> math.sqrt() is correctly rounded.

Where do you want to use your optimized isqrt(i)? 

There could be specialized algorithms that work only in narrow specific
circumstances e.g., the inverse square root (1/sqrt(x)) implementation
from Quake III Arena that has 0x5f3759df constant in it (only of
historical interest now).

If you want to work with very large (thousands, millions of digits)
integers then gmp library might be faster then the default Python
integer implementation.


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Moving a window on the screen

[Akira Li]

"ast"  writes:

> Ok, thx, it works now with:
>
> import tkinter
> fen = tkinter.Tk()
>
> x=0
>
> def moveW():
>global x
>fen.geometry("200x200+%d+10"  %  x)
>x = x + 10
>if (x < 1200):
>fen.after(50, moveW)
>
> moveW() 

In general, to avoid the start time "drift" [1], you could lock the
execution with a timer e.g., to move the window from left to right
*delta_x* pixels at a time every *period* ms [2]:

  #!/usr/bin/env python3
  from time import monotonic
  from tkinter import Tk
  
  def timer():
  return int(monotonic() * 1000) # milliseconds
  
  def call_repeatedly(period, function, *args):
  root.after(period - timer() % period, call_repeatedly, period,
 function, *args) # schedule the next call
  function(*args)
  
  def move(delta_x, max_x, width=200, x=[0]):
  root.geometry("%dx50+%d+100"  %  (width, x[0]))
  x[0] += delta_x # poor man's object
  if x[0] > (max_x - width):
  root.destroy() # exit
  
  root = Tk()
  period = 20 # call every *period* milliseconds
  delta_x = 2 # how many pixels to move at a time
  root.after(period - period % timer(), call_repeatedly, period,
 move, delta_x, root.winfo_screenwidth())
  root.mainloop()
  

[1]: 
http://stackoverflow.com/questions/8600161/executing-periodic-actions-in-python#comment26637231_8600301
[2]: 
http://stackoverflow.com/questions/24174924/how-to-run-a-function-periodically-in-python


Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Moving a window on the screen

[Akira Li]

Terry Reedy  writes:

> On 11/8/2014 11:35 AM, Akira Li wrote:
>> "ast"  writes:
>>
>>> Ok, thx, it works now with:
>>>
>>> import tkinter
>>> fen = tkinter.Tk()
>>>
>>> x=0
>>>
>>> def moveW():
>>> global x
>>> fen.geometry("200x200+%d+10"  %  x)
>>> x = x + 10
>>> if (x < 1200):
>>> fen.after(50, moveW)
>>>
>>> moveW()
>>
>> In general, to avoid the start time "drift" [1],
>
> which is hardly noticeable

Do you mean hardly noticeable for these particular *period*, delta_x and
(small) fixed number of repeatitions?

That is why I said, "in general".

The link [1] that I've provided contains an example where it *is*
noticable.

>> you could lock the
>> execution with a timer e.g., to move the window from left to right
>> *delta_x* pixels at a time every *period* ms [2]:
>
> On my Win7 machine, your complicated code is much worse as it causes
> the window to jump about every half second

Could you add print(timer()) inside move(), to see the values? What
happens if you increase the period to 100 ms?

>>#!/usr/bin/env python3
>>from time import monotonic
>>from tkinter import Tk
>>
>>def timer():
>>return int(monotonic() * 1000) # milliseconds
>>
>>def call_repeatedly(period, function, *args):
>>root.after(period - timer() % period, call_repeatedly, period,
>
> The '- timer() % period' 'correction' is wrong when not 0 as it causes
> jumps.

The formula is correct. It is obvious if you make the period one second
(1000ms) and print the timer() values (in milliseconds):

  52002
  53000
  54000
  55000
  56001
  57000
  58000
  59001
  60001
  61000
  62000
  ...

As you can see the start time is locked with the timer: the function is
called at whole seconds. Individual calls may happens slightly sooner or
later but the timing doesn't drift, the difference
between the expected start time and the actual start time is 1ms:

  >>> M[0], M[-1], M[0] + 1000*(len(M)-1)
  (52002, 224001, 224002)

Here's the same thing if I remove the locking `-timer() % period` and
use just `root.after(period, call..)`:

  34235
  35236
  36236
  37236
  38236
  39237
  40237
  41237
  42237
  43238
  44238
  45238
  46238
  47239
  48239
  ...

The start time drifts:

  >>> L[0], L[-1], L[0] + 1000*(len(L)-1)
  (34235, 206279, 206235)

the difference between the expected start time and the actual start time
is 44ms (4400% worse than the previous method).

I agree, for the purpose of moving a window on the screen, the
difference doesn't matter though if it is a GUI clock then you should
not ignore it otherwise it will be wrong by a minute in a couple of
days.

>>   function, *args) # schedule the next call
>>function(*args)
>>
>>def move(delta_x, max_x, width=200, x=[0]):
>>root.geometry("%dx50+%d+100"  %  (width, x[0]))
>>x[0] += delta_x # poor man's object
>>if x[0] > (max_x - width):
>>root.destroy() # exit
>>
>>root = Tk()
>>period = 20 # call every *period* milliseconds
>>delta_x = 2 # how many pixels to move at a time
>>root.after(period - period % timer(), call_repeatedly, period,
>
> 'period % timer()' is nonsensical as timer() is arbitrary. It will
> typically be 0 anyway.  'after(0, ...)'  works fine.
>

It is a bug (the terms are swapped by mistake). Thank you for
noticing. It should be `timer() % period` instead. The same expression
as used in the call_repeatedly() and in the link [2] that I've provided.

`period - timer() % period` is used here to make the first start time at
the exact boundary so there is the same interval between function(*args)  calls.

[1]: 
http://stackoverflow.com/questions/8600161/executing-periodic-actions-in-python#comment26637231_8600301
[2]: 
http://stackoverflow.com/questions/24174924/how-to-run-a-function-periodically-in-python


Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: html page mail link to webmail program

[Akira Li]

Ethan Furman  writes:

> On 11/11/2014 05:08 PM, Ben Finney wrote:
>> Ethan Furman  writes:
>>
>>> My wife (using a Win7 machine) will be on a web page that has a link
>>> to mail somebody.  She clicks on it, and it opens the currently
>>> installed but unused Thunderbird.
>>>
>>> Ideally, what would happen is a new window/tab would open to gmail
>>> with a new compose window with the email address in place and the
>>> cursor in the subject line.
>>
>> What is the Python question? I can't see anywhere that you would be
>> using Python code to address this.
>
> Really?  Huh.
>
> Okay, the explicit Python question:  Clicking on a mail link in a web
> browser can start an external program.  I would like that external
> program to be a Python script that: opens a new tab in the currently
> running browser (or a new default browser window), loads up the
> default web mail client (or one specified if there is no way to
> know/have a default), navigates to the compose pane (or starts there
> if possible), enters in the email address from the link that was
> passed to it, and, if not too much more, move the cursor to the
> subject field.
>
> Surely this can be done in Python.
>
Related question:
http://stackoverflow.com/questions/14288177/interact-with-other-programs-using-python


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Curious function argument

[Akira Li]

"ast"  writes:

> Hello
>
> I saw in a code from a previous message in this forum
> a curious function argument.
>
> def test(x=[0]):
>   print(x[0])   ## Poor man's object
>   x[0] += 1
>
>>>> test()
> 0
>>>> test()
> 1
>>>> test()
> 2
>>>>
>
> I understand that the author wants to implement a global
> variable x . It would be better to write 'global x' inside the
> function.
>
> At first test() function call, it prints 0, that's OK.
> But at the second call, since we dont pass any argument to test(),
> x should be equal to its default value [0] (a single item list). But
> it seems that Python keeps the original object whose content has
> been changed to 1.
>
> Is it a usual way to implement a global variable ?
>
It is not a global, a global would be easily available (visible) to
other functions. You can get it if you try but Python also allows you to
get private attributes, replace builtins and other things that you
should not normally do.

*"# Poor man's object"* in this context suggests that I'm the
author. The code is meant as a part of a short Python script that is
executed directly. Imagine the *test()* definition is put inside main()
function that is called when the script is run.

`def test(*, _x=[0]):..` attaches some state (_x list) to the function
*test*. It could be used as a way to emulate *nonlocal* keyword:

  def counter():
  count = 0
  def increment():
  nonlocal count
  count += 1
  return count
  return increment
  
  c = counter()
  print(c(), c()) # 0, 1
  c = counter()
  print(c(), c()) # 0, 1

an alternative is to create an object using class:

  class Counter:
  def __init__(self):
  self.count = 0
  def __call__(self):
  self.count += 1
  return self.count

  c = Counter()
  print(c(), c())
  
For this specific case, you could use `itertools.count`:

  c = itertools.count()
  print(next(c), next(c))

which could be implemented here as a generator:

  def count():
  count = 0
  while True:
 count += 1  
 yield count

  c = count()
  print(next(c), next(c))

"Objects are data with methods attached, closures are functions with data
attached." [1]

Sometimes you need to define a class to create objects, in other cases a
function is enough (and of course functions are objects too in Python).

[1]
http://stackoverflow.com/questions/13857/can-you-explain-closures-as-they-relate-to-python


--
Akira


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asyncio problem, looking for advice.

[Akira Li]

Benjamin Risher  writes:

> Hello all,
>
> I'm working on a project to learn asyncio and network programming.  What I'm 
> trying to do is forward a connection from myself to another machine.  Kind of 
> like an asynchronous python implementation of fpipe.
>
> In a nutshell:
>
> 1 --> start a server listening on localhost
> 2 --> connect to server
> 3 --> server connects to a listening service (ssh for instance)
> 4 --> server handles both connections to pass traffic back and forth through 
> each
>
> What I have now *kind of* works.  It sends data back and forth, but when I 
> ssh to localhost -p 12345, i never get the password prompt.  It looks like 
> one packet hangs out and doesn't get sent from what I saw in tcpdump.
>
> Any help would be greatly appreciated.

Do you want to emulate `ssh -L 12345:localhost:22 `?

> Here's a link to the same code as below, just with syntax highlighting etc...
> http://pastebin.com/iLE4GZH3

There are several issue e.g., unnecessary async(), deprecated Task()
calls but the main issue is that _handle_client() doesn't read
concurrently from the server while client writes to it.

You could use asyncio.wait() to run several tasks in parallel
[1]. Here's a forward-port.py example [2]:

  #!/usr/bin/env python3
  """Forward a local tcp port to host:port.
  
  Usage: %(prog)s local_port:host:port
  
  Example:
  
$ python3 forward-port.py 26992:icanhazip.com:80 # start server
  
  and in another window:
  
$ curl localhost:26992 # connect to it
  """
  import asyncio
  import logging
  import sys
  
  info = logging.getLogger('forward-port').info
  
  @asyncio.coroutine
  def copy_stream(reader, writer, bufsize=1<<16):
  while True:
  data = yield from reader.read(bufsize)
  if not data:
  break
  writer.write(data)
  yield from writer.drain()
  writer.close()
  
  def port_forwarder(host, port, *, loop):
  @asyncio.coroutine
  def forward(local_reader, local_writer):
  client = local_writer.get_extra_info('peername')
  info('connected client %s %s', *client)
  remote_reader, remote_writer = yield from 
asyncio.open_connection(host, port, loop=loop)
  yield from asyncio.wait([copy_stream(local_reader, remote_writer),
   copy_stream(remote_reader, local_writer)],
  loop=loop)
  info('disconnected client %s %s', *client)
  
  return forward
  
  # main
  logging.basicConfig(level=logging.INFO,
  format="%(asctime)-15s %(message)s", datefmt="%F %T")
  if len(sys.argv) != 2:
  sys.exit(__doc__)
  local_port, host, port = sys.argv[1].split(':') # e.g., 12345:localhost:22
  
  loop = asyncio.get_event_loop()
  server = loop.run_until_complete(asyncio.start_server(port_forwarder(host, 
int(port), loop=loop),
'localhost', 
int(local_port), loop=loop))
  info('listening on: %s %s', *server.sockets[0].getsockname())
  for closing in range(2):
  try:
  loop.run_until_complete(server.wait_closed())
  except KeyboardInterrupt:
  if not closing:
  server.close()
  info('closing server')
  else:
  break
  info('done')
  loop.close()

[1]
https://docs.python.org/3/library/asyncio-task.html#example-parallel-execution-of-tasks

[2] http://pastebin.com/g08YaJyz


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: localization virt-manager

[Akira Li]

Беляев Игорь  writes:

> I can't install localization for Virt-manager (virt-manager launched on 
> python2.7 (Windows XP)).

virt-manager is a GUI for KVM, Xen, LXC virtual machines. It is a Linux
application.

> How do I correctly install location? 

Do you mean *locale*?

> How can I change the value of the environment variable LANG?

On Windows, you could use *setx* command to set an environment variable.

LANG envvar defines a default value for LC_* envvars on POSIX systems [1]
that define application's locale (after setlocale() call) e.g., in bash:

  $ LANG=en_US.UTF-8 some-program

I don't know whether LANG has any meaning on Windows.

[1] http://pubs.opengroup.org/onlinepubs/007908799/xbd/envvar.html


--
Akira


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Asyncio problem, looking for advice.

[Akira Li]

Benjamin Risher  writes:

> On Friday, November 28, 2014 6:12:20 AM UTC-6, Akira Li wrote:
>> Benjamin Risher writes:
>> 
>> > Hello all,
>> >
>> > I'm working on a project to learn asyncio and network programming.
>> > What I'm trying to do is forward a connection from myself to
>> > another machine.  Kind of like an asynchronous python
>> > implementation of fpipe.
>> >
>> > In a nutshell:
>> >
>> > 1 --> start a server listening on localhost
>> > 2 --> connect to server
>> > 3 --> server connects to a listening service (ssh for instance)
>> > 4 --> server handles both connections to pass traffic back and forth 
>> > through each
>> >
>> > What I have now *kind of* works.  It sends data back and forth,
>> > but when I ssh to localhost -p 12345, i never get the password
>> > prompt.  It looks like one packet hangs out and doesn't get sent
>> > from what I saw in tcpdump.
>> >
>> > Any help would be greatly appreciated.
>> 
>> Do you want to emulate `ssh -L 12345:localhost:22 `?
>> 
>> > Here's a link to the same code as below, just with syntax highlighting 
>> > etc...
>> > http://pastebin.com/iLE4GZH3
>> 
>> There are several issue e.g., unnecessary async(), deprecated Task()
>> calls but the main issue is that _handle_client() doesn't read
>> concurrently from the server while client writes to it.
>> 
>> You could use asyncio.wait() to run several tasks in parallel
>
>> [1]
>> https://docs.python.org/3/library/asyncio-task.html#example-parallel-execution-of-tasks
>> 
>> [2] http://pastebin.com/g08YaJyz
>> 


> Akira, 
>
> First, thank you very much for your response.  It helps tremendously.
> I have a question or two though, if you don't mind.
>
> You said Task() is deprecated, but it's not listed as such in the
> docs.  Is it just that it's preferred to use other methods instead of
> using Task() directly, or am I missing something?

asyncio is a provisional API [3], it may change without notice (though
it shouldn't without a reason). From asyncio docs [4]:

  Don’t directly create Task instances: use the async() function or the
  BaseEventLoop.create_task() method.

The reason is probably to support Trollius (asyncio for Python 2) [5].

> Also, just so I can wrap my head around things, I was trying to mix
> concurrent and procedural programming in the code I provided, correct?
> Do you have any advice on how to avoid mixing types again in the
> future?

In short, the main issue was that your code executed some parts
sequentially e.g., A then B:

  yield from A
  yield from B # this is not reached until A finishes

that needed to be run concurrently:

  yield from asyncio.wait([A, B])

or in general, if you don't need to wait the results:

  asyncio.async(A)
  asyncio.async(B) 

  # ... yield later, to pass the control to the event loop
(Task._step/_fut_waiter dance ☯)

Ask, if you have any specific questions about the code
http://pastebin.com/g08YaJyz 

There is a bug at the end. info('done') should be outside the loop
(improper indent).

[3] https://www.python.org/dev/peps/pep-0411
[4] https://docs.python.org/3/library/asyncio-task.html#asyncio.Task
[5] https://code.google.com/p/tulip/issues/detail?id=185


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Iterate over text file, discarding some lines via context manager

[Akira Li]

Ned Batchelder  writes:

> On 11/28/14 10:22 AM, Dave Angel wrote:
>> On 11/28/2014 10:04 AM, fetchinson . wrote:
>>> Hi all,
>>>
>>> I have a feeling that I should solve this by a context manager but
>>> since I've never used them I'm not sure what the optimal (in the
>>> python sense) solution is. So basically what I do all the time is
>>> this:
>>>
>>> for line in open( 'myfile' ):
>>>  if not line:
>>>  # discard empty lines
>>>  continue
>>>  if line.startswith( '#' ):
>>>  # discard lines starting with #
>>>  continue
>>>  items = line.split( )
>>>  if not items:
>>>  # discard lines with only spaces, tabs, etc
>>>  continue
>>>
>>>  process( items )
>>>
>>> You see I'd like to ignore lines which are empty, start with a #, or
>>> are only white space. How would I write a context manager so that the
>>> above simply becomes
>>>
>>> with some_tricky_stuff( 'myfile' ) as items:
>>>  process( items )
>>>
>>
>> I see what you're getting at, but a context manager is the wrong
>> paradigm.  What you want is a generator.   (untested)
>>
>> def mygenerator(filename):
>>  with open(filename) as f:
>>  for line in f:
>>  if not line: continue
>>  if line.startswith('#'): continue
>>  items = line.split()
>>  if not items: continue
>>  yield items
>>
>> Now your caller simply does:
>>
>> for items in mygenerator(filename):
>>process(items)
>>
>>
>
> I think it's slightly better to leave the open outside the generator:
>
> def interesting_lines(f):
> for line in f:
> line = line.strip()
> if line.startswith('#'):
> continue
> if not line:
> continue
> yield line
>
> with open("my_config.ini") as f:
> for line in interesting_lines(f):
> do_something(line)
>
> This makes interesting_lines a pure filter, and doesn't care what sort
> of sequence of strings it's operating on.  This makes it easier to
> test, and more flexible.  The caller's code is also clearer in my
> opinion.
>
> BTW: this example is taken verbatim from my PyCon presentation on
> iteration, it you are interested:
> http://nedbatchelder.com/text/iter.html

The conditions could be combined in this case:

  def iter_rows(lines):
  for line in lines:
  items = line.split()
  if items and not items[0].startswith('#'):
 yield items # space-separated non-emtpy non-comment items
  
  with open(filename):
  for items in iter_rows(file):
  process(items)


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Do you like the current design of python.org?

[Akira Li]

Peter Otten <__pete...@web.de> writes:

> Did you ever hit the "Socialize" button? Are you eager to see the latest 
> tweets when you are reading a PEP? Do you run away screaming from a page 
> where nothing moves without you hitting a button? Do you appreciate the 
> choice between ten or so links to the documentation?
>
> You can probably guess my opinion -- konqueror just crashed on the PEP index 
> and for some reason I'm more annoyed about the page than about the browser.
>
>
> PS: Is there a twitter.com something that I can block to trade black friday 
> and cyber monkey sales for a box with a good old error message?

I see that pythondotorg accepts pull requests and allows to report issues.
https://github.com/python/pythondotorg


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: time.monotonic() roll over

[Akira Li]

Ian Kelly  writes:

> On Thu, Dec 4, 2014 at 11:09 AM, Marko Rauhamaa  wrote:
>>
>> Chris Angelico :
>>
>> > It's not a Python issue. Python can't do anything more than ask the
>> > system, and if the system's value rolls over several times a year,
>> > Python can't magically cure that. The information has already been
>> > lost.
>>
>> Sure it could by having an invisible background thread occasionally call
>> time.monotonic(). It could even be done on the side without a thread.
>>
>> Anyway, the idea of a clock is complicated:
>>
>>  * the program could be stopped by a STOP signal
>>
>>  * the program could be suspended from power management
>>
>>  * the program could be resurrected from a virtual machine snapshot
>>
>>  * the program could be migrated from a different physical machine
>
> This seems like a lot of effort to unreliably design around a problem that
> will matter to only a tiny fraction of users.

- people's computers are mostly on batteries (laptops, tablets,
  smartphones) -- "suspended from power management" use case
- corporations's computations are mostly virtualized -- possible
  "ressurected", "migrated" use case

i.e., the opposite might be true -- non-virtualized PCs connected to AC
are (becoming) minority.


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: time.monotonic() roll over

[Akira Li]

Dave Angel  writes:

...many words about sleep()...
> Since the OS has no way of knowing whether the thing being waited for
> is a thread, another process, a human being, a network operation, or
> the end of the world, the interpretation of sleep needs to be the most
> conservative one.  There are many ways of suspending a process, and
> some of them will also suspend the external event.  Since the OS
> cannot know which case is significant, it has to return control to the
> caller at the soonest of the many possible interpretations.

That is why there is API (e.g., clock_nanosleep()) that allows us to
choose whether we need a relative delay (e.g., kill a subprocess if it
hasn't finished in 10 seconds) or an absolute deadline (e.g., these
lights should be on 9pm-6am local time).

*Both* use cases are valid.


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to make subprocess run for 60 sec?

[Akira Li]

Dan Stromberg  writes:

> On Fri, Dec 12, 2014 at 12:48 AM, Robert Clove  wrote:
>> Hi All,
>>
>> I have the following python script that runs.
>> I want is to run the subprocess to run for 60 sec and then send the SIGINT
>> signal to subprocess and write the output in file.
>>
>> #!/usr/bin/python
>> import os
>> import subprocess
>> PIPE = subprocess.PIPE
>> import signal
>> import time
>>
>> def handler(signum, frame):
>> pass
>>
>> signal.signal(signal.SIGALRM, handler)
>> signal.alarm(60)
>> command = "strace -c ./server"
>> os.chdir("/root/Desktop/")
>> p = subprocess.Popen(command, stdout=PIPE, stderr=PIPE)
>> time.sleep(60)
>> p.send_signal(signal.SIGINT)
>> signal.alarm(0)
>> print p.communicate()[1]
>
> Perhaps try something like:
>
> #!/usr/bin/python
>
> #import os
> import subprocess
> PIPE = subprocess.PIPE
> import signal
> import time
>
> #def handler(signum, frame):
> #pass
>
> with open('output.txt', 'w') as out_file:
> #signal.signal(signal.SIGALRM, handler)
> #signal.alarm(60)
> #command = "strace -c ./server"
> command = "./test-script"
> #os.chdir("/root/Desktop/")
> p = subprocess.Popen(command, stdout=out_file, stderr=out_file, 
> shell=True)
> time.sleep(5)
> p.send_signal(signal.SIGINT)
> #signal.alarm(0)
> #print p.communicate()[1]
>
> with open('output.txt', 'r') as in_file:
> output = in_file.read()
>
> print(output)

You should probably call p.wait() after p.send_signal(), to wait until
the child process exits (possibly properly flushing its stdout buffer). 
It might make the output.txt content less garbled.

>
> BTW, killing an active strace may leave strace's subprocess stuck in
> an unkillable state.
>
> Also note that this will sleep for n seconds whether the subprocess
> takes that long or not.

The answer on StackOverflow [1] shows how to avoid waiting n seconds if the
subprocess finishes sooner.

[1] 
http://stackoverflow.com/questions/27443480/how-to-make-subprocess-run-for-60-sec

> If you just want a program that does this, and it doesn't have to be
> in Python, you might try
> http://stromberg.dnsalias.org/~strombrg/maxtime.html
> It's in C, and is the product of considerable real-world use.  It
> exits almost immediately after its subprocess exits, FWIW.


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Creating interactive command-line Python app?

[Akira Li]

Steven D'Aprano  writes:

> rfreundlic...@colonial.net wrote:
>
>> um, what if I want to USE a command line for python WITHOUT downloading or
>> installing it
>
> Who are you talking to? What is the context?
>
> Like all software, you can't use Python apps without all their dependencies
> being installed. If you use the Linux operating system, it will have Python
> already installed. Otherwise, you will have to install it.
>
> If you can't install it, or don't want to, you can't use Python.

cx_Freeze, PyInstaller, py2exe, etc allow to create a standalone
distribution i.e., you could ship your executable with a bundled Python
interpreter.


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Simple background sound effect playback

[Akira Li]

"Jacob Kruger"  writes:

> Would prefer to use something free, that could work somewhat
> cross-platform, but, my primary target is for windows OS, and would
> primarily just want to be able to easily trigger playback of either
> .wav or .mp3 background sound effects, but, yes, would also be nice to
> be able to control them a little bit in terms of volume, possibly
> stereo panning as well as playback rate/frequency/pitch?
>
> I have used something called sound_lib, as well as another one
> relating to a sort of windows directSound effect, but, both of them
> had issues when working with either py2exe or cx_freeze when it came
> to compiling into executable, and main thing is would like to keep it
> simple...
>
> Suggestions?
>

You could try GStreamer: it is free, cross-platform, it allows you to
play media files in the background, to control volume, stereo panning
(e.g., GstAudioPanorama), to change the playback rate while preserving
pitch, it can use DirectShow on Windows, etc -- see
http://gstreamer.freedesktop.org/features/

The downside is that it may be complex to install and use e.g., it
probably works with py2exe but it won't be simple to configure.

If you know other library that provides similar feature list while being
less complex; do tell.


--
Akira.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: httplib with NETRC authentication

[Akira Li]

Chris Angelico  writes:

> On Wed, May 14, 2014 at 9:33 AM, pratibha natani  
> wrote:
>> I am trying to establish http connection to a gerrit host using
>> netrc authentication. I have a netrc file created with following
>> entries:
>> machine host1.com login name password pass
>>
>> I did a debug and saw that my netrc file is being read
>> correctly. Also in the connection object(after sending request) a
>> header got created with appropriate credentials:
>>  'headers': {'Authorization': 'Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ='}
>>
>> Still I get 401,Unauthorized in response. Any help would be greatly 
>> appreciated!
>
> The obvious question is: What *does* work? Does it work when you use
> wget, or some other application? Then go and look at what that sends
> for its authentication headers. Tip: It won't be "Authorization".

"Authorization" is the appropriate header for basic http authentication:
http://tools.ietf.org/html/rfc2617#section-2

Here's a code example for urllib2:
https://gist.github.com/kennethreitz/973705#comment-56387


--
akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python3; ftplib: TypeError: Can't convert 'bytes' object to str implicitly

[Akira Li]

Antoon Pardon  writes:

> This is the code I run (python 3.3)
>
> host = ...
> user = ...
> passwd = ...
>
> from ftplib import FTP
>
> ftp = FTP(host, user, passwd)
> ftp.mkd(b'NewDir')
> ftp.rmd(b'NewDir')
>
> This is the traceback
>
> Traceback (most recent call last):
>   File "ftp-problem", line 9, in 
> ftp.mkd(b'NewDir')
>   File "/usr/lib/python3.3/ftplib.py", line 612, in mkd
> resp = self.voidcmd('MKD ' + dirname)
> TypeError: Can't convert 'bytes' object to str implicitly
>
> The problem is that I do something like this in a backup program.
> I don't know the locales that other people use. So I manipulate
> all file and directory names as bytes.
>
> Am I doing something wrong?

The error message shows that ftplib expects a string here, not bytes.
You could use `ftp.mkd(some_bytes.decode(ftp.encoding))` as a
workaround.


--
akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: httplib with NETRC authentication

[Akira Li]

pratibha natani  writes:

> Hi,
>
> I am trying to establish http connection to a gerrit host using netrc 
> authentication. I have a netrc file created with following entries:
> machine host1.com login name password pass
>
> I did a debug and saw that my netrc file is being read correctly. Also in the 
> connection object(after sending request) a header got created with 
> appropriate credentials:
>  'headers': {'Authorization': 'Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ='}
>
It should be QWxhZGRpbjpvcGVuIHNlc2FtZQ== (note: the second '=')

Also, make sure that you use the correct case for the username. Userids
might be case sensitive.


--
akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python3; ftplib: TypeError: Can't convert 'bytes' object to str implicitly

[Akira Li]

Antoon Pardon  writes:

> op 14-05-14 18:24, Akira Li schreef:
>> Antoon Pardon  writes:
>>
>>> This is the code I run (python 3.3)
>>>
>>> host = ...
>>> user = ...
>>> passwd = ...
>>>
>>> from ftplib import FTP
>>>
>>> ftp = FTP(host, user, passwd)
>>> ftp.mkd(b'NewDir')
>>> ftp.rmd(b'NewDir')
>>>
>>> This is the traceback
>>>
>>> Traceback (most recent call last):
>>>   File "ftp-problem", line 9, in 
>>> ftp.mkd(b'NewDir')
>>>   File "/usr/lib/python3.3/ftplib.py", line 612, in mkd
>>> resp = self.voidcmd('MKD ' + dirname)
>>> TypeError: Can't convert 'bytes' object to str implicitly
>>>
>>> The problem is that I do something like this in a backup program.
>>> I don't know the locales that other people use. So I manipulate
>>> all file and directory names as bytes.
>>>
>>> Am I doing something wrong?
>>
>> The error message shows that ftplib expects a string here, not bytes.
>> You could use `ftp.mkd(some_bytes.decode(ftp.encoding))` as a
>> workaround.
>
> Sure but what I like to know: Can this be considered a failing of
> ftplib. Since python3 generally allows paths to be strings as
> well as bytes can't we expect the same of ftplib?
>
> Especially as I assume that path will be converted to bytes anyway
> in order to send it over the network.

bytes are supported for filenames because POSIX systems provide
bytes-based interface e.g., on my system anything except / and NUL could
be used. You can get away with passing opaque bytes filenames for some
time.

rfc 959 expects ascii filenames. rfc 2640 recommends UTF8 (if "feat"
command returns it). rfc 3659: pathnames could be send as utf-8 *and*
"raw". (plus CR LF or CR NUL or IAC or other telnet control codes
handling). Using utf-8 might have security implications and some
firewalls might interfere with OPTS command and FEAT response. Popular
clients such as FileZilla may break on non-utf-8 filenames.

It is less likely that ftp clients use the same character encoding and
it is more likely that an ftp server performs some unexpected character
encoding conversion despite it being non-standard-compliant.

You could try to post on python-ideas mailing list anyway, to suggest
the enhancement (support bytes where filenames are expected) for your
backup application use case:

- you might not avoid undecodable filenames -- UnicodeEncodeError in the
  current implementation if you pass Unicode string created using
  os.fsdecode(undecodable_bytes) to ftplib

- non-python ftp clients should be able to access the content -- no
  Python error handlers such as surrogateescape or backslashreplace are
  allowed

- set ftp.encoding to utf-8 and pass non-utf-8 filenames as bytes -- to
  avoid '\U0001F604'.encode('utf-8').decode(ftp.encoding)


--
akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Segmentation fault (core dumped) while using Cplex Python API

[Akira Li]

varun...@gmail.com writes:

> Hello Everyone, 
>
> I am trying to solve a mixed-integer problem using Cplex Python API
> and I get this error Segmentation fault (core dumped). i am not able
> to figure out the reason for this.
> Traceback
>
> srva@hades:~$ python RW10.py --output test --logPath log --xml topology.xml
> Start Time: 2014-6-1-20-56-39
> CPLEX Parameter File Version 12.5.0.0
> CPX_PARAM_TILIM 3600
> CPX_PARAM_TRELIM 2.00
> CPX_PARAM_EPGAP 0.03
> CPX_PARAM_EACHCUTLIM 21
> CPX_PARAM_FLOWCOVERS 0
> CPX_PARAM_FLOWPATHS 0
> CPX_PARAM_CLIQUES 0
> CPX_PARAM_DISJCUTS 0
> CPX_PARAM_COVERS 0
> CPX_PARAM_ZEROHALFCUTS 0
> CPX_PARAM_MIRCUTS 0
> CPX_PARAM_MCFCUTS 0
> CPX_PARAM_IMPLBD 0
> CPX_PARAM_GUBCOVERS 0
> CPX_PARAM_AGGCUTLIM 3
>
> Completion Time: 972.63
> Num Columns: 1350888
> Num Rows: 25488
> Solving
> Segmentation fault (core dumped)
>
> A few lines from the log file:
>
> Completion Time: 972.63
> Num Columns: 1350888
> Num Rows: 25488
> Tried aggregator 1 time.
> MIP Presolve eliminated 12648 rows and 1321088 columns.
> MIP Presolve modified 180 coefficients.
> Reduced MIP has 12840 rows, 29800 columns, and 136000 nonzeros.
> Reduced MIP has 29800 binaries, 0 generals, 0 SOSs, and 0 indicators.
> Presolve time = 0.49 sec. (368.57 ticks)
>
> I would be grateful if someone can help me fix this.

Install and enable faulthandler
https://pypi.python.org/pypi/faulthandler/
to see where Segmentation fault happens


--
akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Matplotlib X-axis timezone trouble

[Akira Li]

Peter Pearson  writes:

> The following code produces a plot with a line running from (9:30, 0) to
> (10:30, 1), not from (8:30, 0) to (9:30, 1) as I desire.
>
> If I use timezone None instead of pacific, the plot is as desired, but
> of course that doesn't solve the general problem of which this is a
> much-reduced example.
>
> If I use timezone US/Central, I get the same (bad) plot.
>
> import matplotlib.pyplot as plt
> import datetime
> import pytz
> pacific = pytz.timezone("US/Pacific")
> fig = plt.figure()
> plt.plot([datetime.datetime(2014, 10, 7, 8, 30, tzinfo=pacific),
>   datetime.datetime(2014, 10, 7, 9, 30, tzinfo=pacific)],
>  [0,1], marker="o", color="green")
> fig.autofmt_xdate()
> plt.show()
>
> Does anybody know why this shift is occurring?  Is Matplotlib
> confused about what timezone to use in labeling the axis?  How
> would I tell it what timezone to use (preferably explicitly in
> the code, not in matplotlibrc)?
>

Your pytz usage is incorrect.

Don't pass a pytz tzinfo object to the datetime construtor directly, use
`.localize()` method instead. Read the note at the very beginning of
pytz docs http://pytz.sourceforge.net/


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Convert between timezones

[Akira Li]

Thomas 'PointedEars' Lahn  writes:

> [X-Post & F'up2 comp.unix.shell]
>
> Chris Angelico wrote:
>
>> On Fri, Jul 31, 2015 at 6:15 PM, Cameron Simpson  wrote:
>>> Actually, bash has no timezone support but the date command _does_, and
>>> probably neither better nor worse than Python. All one has to do is set
>>> the TZ environment variable, eg (untested):
>>>
>>>  _year_gmt=$( TZ=GMT date +%Y )
>> 
>> That's assuming that it's converting against the current system
>> timezone. I don't know how you'd use `date` to convert between two
>> arbitrary timezones. […]
>
> With POSIX date(1), ISTM all you could do is set the system time and for an 
> additional invocation the TZ variable accordingly for output.
>
> 
>
> With GNU date(1):
>
> $ (tz_source="Asia/Dubai"; time_source="$(LC_TIME=C TZ=$tz_source date -d 
> "today 00:00 UTC+4" -Im)"; tz_target="America/Chicago"; echo "When it was 
> $time_source in $tz_source, it was $(LC_TIME=C TZ=$tz_target date -d 
> "$time_source") in $tz_target.")
> When it was 2015-07-31T00:00+0400 in Asia/Dubai, it was Thu Jul 30 15:00:00 
> CDT 2015 in America/Chicago.
>
> $ date --version
> date (GNU coreutils) 8.23
> […]
>

Here's a corresponding Python code. I haven't seen the beginning of the
discussion. I apologize if it has been already posted:

  #!/usr/bin/env python
  from datetime import datetime
  import pytz # $ pip install pytz
  
  source_tz, target_tz = map(pytz.timezone, ['Asia/Dubai', 'America/Chicago'])
  d = datetime.now(source_tz) # the current time in source_tz timezone
  midnight = source_tz.localize(datetime(d.year, d.month, d.day), is_dst=None)  
  
  fmt = "%Y-%m-%dT%H:%M:%S%z"
  print("When it was {:{fmt}} in {}, it was {:{fmt}} in {}".format(
  midnight, source_tz.zone, target_tz.normalize(midnight),
  target_tz.zone, fmt=fmt))

Output:

  When it was 2015-08-01T00:00:00+0400 in Asia/Dubai, it was
  2015-07-31T15:00:00-0500 in America/Chicago





-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Who uses IDLE -- please answer if you ever do, know, or teach

[Akira Li]

Terry Reedy  writes:

> There have been discussions, such as today on Idle-sig , about who
> uses Idle and who we should design it for.  If you use Idle in any
> way, or know of or teach classes using Idle, please answer as many of
> the questions below as you are willing, and as are appropriate
>

Running Python scripts in Windows console that may produce Unicode
output is a worth-mentioning use-case [1] (as you might know) i.e., if
you run:

  T:\> py print_unicode.py

and get the error:

  UnicodeEncodeError: 'charmap' codec can't encode character '...'

then a workaround that works out of the box is to run:

  T:\> py -m idlelib -r print_unicode.py

that can display Unicode (BMP) output in IDLE.

[1] 
http://stackoverflow.com/questions/28521944/python3-print-unicode-to-windows-xp-console-encode-cp437

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Sandboxing Python

[Akira Li]

Mark Lawrence  writes:

> I was always led to believe that the subject was a difficult thing to
> do, but here
> https://www.reddit.com/r/learnpython/comments/3huz4x/how_to_do_math_inside_raw_input/
> is a safe solution in only 23 characters, or are there any discernable
> flaws in it?

Related: 
http://stackoverflow.com/questions/2371436/evaluating-a-mathematical-expression-in-a-string

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: asyncio, coroutines, etc. and simultaneous execution

[Akira Li]

Charles Hixson  writes:

> If I understand correctly asyncio, coroutines, etc. (and, of course,
> Threads) are not simultaneously executed, and that if one wants that
> one must still use multiprocessing.  But I'm not sure.  The note is
> still there at the start of threading, so I'm pretty sure about that
> one.

Due to GIL (used by CPython, Pypy, not used by Jython, IronPython,
pypy-stm) only one thread executes Python bytecode at a time. GIL can be
released on I/O or in a C extension such as numpy, lxml, regex. It is
true for any Python module or concept you use.

Unrelated: use concurrent and parallel execution instead of
"simultaneously executed."

Parallelism might make a program faster (it implies that hardware
supports it).

Concurrency is a way to structure the code. The same concurrent program
can run in parallel and without parallelism.

Recommended: "Concurrency is not Parallelism
(it's better!)" talk by Rob Pike's talk
http://stackoverflow.com/questions/1050222/concurrency-vs-parallelism-what-is-the-difference


> The requirement that coroutines always be awaited seems to confirm
> this, but doesn't really say so explicitly. And the concurrent.futures
> can clearly be either, depending on your choices, but the chart in
> 18.5.3.1.3 Example: Chain coroutines is of a kind that I am more
> familiar with in the context of multiprocessing. (E.g., the only gap
> in the chart, which extends across all headings is when a result is
> being waited for during a sleep.)  For threaded execution I would
> expect there to be a gap whenever processing is shifted from one
> column to another.
>
> If someone has authority to edit the documentation a comment like:

Anybody can suggest a patch
https://docs.python.org/devguide/docquality.html

> If you want your application to make better use of the computational
> resources of multi-core machines, you are advised to use
> multiprocessing
> 
> or concurrent.futures.ProcessPoolExecutor
> .
>  However,
> threading is still an appropriate model if you want to run multiple
> I/O-bound tasks simultaneously.
>
> (to quote from the threading documentation) would be helpful at or
> very near the top of each of the appropriate modules.  It would also
> be useful if the modules that were definitely intended to result in
> simultaneous execution, when feasible, were so marked quite near the
> top.

It is not necessary that multiprocessing will make your code faster on a
multi-core machine. It may make it slower depending on the task.

Performance optimization is a vast topic. Short remarks such as you've
suggested are likely misleading.

> OTOH, I may be mistaken about coroutines.  I haven't been able to tell.

Many cooperative multitasking implementations use a *single*
thread. There is a way to offload blocking code e.g.,
loop.run_in_executor() in asyncio.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: passing double quotes in subprocess

[Akira Li]

loial  writes:

> I need to execute an external shell script via subprocess on Linux.
>
> One of the parameters needs to be passed inside double quotes 
>
> But the double quotes do not appear to be passed to the script
>
> I am using :
>
> myscript = '/home/john/myscript'
> commandline = myscript + ' ' + '\"Hello\"'
>
> process = subprocess.Popen(commandline, shell=True, stdout=subprocess.PIPE, 
> stderr=subprocess.PIPE)
> output,err = process.communicate()
>
>
> if I make the call from another shell script and escape the double
> quotes it works fine, but not when I use python and subprocess.
>
> I have googled this but cannot find a solution...is there one?

You don't need shell=True here:

  #!/usr/bin/env python3
  from subprocess import Popen, PIPE

  cmd = ['/home/john/myscript', 'Hello'] # if myscript don't need quotes
  # cmd = ['/home/john/myscript', '"Hello"'] # if myscript does need quotes
  with Popen(cmd, stdout=PIPE, stderr=PIPE) as process:
 output, errors = process.communicate()

In general, to preserve backslashes, use raw-string literals:

  >>> print('\"')
  "
  >>> print(r'\"')
  \"
  >>> print('\\"')
  \"


  >>> '\"' == '"'
  True 
  >>> r'\"' == '\\"'
  True

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python handles globals badly.

[Akira Li]

Vladimir Ignatov  writes:

>>> I had some experience programming in Lua and I'd say - that language
>>> is bad example to follow.
>>> Indexes start with 1  (I am not kidding)
>>
>> What is so bad about that?
>
> It's different from the rest 99.9% of languages for no particular reason.
>
> ( => perfect example of "design smell" => not a good example to follow)
>

It is not just a matter of tradition. Even if you were to invent the
very first programming language; there are reasons to prefer the
zero-based indexing. See "Why numbering should start at zero"
https://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html




-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Context-aware return

[Akira Li]

Grant Edwards  writes:

> On 2015-09-10, Steven D'Aprano  wrote:
>
>> I have a function which is intended for use at the interactive interpreter,
>> but may sometimes be used non-interactively. I wish to change it's output
>> depending on the context of how it is being called.
>
> [...]
>
> Sounds like an excellent way to waste somebody's afternoon when they
> start to troubleshoot code that's using your function.  Over and over
> and over we tell newbies who have questions about what something
> returns or how it works
>
> "Start up an interactive session, and try it!".
>
> If word gets out about functions like yours, we sort of end up looking
> like twits.  
>
>> If I did this thing, would people follow me down the street booing
>> and jeering and throwing things at me?
>
> Only the people who use your function. :)

There are cases when it might be justified to alter the behavior e.g.,
*colorama* allows to strip ANSI codes (e.g., disable colored output) if
stdout is not a tty or *win-unicode-console* make sys.stdout to use
WriteConsoleW() to write Unicode to Windows console (interactive case).

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Terminology: "reference" versus "pointer"

[Akira Li]

Rustom Mody  writes:

> On Saturday, September 12, 2015 at 8:11:49 PM UTC+5:30, Laura Creighton wrote:
>> In a message of Sat, 12 Sep 2015 05:46:35 -0700, Rustom Mody writes:
>> >How about lay-English ontology in which "point to" and "refer to" are fairly
>> >synonymous?
>> 
>> This I have found is important in teaching, which is why I favour 'bind'
>> and 'binding' -- rather than pointer, pointer, refer to, referring.
>
> Well we can play humpty dumpty and make any word mean whatever we like.
> However if you are a teacher you will recognize a need for pictures.
> And (as far as I can tell) "Random832" finds a need for the box-n-arrow
> diagrams of classic data-structure books

Speaking of pictures and names in Python
http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html#other-languages-have-variables


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Are there any "correct" implementations of tzinfo?

[Akira Li]

Random832  writes:

> I was trying to find out how arithmetic on aware datetimes is "supposed
> to" work, and tested with pytz. When I posted asking why it behaves this
> way I was told that pytz doesn't behave correctly according to the way
> the API was designed. The tzlocal module, on the other hand, appears to
> simply defer to pytz on Unix systems.
>
> My question is, _are_ there any correct reference implementations that
> demonstrate the proper behavior in the presence of a timezone that has
> daylight saving time transitions?

The only way to get correct[*] results now is to use *pytz*. PEP 495
might allow to fix some of non-avoidable (at the moment) bugs[1] in
*dateutil* (another library that provides access to the tz database).

Some core Python developers feel that pytz model does not implement the
initial datetime design intent: both naive and timezone-aware datetime
objects use the same model for arihtmetic (though the actual
implementation contains a mix: aware datetime objects are treated as
naive datetime objects in some cases but in others they behave as though
they are utc-based).

pytz model: aware datetime objects behave *as if* they are converted to
UTC during arithmetic operations, comparisons, etc:

  # d = datetime.now(tz)
  (d2 - d1) == (d2.astimezone(utc) - d1.astimezone(utc))
  tz.normalize(d + delta) == (d.astimezone(utc) + delta).astimezone(tz)

tz.normalize() is necessary to get the correct local time (utc time is
correct even without tz.normalize()) in presence of DST transitions (or
other changes in UTC offset for any reason).

Here's how the stdlib implementation behaves at the moment for a
timezone-aware datetime object that represents local time (the only
timezone with a non-fixed utc offset that is available in stdlib):

  # d = datetime.now(utc).astimezone()
  (d2 - d1) == (d2.astimezone(utc) - d1.astimezone(utc))
  (d + delta).astimezone() == (d.astimezone(utc) + delta).astimezone()

If utc offset is fixed then both naive and aware models are the same.

[*] according to the tz database
[1] https://github.com/dateutil/dateutil/issues/112

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Terminology: "reference" versus "pointer"

[Akira Li]

Rustom Mody  writes:

> On Saturday, September 12, 2015 at 11:26:18 PM UTC+5:30, Akira Li wrote:
>> Rustom Mody  writes:
>> 
>> > On Saturday, September 12, 2015 at 8:11:49 PM UTC+5:30, Laura Creighton 
>> > wrote:
>> >> In a message of Sat, 12 Sep 2015 05:46:35 -0700, Rustom Mody writes:
>> >> >How about lay-English ontology in which "point to" and "refer to" are 
>> >> >fairly
>> >> >synonymous?
>> >> 
>> >> This I have found is important in teaching, which is why I favour 'bind'
>> >> and 'binding' -- rather than pointer, pointer, refer to, referring.
>> >
>> > Well we can play humpty dumpty and make any word mean whatever we like.
>> > However if you are a teacher you will recognize a need for pictures.
>> > And (as far as I can tell) "Random832" finds a need for the box-n-arrow
>> > diagrams of classic data-structure books
>> 
>> Speaking of pictures and names in Python
>> http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html#other-languages-have-variables
>
> Yeah cute
> [I think I will even use these in my classes]
> However they dont address the issue that I think random832 is
> referring to.

The pictures despite their simplicity reflect the actual model that
Python language uses i.e., any deviations are an implementation artifact
and may be ignored.

> viz. I have two variables (or names!) say a and b which look the same
>>>> a
> [[1,2],[1,2]]
>>>> b
> [[1,2],[1,2]]
> And yet doing
>>>> a[0][0] = "Oops!"
> gives a data structure one "Oops!"
> whereas doing it to b mysteriously gives 2

Sorry, I haven't followed the whole thread. Could your provide a
complete code example? Mention what you expect to happen and what
happens instead in your case.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Terminology: "reference" versus "pointer"

[Akira Li]

Random832  writes:

> Akira Li <4kir4...@gmail.com> writes:
>>Rustom Mody  writes:
>>> viz. I have two variables (or names!) say a and b which look the same
>>>>>> a
>>> [[1,2],[1,2]]
>>>>>> b
>>> [[1,2],[1,2]]
>>> And yet doing
>>>>>> a[0][0] = "Oops!"
>>> gives a data structure one "Oops!"
>>> whereas doing it to b mysteriously gives 2
>>
>> Sorry, I haven't followed the whole thread. Could your provide a
>> complete code example? Mention what you expect to happen and what
>> happens instead in your case.
>
> a0 = a1 = [1, 2]
> b0 = [1, 2]
> b1 = [1, 2]
> a = [a0, a1]
> b = [b0, b1]
> del a0, a1, b0, b1
>
> There's nothing about *him* expecting anything wrong to happen. The
> question is how to draw a diagram that unambiguously shows the resulting
> structure using the "parcel tags" model shown in the diagrams (and
> without having a0/a1/etc as actual names)

I'm not sure what "parcel tags" model is but if you mean these
pictures[1] than it works in this case as well as any other (take *a*,
*b* nametags, put them on the corresponding balloons that represents
list objects).

The only names left are *a* and *b* that refer to the corresponding
lists. There is no ambiguity there to put *a*, *b* nametags.

Lists as any other containers contain references to other objects and
therefore "box and arrows" model provides _more details_ here[2,3]

> If the "parcel tags" model can't show it, then the "parcel tag" model
> clearly is not a correct and complete depiction of how Python actually
> works.
> (If I were drawing a picture rather than ASCII I'd add something to make
> it clear that the pairs shown are list objects Like, it's a circle with
> the word "list" and two pointer-boxes inside it.)

"correct and complete" is impossible in the general case for any model.

Imagine what happens if numpy arrays are used instead of Python lists:

  lst = [numpy.array([1, 2]) for _ in range(3)]
  a = [lst[0], lst[0]]
  b = [lst[1], lst[2]]

Note: there could be no a[0][0], a[0][1], etc corresponding Python
objects until you explicitly reference them in the code. If there are no
Python objects then you can't draw any arrows and therefore "box and
arrows" model is incomplete.

Or consider this collections of all Unicode characters:

  class U:
 def __len__(self):
 return sys.maxunicode + 1
 def __getitem__(self, index):
 if index < 0:
 index += len(self)
 if 0 <= index < len(self):
 return chr(index)
 raise IndexError(index)

  u = U()
  print(u[0x61]) # -> a

In this example, it is even more clear that the corresponding items
might not exist until they are explicitly referenced in a program. *u*
is a sequence as any other in Python (it is easy to make it
*collections.abc.Sequence* compatible).

Or this:

  lst = [range(1, 3) for _ in range(3)]
  a = [lst[0], lst[0]]
  b = [lst[1], lst[2]]

In Python 2, it is the exact replica of the original example. In Python
3, it is the same if you don't need to mutate the ranges. Again, the
leaf objects might not exist until you reference them in the code in
Python 3.

Finally, consider a Python sequence that uses os.urandom()
internally. No model will predict the result (and therefore no model is
complete) in this case.


[1]
http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html#python-has-names
[2]
http://www.pythontutor.com/visualize.html#code=a0+%3D+a1+%3D+%5B1,+2%5D%0Ab0+%3D+%5B1,+2%5D%0Ab1+%3D+%5B1,+2%5D%0Aa+%3D+%5Ba0,+a1%5D%0Ab+%3D+%5Bb0,+b1%5D%0Adel+a0,+a1,+b0,+b1%0A&mode=display&origin=opt-frontend.js&cumulative=false&heapPrimitives=false&textReferences=false&py=3&rawInputLstJSON=%5B%5D&curInstr=6
[3]
http://www.pythontutor.com/visualize.html#code=a0+%3D+a1+%3D+%5B1,+2%5D%0Ab0+%3D+%5B1,+2%5D%0Ab1+%3D+%5B1,+2%5D%0Aa+%3D+%5Ba0,+a1%5D%0Ab+%3D+%5Bb0,+b1%5D%0Adel+a0,+a1,+b0,+b1%0Aa%5B0%5D%5B0%5D+%3D+%22Oops!%22&mode=display&origin=opt-frontend.js&cumulative=false&heapPrimitives=false&textReferences=false&py=3&rawInputLstJSON=%5B%5D&curInstr=7


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Terminology: "reference" versus "pointer"

[Akira Li]

Random832  writes:

> Akira Li <4kir4...@gmail.com> writes:
>> I'm not sure what "parcel tags" model is but if you mean these
>> pictures[1] than it works in this case as well as any other (take *a*,
>> *b* nametags, put them on the corresponding balloons that represents
>> list objects).
>>
>> The only names left are *a* and *b* that refer to the corresponding
>> lists. There is no ambiguity there to put *a*, *b* nametags.
>
> But how do you make an a[0][0]/a[1][0] nametag to put on the "1" object?

a[0][0]/a[1][0] are not names. Though if we want to talk about the
corresponding objects then a[0][0]/a[1][0] could be used instead of
names (as a way to identify them). 

>> Lists as any other containers contain references to other objects and
>> therefore "box and arrows" model provides _more details_ here[2,3]
>
> Right, but why not use the *same* model to represent *namespaces*?

If it works for your case; use it. The links [2,3] show that It does
work in this case (if it is correct to call what they show "box and
arrows" model).

> It seems like the "tags" model only exists to make the incorrect claim
> that python doesn't have variables (the boxes are variables).

If you mean this quote from [1]:

  Although we commonly refer to "variables" even in Python (because it's
  common terminology), we really mean "names" or "identifiers". In
  Python, "variables" are nametags for values, not labelled boxes.

then *name* is the term that is defined in the Python language
reference. The word "variable" we can use in day-to-day programming
_unless_ we are discussing issues that are specific to _naming and
binding_.  In that case, we should use the _exact_
terminology. Otherwise there is nothing wrong with using "variables"
casually in Python.

Notice: that [2,3] model is different from "labelled boxes" model. There
are arrows from *a*/*b* _to_ list objects i.e., *a*/*b* are not boxes
that _contain_ list objects within --
_otherwise you have to put the *same* list into two *different* boxes_ --
let's not investigate quantum paradoxes here. The only difference from
"parcel tags" model is that list items do not have explicit names.

>>> If the "parcel tags" model can't show it, then the "parcel tag" model
>>> clearly is not a correct and complete depiction of how Python actually
>>> works.
>>> (If I were drawing a picture rather than ASCII I'd add something to make
>>> it clear that the pairs shown are list objects Like, it's a circle with
>>> the word "list" and two pointer-boxes inside it.)
>>
>> "correct and complete" is impossible in the general case for any
>> model.
>
> Everything you wrote here has the same issue: The "objects" you are
> talking about do not physically exist, but are simply the result of
> calling a method on the object. Therefore they do not *belong* on the
> diagram, and the diagram not showing them does not mean the diagram is
> not complete.

"do not physically exist" does not make sense. Objects are *never*
destroyed explicitly in Python (you can only make them
*unreachable*). You can disable garbage collection completely and it is
still will be Python. Immutable objects can be considered immortal e.g.:

 (1+1) -- question: does the object that represents int(2) exist before
the expression is evaluated?

The correct answer: it does not matter: int(2) can be created on the
fly, a cached int(2) can be reused by a specific implementation --
Python doesn't care.

I don't see why the model that can't describe range(1) in Python 3
pretends to be complete.

[1]
http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html#python-has-names
[2]
http://www.pythontutor.com/visualize.html#code=a0+%3D+a1+%3D+%5B1,+2%5D%0Ab0+%3D+%5B1,+2%5D%0Ab1+%3D+%5B1,+2%5D%0Aa+%3D+%5Ba0,+a1%5D%0Ab+%3D+%5Bb0,+b1%5D%0Adel+a0,+a1,+b0,+b1%0A&mode=display&origin=opt-frontend.js&cumulative=false&heapPrimitives=false&textReferences=false&py=3&rawInputLstJSON=%5B%5D&curInstr=6
[3]
http://www.pythontutor.com/visualize.html#code=a0+%3D+a1+%3D+%5B1,+2%5D%0Ab0+%3D+%5B1,+2%5D%0Ab1+%3D+%5B1,+2%5D%0Aa+%3D+%5Ba0,+a1%5D%0Ab+%3D+%5Bb0,+b1%5D%0Adel+a0,+a1,+b0,+b1%0Aa%5B0%5D%5B0%5D+%3D+%22Oops!%22&mode=display&origin=opt-frontend.js&cumulative=false&heapPrimitives=false&textReferences=false&py=3&rawInputLstJSON=%5B%5D&curInstr=7



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Terminology: "reference" versus "pointer"

[Akira Li]

Chris Angelico  writes:

> On Mon, Sep 14, 2015 at 9:17 AM, Akira Li <4kir4...@gmail.com> wrote:
>> If you mean this quote from [1]:
>>
>>   Although we commonly refer to "variables" even in Python (because it's
>>   common terminology), we really mean "names" or "identifiers". In
>>   Python, "variables" are nametags for values, not labelled boxes.
>>
>> then *name* is the term that is defined in the Python language
>> reference. The word "variable" we can use in day-to-day programming
>> _unless_ we are discussing issues that are specific to _naming and
>> binding_.  In that case, we should use the _exact_
>> terminology. Otherwise there is nothing wrong with using "variables"
>> casually in Python.
>
> Since you're talking about precise terminology, I think it would be
> better to say "name binding", rather than "naming and binding". When
> you talk of naming something (or someone!), you generally mean that
> it's possible to go from the thing to the (canonical) name. With
> people, for instance, you can walk up to someone and say "Hi! What's
> your name?", but with Python objects, you fundamentally can't.

"Naming and binding" is the title from the Python language reference
https://docs.python.org/3/reference/executionmodel.html#naming-and-binding

Otherwise, I agree with you ("name" is better here).

>>> Everything you wrote here has the same issue: The "objects" you are
>>> talking about do not physically exist, but are simply the result of
>>> calling a method on the object. Therefore they do not *belong* on the
>>> diagram, and the diagram not showing them does not mean the diagram is
>>> not complete.
>>
>> "do not physically exist" does not make sense. Objects are *never*
>> destroyed explicitly in Python (you can only make them
>> *unreachable*). You can disable garbage collection completely and it is
>> still will be Python. Immutable objects can be considered immortal e.g.:
>>
>>  (1+1) -- question: does the object that represents int(2) exist before
>> the expression is evaluated?
>>
>> The correct answer: it does not matter: int(2) can be created on the
>> fly, a cached int(2) can be reused by a specific implementation --
>> Python doesn't care.
>>
>> I don't see why the model that can't describe range(1) in Python 3
>> pretends to be complete.
>
> "Physically" isn't really the right word for it, given that objects in
> Python code aren't physical and therefore don't *ever* "physically
> exist". My bad there.
>
> But the objects completely do not exist. Yes, with integers you can't
> tell... but what about here?
>
> dependencies = collections.defaultdict(list)
> for fn in files:
> for dep in gather_deps(fn):
> dependencies[dep].append(fn)
>
> Until the moment when dependencies[dep] is requested for some new
> value of dep, the list *does not exist*. It is constructed anew.
> Obviously the end result of this is a dict of lists, and since those
> lists aren't accessible from anywhere else, it makes fine sense to
> talk about those lists as being "contained within" the (default)dict;
> but they clearly didn't exist until they were poked at. They're
> Heisenburg's Lists, I guess...

lists are _mutable_ in Python.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Terminology: "reference" versus "pointer"

[Akira Li]

Steven D'Aprano  writes:

> On Mon, 14 Sep 2015 09:17 am, Akira Li wrote:
>
>> I don't see why the model that can't describe range(1) in Python 3
>> pretends to be complete.
>
>
> Please explain.
>
> range(1) returns a range instance. What is hard about that?

Look at the last example:
http://thread.gmane.org/gmane.comp.python.general/782626/focus=782704

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Terminology: "reference" versus "pointer"

[Akira Li]

Chris Angelico  writes:

> On Mon, Sep 14, 2015 at 11:22 AM, Akira Li <4kir4...@gmail.com> wrote:
>> Steven D'Aprano  writes:
>>
>>> On Mon, 14 Sep 2015 09:17 am, Akira Li wrote:
>>>
>>>> I don't see why the model that can't describe range(1) in Python 3
>>>> pretends to be complete.
>>>
>>>
>>> Please explain.
>>>
>>> range(1) returns a range instance. What is hard about that?
>>
>> Look at the last example:
>> http://thread.gmane.org/gmane.comp.python.general/782626/focus=782704
>
> Still not sure what the problem is. As per Python's object model, the
> lists contain references to range objects. a contains two references
> to the same range object, b contains references to each of two
> distinct range objects. What of it?

For me, there is no problem. "parcel tags" [1], "box and arrows"[2] are
all the same (note: "labelled box"[3] that may contain an object is a
different model).

The talk about being "complete" is caused by the following quote from
Random832 message [4]:

  If the "parcel tags" model can't show it, then the "parcel tag" model
  clearly is not a correct and complete depiction of how Python actually
  works.

The purpose of the examples in [5] is to demonstate that neither models
are "complete" i.e., there is (obviously) behavior of a Python program
that they can't describe.

[1]
http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html#python-has-names
[2]
http://www.pythontutor.com/visualize.html#code=a0+%3D+a1+%3D+%5B1,+2%5D%0Ab0+%3D+%5B1,+2%5D%0Ab1+%3D+%5B1,+2%5D%0Aa+%3D+%5Ba0,+a1%5D%0Ab+%3D+%5Bb0,+b1%5D%0Adel+a0,+a1,+b0,+b1%0A&mode=display&origin=opt-frontend.js&cumulative=false&heapPrimitives=false&textReferences=false&py=3&rawInputLstJSON=%5B%5D&curInstr=6
[3]
http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html#other-languages-have-variables
[4]
http://thread.gmane.org/gmane.comp.python.general/782626/focus=782645
[5]
http://thread.gmane.org/gmane.comp.python.general/782626/focus=782704

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Terminology: "reference" versus "pointer"

[Akira Li]

Steven D'Aprano  writes:

> On Mon, 14 Sep 2015 11:22 am, Akira Li wrote:
>> Look at the last example:
>> http://thread.gmane.org/gmane.comp.python.general/782626/focus=782704
>
>
> I'm afraid that page is broken in my browser. Can you not summarise, or link
> to the specific message? I may be able to use another browser in a day or
> two, but hopefully the discussion will have moved on by then.

https://mail.python.org/pipermail/python-list/2015-September/696631.html

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Terminology: "reference" versus "pointer"

[Akira Li]

Random832  writes:
...
> Why can't it describe range(1)? A range object in my model would include
> the start, stop, and step; _not_ the contents of what you would get by
> iterating over it; since that's not part of the physical structure of
> the object, but the consequences of calling methods on it.

start, stop, step attributes (corresponding Python ints) may not exist
("the objects we've talking about have never been created") until you
request them explicitly.

I've mentioned it in another message but to be clear, I consider "parcel
tags" [1] and "box and arrows" [2] (boxes are always empty, they only
point to objects) models to be the same and different from "labelled
box" [3] model (boxes contain objects).

[1]
http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html#python-has-names
[2]
http://www.pythontutor.com/visualize.html#code=a0+%3D+a1+%3D+%5B1,+2%5D%0Ab0+%3D+%5B1,+2%5D%0Ab1+%3D+%5B1,+2%5D%0Aa+%3D+%5Ba0,+a1%5D%0Ab+%3D+%5Bb0,+b1%5D%0Adel+a0,+a1,+b0,+b1%0A&mode=display&origin=opt-frontend.js&cumulative=false&heapPrimitives=false&textReferences=false&py=3&rawInputLstJSON=%5B%5D&curInstr=6
[3]
http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html#other-languages-have-variables
 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Terminology: "reference" versus "pointer"

[Akira Li]

Steven D'Aprano  writes:

> On Mon, 14 Sep 2015 01:23 pm, Akira Li wrote:
>
>> Steven D'Aprano  writes:
>> 
>>> On Mon, 14 Sep 2015 11:22 am, Akira Li wrote:
>>>> Look at the last example:
>>>> http://thread.gmane.org/gmane.comp.python.general/782626/focus=782704
>>>
>>>
>>> I'm afraid that page is broken in my browser. Can you not summarise, or
>>> link to the specific message? I may be able to use another browser in a
>>> day or two, but hopefully the discussion will have moved on by then.
>> 
>> https://mail.python.org/pipermail/python-list/2015-September/696631.html
>
> Thanks. You mean this example?
>
>   lst = [range(1, 3) for _ in range(3)]
>   a = [lst[0], lst[0]]
>   b = [lst[1], lst[2]]
>
>
> I don't see what's difficult about this example. Here's a simple ASCII
> drawing:
>
>
> lst > [ range-object-1 , range-object-2 , range-object-3 ]
>
> a --> [ range-object-1 , range-object-1 ]
>
> b --> [ range-object-2 , range-object-3 ]
>

I should have mentioned that lst plays the role of range-object-X in
your diagram i.e., only *a*, *b* names actualy exits (I should add `del
lst` to the example) -- as the original example requires.


> Trying to draw an arrow diagram using text is not my idea of a good time,
> but I'll give it a go. Requires a monospaced font and an email client that
> won't reflow the text:
>
>   +-+--+
>   | |  | <--- a
>   +--|--+---|--+
>  |  |
>  |  |
>  V  |
>   +-+ <-+ ++
>   |range| <---|-   |< lst 
>   +-+ ++
>   +---|-   |
>   +-+ |   ++
>   +-> |range| <---++--|-   |
>   |   +-+  |  ++
>   ||
>   |   +-+  |
>   |   |range| <+
>   |   +-+
>   |  ^
> +-|-+|
> |   ||
> +---+|
> |  -|+
> +---+
>   ^
>   |
>   +--- b
>
>
> Out of the two, I know which one I prefer.

I don't know what it means. If you mean "box and arrows" is superior
somehow then I don't see any difference from the "parcel tags" model
here.

My point is that neither models are detailed enough to describe
meaningfully the behavior of Python code in the general case.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Terminology: "reference" versus "pointer"

[Akira Li]

Random832  writes:

> On Mon, Sep 14, 2015, at 10:48, Akira Li wrote:
>> start, stop, step attributes (corresponding Python ints) may not exist
>> ("the objects we've talking about have never been created") until you
>> request them explicitly.
>
> That's not true in CPython. In fact, the range object in python contains
> *four* reference boxes - one more for length.

Even if it true in CPython. Specific implementations are irrelevant for
the discussions. Python -- the language -- does not mandate any
reference boxes (and given how you interpret the term "box" -- boxes are
not used anywhere).

>> I've mentioned it in another message but to be clear, I consider "parcel
>> tags" [1] and "box and arrows" [2] (boxes are always empty, they only
>> point to objects) models to be the same and different from "labelled
>> box" [3] model (boxes contain objects).
>
> See, I consider the box and arrow to be the same as the labeled box
> model - only the object the boxes contain is an arrow. Except for
> special kinds of boxes implemented in some other language, such as the
> elements of an array from the array module

   [box + arrow pointing to] + object == parcel tag + object

I could draw a picture but it won't be pretty.

"labelled box" model that assumes that objects are inside boxes is
clearly wrong -- it fails if you have two names that refer to the same
object in Python (you can't put the same object into different
boxes).

If a box contains nothing then there is no need for the box. If you
think the box contains "something" then find me the corresponding term
in Python language reference. I don't understand what is "arrow" that
the box might contain exactly. For example I can find what "name",
"object" mean in Python.

> The problem with "parcel tags" is that it represents namespaces - or one
> particular namespace, I've never seen any version of it that even
> clearly talks about locals, let alone different modules - differently
> from other kinds of objects.

I don't understand what are you trying to say here. If you have a
specific example when the "parcel tags" [1] model predicts a _wrong_
behavior; please do provide it. If your point is that "parcel tag" does
not describe in full detail all aspect of the behavior of an arbitrary
Python code then I agree with you (though none of the discussed models
can or should do it).

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Terminology: "reference" versus "pointer"

[Akira Li]

Random832  writes:

> On Mon, Sep 14, 2015, at 13:45, Akira Li wrote:
>>[box + arrow pointing to] + object == parcel tag + object
>
> The problem is that if there are multiple namespaces, or if you've also
> got to include references from within other objects such as list, then
> you've got to write all that somehow on the parcel tag, instead of just
> having different places the arrows can start from.
>
> The box and arrow model easily extends to this, and the fact that no-one
> can make up their mind on what to *call* the thing the arrows represent
> doesn't mean they don't exist.

box has arrow, parcel tag has string/thread.
We could attach an arrow instead of a string to the parcel tag.

The box does not contain anything inside (we can attach the arrow
outside the box -- nothing changes).

If the box does not contain anything then we could use a parcel tag
instead.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Terminology: "reference" versus "pointer"

[Akira Li]

Ned Batchelder  writes:
...
> What do you feel is missing from Steven's diagram?

I don't feel anything missing because I don't expect the model to be
more detailed.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Terminology: "reference" versus "pointer"

[Akira Li]

Ned Batchelder  writes:

> On Monday, September 14, 2015 at 3:32:46 PM UTC-4, Akira Li wrote:
>> Ned Batchelder  writes:
>> ...
>> > What do you feel is missing from Steven's diagram?
>> 
>> I don't feel anything missing because I don't expect the model to be
>> more detailed.
>
> Akira, you said, "neither models are detailed enough to describe
> meaningfully the behavior of Python code in the general case."   
>
> I'm wondering what aspect of the code's behavior isn't captured
> in this model?  I know you didn't expect it to be complete, but I'm
> not sure what you think is missing.
>

I don't understand what you are talking about.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Shutting down a cross-platform multithreaded app

[Akira Li]

"James Harris"  writes:

...
> Needless to say, on a test Windows machine AF_UNIX is not present. The
> only cross-platform option, therefore, seems to be to use each
> subthread's select()s to monitor two AF_INET sockets: the one to the
> client and a control one from the master thread. I would seem to need
> IP socket pairs between the master thread and the subthreads. If the
> master thead receives a shutdown signal it will send a shutdown
> command to each subthread.

There is socket.socketpair() on Windows too (since Python 3.5)

  https://docs.python.org/3/library/socket.html#socket.socketpair
  http://bugs.python.org/issue18643

Note: you could use select() to handle signals in the main thread too
(even on Windows since Python 3.5) if you use signal.set_wakeup_fd()

  https://docs.python.org/3/library/signal.html#signal.set_wakeup_fd

It is known as a self-pipe trick

  http://www.sitepoint.com/the-self-pipe-trick-explained/

Look at *asyncio* source code, to see how to get a portable
implementation for various issues with signals. Some issues might still
be opened e.g., Ctrl+C behavior

  http://bugs.python.org/issue24080

Here's how to combine SIGCHLD signal handling with tkinter's event
loop

  http://stackoverflow.com/questions/30087506/event-driven-system-call-in-python

SIGCHLD, createfilehandler() are not portable but the code demonstrates
possible set_wakeup_fd() issues and their solutions (O_NONBLOCK, dummy
signal handler, SA_RESTART, signal coalescing).

On threads and signals in CPython

  http://bugs.python.org/issue5315#msg102829

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lightwight socket IO wrapper

[Akira Li]

"James Harris"  writes:

> I guess there have been many attempts to make socket IO easier to
> handle and a good number of those have been in Python.
>
> The trouble with trying to improve something which is already well
> designed (and conciously left as is) is that the so-called improvement
> can become much more complex and overly elaborate. That can apply to
> the initial idea, for sure, but when writing helper or convenience
> functions perhaps it applies more to the temptation to keep adding
> just a little bit extra. The end result can be overly elaborate such
> as a framework which is fine where such is needed but is overkill for
> simpler requirements.
>
> Do you guys have any recommendations of some *lightweight* additions
> to Python socket IO before I write any more of my own? Something built
> in to Python would be much preferred over any modules which have to be
> added. I had in the back of my mind that there was a high-level
> socket-IO library - much as threading was added as a wrapper to the
> basic thread module - but I cannot find anything above socket. Is
> there any?

Does ØMQ qualify as lightweight?

> A current specific to illustrate where basic socket IO is limited: it
> normally provides no guarantees over how many bytes are transferred at
> a time (AFAICS that's true for both streams and datagrams) so the
> delimiting of messages/records needs to be handled by the sender and
> receiver. I do already handle some of this myself but I wondered if
> there was a prebuilt solution that I should be using instead - to save
> me adding just a little bit extra. ;-)

There are already convenience functions in stdlib such as
sock.sendall(), sock.sendfile(), socket.create_connection() in addition
to BSD Sockets API.

If you want to extend this list and have specific suggestions; see
  https://docs.python.org/devguide/stdlibchanges.html

Or just describe your current specific issue in more detail here.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lightwight socket IO wrapper

[Akira Li]

"James Harris"  writes:
...
> There are a few things and more crop up as time goes on. For example,
> over TCP it would be helpful to have a function to receive a specific
> number of bytes or one to read bytes until reaching a certain
> delimiter such as newline or zero or space etc. 

The answer is sock.makefile('rb') then `file.read(nbytes)` returns a
specific number of bytes.

`file.readline()` reads until newline (b'\n') There is Python Issue:
"Add support for reading records with arbitrary separators to the
standard IO stack"
  http://bugs.python.org/issue1152248
See also
  http://bugs.python.org/issue17083

Perhaps, it is easier to implement read_until(sep) that is best suited
for a particular case.

> Even better would be to be able to use the iteration protocol so you
> could just code next() and get the next such chunk of read in a for
> loop.

file is an iterator over lines i.e., next(file) works.

> When sending it would be good to just say to send a bunch of bytes but
> know that you will get told how many were sent (or didn't get sent) if
> it fails. Sock.sendall() doesn't do that.

sock.send() returns the number of bytes sent that may be less than given.
You could reimplement sock.sendall() to include the number of bytes
successfully sent in case of an error.

> I thought UDP would deliver (or drop) a whole datagram but cannot find
> anything in the Python documentaiton to guarantee that. In fact
> documentation for the send() call says that apps are responsible for
> checking that all data has been sent. They may mean that to apply to
> stream protocols only but it doesn't state that. (Of course, UDP
> datagrams are limited in size so the call may validly indicate
> incomplete transmission even when the first part of a big message is
> sent successfully.)
>
> Receiving no bytes is taken as indicating the end of the
> communication. That's OK for TCP but not for UDP so there should be a
> way to distinguish between the end of data and receiving an empty
> datagram.

There is no end of communication in UDP and therefore there is no end of
data. If you've got a zero bytes in return then it means that you've
received a zero length datagram.

sock.recvfrom() is a thin wrapper around the corresponding C
function. You could read any docs you like about UDP sockets.
  
http://stackoverflow.com/questions/5307031/how-to-detect-receipt-of-a-0-length-udp-datagram

> The recv calls require a buffer size to be supplied which is a
> technical detail. A Python wrapper could save the programmer dealing
> with that.

It is not just a buffer size. It is the maximum amount of data to be
received at once i.e., sock.recv() may return less but never more.
You could use makefile() and read() if recv() is too low-level.

> Reminder to self: encoding issues.
>
> None of the above is difficult to write and I have written the bits I
> need myself but, basically, there are things that would make socket IO
> easier and yet still compatible with more long-winded code. So I
> wondered if there were already some Python modules which were more
> convenient than what I found in the documentation.
>
> James

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A little test for you Guys😜

[Akira Li]

Python_Teacher via Python-list  writes:

...
> Let's define the function plural :
>
> def plural(words):
> plurals = []
> for word in words:
>plurals.append(word + 's')
> return plurals
>
> for word in plural(['cabagge','owl','toy']):
> print word

plural() should accept a single word. To handle list of words, call
map(plural, words)

...
> def str2print(f):
> def str2print_wrap(*args, **kwargs):
> """wrapper"""
> s = f(*args, **kwargs)
> print s
>return str2print_wrap
>
> def hello(s):
> """ Return "Hello $s" """
> return "%s %s" % ("Hello", s)

Use functools.wraps() to preserve the function info for introspection:

  import functools
  
  def prints_result(function):
  @functools.wraps(function)
  def wrapper(*args, **kwargs):
  result = function(*args, **kwargs)
  print(result)
  return result #XXX return
  return wrapper
  
  @prints_result
  def hello(...):
  pass

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Modify environment variable for subprocess

[Akira Li]

loial  writes:

> I need to modify the LIBPATH environment variable when running a
> process via subprocess, but otherwise retain the existing environment.
>
> Whats the best way to do that?

Pass env=dict(os.environ, LIBPATH=value) parameter:

  import os
  import subprocess 
  
  subprocess.check_call('echo $LIBPATH', shell=True,
env=dict(os.environ, LIBPATH='/some/path'))


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Idiosyncratic python

[Akira Li]

Mark Lawrence  writes:

> On 24/09/2015 07:02, Steven D'Aprano wrote:
>> I was looking at an in-house code base today, and the author seems to have a
>> rather idiosyncratic approach to Python. For example:
>>
>> for k, v in mydict.items():
>>  del(k)
>>  ...
>>
>> instead of the more obvious
>>
>> for v in mydict.values():
>>  ...
>>
>> What are your favorite not-wrong-just-weird Python moments?
>>
>
> My favourite was from a guy I worked with years ago.  In C but I'm
> sure you'll enjoy it.  In all functions, something like:-
>
> int flag = 0;
> if flag {
> printf("\nthe string");
> }
> else{
> printf("the string");
> flag = 1;
> }
>
> At least I think I've got it correct, too lazy to check, sorry :)

It looks like a sys.stdout.softspace hack in Python 2:

  print line, # comma!

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: threading bug in strptime

[Akira Li]

Larry Martell  writes:

> We have been trying to figure out an intermittent problem where a
> thread would fail with this:
>
> AttributeError: 'module' object has no attribute '_strptime'
>
> Even though we were importing datetime. After much banging our heads
> against the wall, we found this:
>
> http://code-trick.com/python-bug-attribute-error-_strptime/
>
> The workaround suggested there, to call strptime before starting your
> threads, seems to have fixed the issue.
>
> I thought I'd mention it here in case anyone else is facing this.

I can reproduce it in Python 2 (but not in Python 3) even with
*threading* module:

  #!/usr/bin/env python
  import threading
  import time

  for _ in range(10):
  threading.Thread(target=time.strptime,
   args=("2013-06-02", "%Y-%m-%d")).start()

Don't use *thread* directly (it is even renamed to *_thread* in Python
3, to discourage an unintended usage), use *threading* module instead.

In Python 3.3+, PyImport_ImportModuleNoBlock()  is deprecated
  https://bugs.python.org/issue9260

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.2 has some deadly infection

[Akira Li]

Steven D'Aprano  writes:

> On Tue, 03 Jun 2014 15:18:19 +0100, Robin Becker wrote:
>
>> Isn't it a bit old fashioned to think everything is connected to a
>> console?
>
> The whole concept of stdin and stdout is based on the idea of having a 
> console to read from and write to. Otherwise, what would be the point? 
> Classic Mac (pre OS X) had no command line interface nothing, and nothing 
> even remotely like stdin and stdout. But once you have a console, stdin, 
> stdout, and stderr become useful. And once you have them, then you can 
> extend the concept using redirection and pipes. But fundamentally, stdin 
> and stdout are about consoles.
>
We can consider "pipes" abstraction to be fundumental. Decades of usage
prove a pipeline of processes usefulness e.g.,

  tr -cs A-Za-z '\n' |
  tr A-Z a-z |
  sort |
  uniq -c |
  sort -rn |
  sed ${1}q

See http://www.leancrew.com/all-this/2011/12/more-shell-less-egg/

Whether or not a pipe is connected to a tty is a small
detail. stdin/stdout is about pipes, not consoles.


--
akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.2 has some deadly infection

[Akira Li]

Marko Rauhamaa  writes:

> Steven D'Aprano :
>
>> Nevertheless, there are important abstractions that are written on top
>> of the bytes layer, and in the Unix and Linux world, the most
>> important abstraction is *text*. In the Unix world, text formats and
>> text processing is much more common in user-space apps than binary
>> processing.
>
> That linux text is not the same thing as Python's text. Conceptually,
> Python text is a sequence of 32-bit integers. Linux text is a sequence
> of 8-bit integers.

_Unicode string in Python is a sequence of Unicode codepoints_. It is
correct that 32-bit integer is enough to represent any Unicode
codepoint: \u...\U0010 

It says *nothing* about how Unicode strings are represented
*internally* in Python. It may vary from version to version, build
options and even may depend on the content of a string at runtime.

In the past, "narrow builds" might break the abstraction in some cases
that is why Linux distributions used wide python builds.


_Unicode codepoint is  not a Python concept_. There is Unicode
standard http://unicode.org Though intead of following the
self-referential defenitions web, I find it easier to learn from
examples such as http://codepoints.net/U+0041 (A) or
http://codepoints.net/U+1F3A7 (🎧)

_There is no such thing as 8-bit text_
http://www.joelonsoftware.com/articles/Unicode.html

If you insert a space after each byte (8-bit) in the input text then you
may get garbage i.e., you can't assume that a character is a byte:

  $ echo "Hyvää yötä" | perl -pe's/.\K/ /g'
  H y v a � � � �   y � � t � �

In general, you can't assume that a character is a Unicode codepoint:

  $ echo "Hyvää yötä" | perl -C -pe's/.\K/ /g'
  H y v a ̈ ä   y ö t ä

The eXtended grapheme clusters (user-perceived characters) may be useful
in this case:

  $ echo "Hyvää yötä" | perl -C -pe's/\X\K/ /g'
  H y v ä ä   y ö t ä

\X pattern is supported by `regex` module in Python i.e., you can't even
iterate over characters (as they are seen by a user) in Python using
only stdlib. \w+ pattern is also broken for Unicode text
http://bugs.python.org/issue1693050 (it is fixed in the `regex` module)
i.e., you can't select a word in Unicode text using only stdlib.

\X along is not enough in some cases e.g., "“ch” may be considered a
grapheme cluster in Slovak, for processes such as collation" [1]
(sorting order). `PyICU` module might be useful here.

Knowing about Unicode normalization forms (NFC, NFKD, etc)
http://unicode.org/reports/tr15/ Unicode
text segmentation [1] and Unicode collation algorithm
http://www.unicode.org/reports/tr10/ concepts is also 
useful; if you want to work with text. 

[1]: http://www.unicode.org/reports/tr29/


--
akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Forking PyPI package

[Akira Li]

Wiktor  writes:

> On Fri, 6 Jun 2014 03:37:56 +1000, Chris Angelico wrote:
>
>> On Fri, Jun 6, 2014 at 2:56 AM, Wiktor  wrote:
>>>   I guess, I'll try to do what Chris proposed. Forget about this
>>> implementation and write python script from the scratch looking only at the
>>> original JavaScript version. :-/
>> 
>> Sadly, that may be your only safe option.
>> 
>> Let this be a lesson to all: If you want something to be free
>> software, make it very clear, because "it looks like he meant that to
>> be open source" just isn't enough :(
>
>   Lesson taken. ;-)
>   Interesting thing is, that for another 4 people, lack of license in this
> script wasn't problem to publish its modified version. I've just searched
> phrase "pwdhash" on GitHub, to simply check if someone else hadn't port
> this script to Python3 earlier, or maybe ported it (with proper license) to
> Python2 so I would have better start. And I've found practically the same
> script here: https://github.com/ali01/pwdhash.py, forked then 3 times.
>   Of course I'm not going to act now "Oh, they could do that without
> consequences, so why should I bother?" - no, I'm going to do this right (as
> a good start in OS community) - but it feels awkward now. ;-)

Have you tried to open an issue about clarifying the license terms [1] ?

[1]: https://github.com/abbot/pwdhash/issues/new

Or better yet, submit a pull request that specifies the license to the
standard you need?

I've dealt with the author in the past. I see no reason, he would refuse
to accept PR if license=BSD in setup.py is not enough.


--
akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: asyncio: wrapping a subprocess in a server

[Akira Li]

David Farler  writes:

> Hi all,
>
> I'm trying to vend some kind of server that spawns a client-unique
> subprocess when a client connects. The server acts as a middleman,
> receiving structure messages from the client, sending input to the
> subprocess, and packaging up subprocess data before sending back to
> the client. The connection is long-lived and the client can continue
> to send "requests". If the subprocess crashes or dies, the server
> should be able to revive it transparently.
>
> I know how to separately create server and subprocess protocols and
> having the event loop run either until completion, but what is the
> best way to chain two protocols together?
>
> || Client || <~~ TCP ~~> || Server <--> Subprocess ||

Here's an example that uses Twisted http://stackoverflow.com/a/11729467

  javascript client <--> websocket <--> twisted server <--> subprocess

Do you want the same but using asyncio module?


--
Akira



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to get Timezone from latitude/longitude ?

[Akira Li]

Larry Martell  writes:

> On Wed, Jun 25, 2014 at 8:53 AM,   wrote:
>> Hi,
>>
>> I'm looking for a python-library which can help me to get Timezone
>> and Timezone-offset(UTC) from latitude/longitude.
>>
>> I'm not able to find an easy way to do it.
>>
>> Thanks in advance.
>
> It took me 30 seconds on google to find this:
>
> https://github.com/pegler/pytzwhere

Note: tzwhere may consume 750GB memory doing nothing
https://github.com/pegler/pytzwhere/pull/10

You could generate a postgis timezone database from efele.net/tz map or
use an online service instead http://stackoverflow.com/a/16519004


--
akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: OOP with MyTime

[Akira Li]

kjaku...@gmail.com writes:

> I'm trying to write a boolean function that takes two Mytime objects, t1 and 
> t2 as arguments, and returns True if the object falls inbetween the two times.
>
> This is a question from the How to Think Like a Computer Scientist book, and 
> I need help.
>
> What I've gotten so far:
>
> class MyTime:
> def __init__(self, hrs=0, mins=0, secs=0):
> self.hours = hrs
> self.minutes = mins
> self.seconds = secs
> def between(t1, t2):
> if float(t1 <= t3) and float(t3 < t2):
> return True
> else:
> return False
>
> I just don't understand how to make a function that uses MyTime objects into 
> the boolean function? Any help would be great.

A method accepts the object itself as the first parameter:

  import functools

  @functools.total_ordering
  class MyTime:
...
def inbetween(self, t1, t2):
"""Return whether `self` is in [t1, t2) right-open range."""
return t1 <= self < t2 # `self` is the object itself
def _astuple(self):
return (self.hours, self.minutes, self.seconds)
def __lt__(self, other):
"""Return whether `self < other`."""
return self._astuple() < other._astuple()
def __eq__(self, other):
"""Return whether `self == other`."""
return self._astuple() == other._astuple()

See
https://docs.python.org/3/library/functools.html#functools.total_ordering

Example:

  >>> MyTime(1).inbetween(MyTime(0), MyTime(2))
  True

It is equivalent to:

  >>> MyTime(0) <= MyTime(1) < MyTime(2)
  True


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Unicode, stdout, and stderr

[Akira Li]

"Frank Millman"  writes:

> "Steven D'Aprano"  wrote in message 
> news:53ce0b96$0$29897$c3e8da3$54964...@news.astraweb.com...
>> On Tue, 22 Jul 2014 08:18:08 +0200, Frank Millman wrote:
>>
>>> This is not important, but I would appreciate it if someone could
>>> explain the following, run from cmd.exe on Windows Server 2003 -
>>>
>>> C:\>python
>>> Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:38:22) [MSC v.1600 32
>>> bit (In
>>> tel)] on win32
>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>> x = '\u2119'
>>>>>> x  # this uses stderr
>>> '\u2119'
>>

>>> It seems that there is a difference between writing to stdout and
>>> writing to stderr.
>>
>> I would be surprised if that were the case, but I don't have a Windows
>> box to test it. Try this:
>>
>>
>> import sys
>> print(x, file=sys.stderr)  # I expect this will fail
>
> It does not fail.
>> print(repr(x), file=sys.stdout)  # I expect this will succeed
>
> It fails.

Check sys.stderr.errors attribute. Try

>>> import sys
>>> x = '\u2119'
>>> x.encode(sys.stderr.encoding, sys.stderr.errors) # succeed
>>> x.encode(sys.stdout.encoding, sys.stdout.errors) # fail

sys.stderr uses 'backslashreplace' error handler that is why you see
\u2119 instead of ℙ.

On Linux with utf-8 locale:

  >>> print('\u2119')
  ℙ
  >>> print(repr('\u2119'))
  'ℙ'
  >>> print(ascii('\u2119'))
  '\u2119'
  >>> '\u2119'
  'ℙ'
  >>> repr('\u2119')
  "'ℙ'"
  >>> ascii('\u2119')
  "'\\u2119'"

On Windows, try https://pypi.python.org/pypi/win_unicode_console

  C:\> pip install win-unicode-console
  C:\> py -i -m run

It is alpha but your feedback may improve it
https://github.com/Drekin/win-unicode-console/issues 

If you could also use a GUI console e.g.:

  C:\> py -3 -m idlelib

Or http://ipython.org/notebook.html

There are many other IDEs for Python e.g.,

http://stackoverflow.com/q/81584/what-ide-to-use-for-python


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to index an array with even steps?

[Akira Li]

fl  writes:
> In Python, ':' is used to indicate range (while in Matlab I know it can be 
> used
> to control steps). How to index an array with 0, 5, 10, 15...995?

Just use slicing:

  >>> L = range(1000) # your array goes here
  >>> L[::5] 
  [0, 5, 10, 15, ..., 995] # Python 2
  range(0, 1000, 5)# Python 3


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Getting a list of all modules

[Akira Li]

Steven D'Aprano  writes:

> I'm looking for a programmatic way to get a list of all Python modules 
> and packages. Not just those already imported, but all those which 
> *could* be imported.
...
> Is this problem already solved? Can anyone make any suggestions?

Look at how `help('modules')` is implemented. Though it crashes on my
system.

See also,

How can I get a list of locally installed Python modules? [1]
python - get available modules [2]

[1]
http://stackoverflow.com/questions/739993/how-can-i-get-a-list-of-locally-installed-python-modules
[2]
http://stackoverflow.com/questions/3952513/python-get-available-modules


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What is best way to learn Python for advanced developer?

[Akira Li]

guirec.cor...@gmail.com writes:

> I am a Ruby developer and I want to program in Python. I know how to
> do simple things like create classes, methods, variables and all the
> basics. I want to know more. I want to know what is the Python
> philosophy, how to test, how to create maintenable software, etc.
>
> I'm looking for online courses and any ressources I can have on the subject.
>
> Can you help me?
To answer the question in the Subject:

Idiomatic Python (basic DO's & DON'Ts of the language):
http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html

Tutorial (get acquainted with the language and stdlib):
https://docs.python.org/3/tutorial/index.html

Batteries (get acquainted with ecosystem):
https://github.com/vinta/awesome-python

Do 100+ projects in Python to learn the language (or something from your
domain):
http://www.reddit.com/r/learnpython/comments/1huuqk/im_doing_100_projects_in_python_to_learn_the


Personal: learn both Python 2 & 3.


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bug with help (was Re: Getting a list of all modules)

[Akira Li]

Mark Lawrence  writes:

> On 31/07/2014 19:55, Akira Li wrote:
>> Steven D'Aprano  writes:
>>
>>> I'm looking for a programmatic way to get a list of all Python modules
>>> and packages. Not just those already imported, but all those which
>>> *could* be imported.
>> ...
>>> Is this problem already solved? Can anyone make any suggestions?
>>
>> Look at how `help('modules')` is implemented. Though it crashes on my
>> system.
>>
>
> Have you reported this at bugs.python.org or is there already an issue
> for the problem that you see?

It is this issue for python2.7:

https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/896836

python3 doesn't crash with currently installed packages.


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: converting ISO8601 date and time string representations to datetime

[Akira Li]

Wolfgang Maier  writes:

> On 08/01/2014 01:30 AM, Roy Smith wrote:
>> In article ,
>>   Albert-Jan Roskam  wrote:
>>
>>>> In article ,
>>>> Wolfgang Maier  wrote:
>>>>
>>>>> Hi,
>>>>> I'm trying to convert ISO8601-compliant strings representing dates or
>>>>> dates and times into datetime.datetime objects.
>>>>
>>>> https://pypi.python.org/pypi/iso8601
>>>
>>> Yikes, what a regex. It must have been painstaking to get that right.
>>> https://bitbucket.org/micktwomey/pyiso8601/src/2bd28b5d6cd2481674a8b0c54a8bba6
>>> 4ab775f81/iso8601/iso8601.py?at=default
>>
>> It is a thing of beauty.
>>
>
> No wonder I found it hard to write something that seemed bulletproof !

It seems it supports only some custom subset of ISO 8601. There is rfc
3339 [1] that describes a profile of the ISO 8601 standard.

rfc 3339 combines human readability with the simplicity of machine parsing.

[1] http://tools.ietf.org/html/rfc3339


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Correct type for a simple "bag of attributes" namespace object

[Akira Li]

Albert-Jan Roskam  writes:

> I find the following obscure (to me at least) use of type() useful
> exactly for this "bag of attributes" use case:
>>>> employee = type("Employee", (object,), {})
>>>> employee.name = "John Doe"
>>>> employee.position = "Python programmer"

You  could write it as:

  class Employee:
  name = "Johh Doe"
  position = "Python programmer"

It also makes it clear that `type()` returns a *class Employee*, not its
instance (Employee()) and therefore name, position are class attributes.


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: cmd.exe on WIndows - problem with displaying some Unicode characters

[Akira Li]

Wiktor  writes:

> On Mon, 04 Aug 2014 15:17:04 -0400, Terry Reedy wrote:
>
>>>I'm taking next step, so I tried to draw nice frame around menu (that's
>>> why I posted yesterday).
>>
>> Is there no working codepage with ascii text and the line chars? I
>> suppose I am not surprised if not.
>
>   With single line (└┘┌┐─│├┤┬┴┼) and double line (╣║╗╝╚╔╩╦╠═╬) - many
> codepages, CP852 for sure.
>
>   With corners/crosses where single and double lines meet (╖╘╡╢╕╜╛╞╟
> ╧╨╤╥╙╘╒╓╫╪) - I know only one: CP437.
>
>   But I can't have both - Polish letters and all those line chars, so I
> can't do this fancy frame from first post with Polish strings inside. There
> will be simpler version instead.

Unicode has line drawing characters [1]. win_unicode_console [2] allows
to print Unicode in cmd.exe. win_unicode_console and colorama will
probably conflict. You could look at the source to see how hard to
combine both functionalities.

[1] http://en.wikipedia.org/wiki/Box-drawing_character
[2] https://pypi.python.org/pypi/win_unicode_console

btw, blessings [3] provides an easy-to-use interface if you need to add
colors and move text in a terminal. It claims that it also supports
colors on Windows if used with colorama.

[3] https://pypi.python.org/pypi/blessings/


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Test for an empty directory that could be very large if it is not empty?

[Akira Li]

Virgil Stokes  writes:

> Suppose I have a directory C:/Test that is either empty or contains
> more than 200 files, all with the same extension (e.g. *.txt). How
> can I determine if the directory is empty WITHOUT the generation of a
> list of the file names in it (e.g. using os.listdir('C:/Test')) when
> it is not empty?

  def is_empty_dir(dirpath):
  return next(scandir(dirpath), None) is None

 https://github.com/benhoyt/scandir


 --
 Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: how to change the time string into number?

[Akira Li]

luofeiyu  writes:

> s="Aug"
>
> how can i change it into 8 with some python time module?

  months = (None, # dummy, to start month indices from 1
  "Jan","Feb","Mar","Apr","May","Jun",
  "Jul","Aug","Sep","Oct","Nov","Dec"
  )
  month_number = months.index(month_abbr) # month_abbr == "Aug"

Note:
- time.strptime(month_abbr, "%b").tm_mon may fail in non-English locale
- list(calendar.month_abbr).index(month_abbr) is also locale-specific


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: pexpect - logging input AND output

[Akira Li]

sj.constant...@gmail.com writes:

> i have a script running a few commands on a network device. i can't
> seem to figure out how to log both the input and output of what the
> pexpect script initiates and responds to.
>
> child = pexpect.spawn ('telnet '+ ip)
> child.expect ('.*:*')
> child.sendline (user)
> child.expect ('.*:*')
> child.sendline (password)
> child.expect(['.*#*', '.*>*'])
> child.sendline ('enable')
> child.expect (['Password:', '.*#*'])
> child.sendline (password)
> child.expect ('.*#*')
> child.sendline ('conf t')
> child.expect ('.*#*')
> child.sendline ('line vty 0 4')
>
> i have tried both these logging commands:
>
> child.logfile = open('log.txt', 'w')
> child.logfile=sys.stdout
>
> all i get is the input i send with expect/sendline combinations, i
> don't get any of what the device sends, only what i send the device:
>
> user
> password
> enable
> password
> conf t
> line vty 0 4
>
> any ideas of what is the correct way to go about this? just can't get the 
> output!

To be clear, expect() doesn't send anything to the device. expect()
matches as little as possible therefore '.*:*' matches *nothing*.

If it is Python 3 then use pexpect.spawnu(). Otherwise, assigning to
child.logfile should work as is.

There is a telnetlib module in stdlib. x/84 python telnet server might
contain a client too.


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: how to copy emails into local directory?

[Akira Li]

luofeiyu  writes:

> import imaplib,email
> user=""
> password=""
> con=imaplib.IMAP4_SSL('imap.gmail.com')
> con.login(user,password)
> con.list()
> ('OK', [b'(\\HasNoChildren) "/" "INBOX"', b'(\\Noselect \\HasChildren)
> "/" "[Gma
> il]"', b'(\\HasNoChildren \\Junk) "/" "[Gmail]/&V4NXPpCuTvY-"',
> b'(\\HasNoChildr
> en \\Trash) "/" "[Gmail]/&XfJSIJZkkK5O9g-"', b'(\\HasNoChildren
> \\Flagged) "/" "
> [Gmail]/&XfJSoGYfaAc-"', b'(\\HasNoChildren \\Sent) "/"
> "[Gmail]/&XfJT0ZCuTvY-"'
> , b'(\\HasNoChildren \\All) "/" "[Gmail]/&YkBnCZCuTvY-"',
> b'(\\HasNoChildren \\D
> rafts) "/" "[Gmail]/&g0l6Pw-"', b'(\\HasNoChildren \\Important) "/"
> "[Gmail]/&kc
> 2JgQ-"'])
>
> Now, i want to  copy all the emails in the gmailbox of
> "[Gmail]/&kc2JgQ-"  (the important mailbox in my gmail"
> into local directory  "g:\emails",how can i do that in python code?

You could try an already-made application something like *gmvault*
instead of implementing the functionality from scratch on top of
imaplib. It is easy [1] to create a version that can perform the backup in
ideal conditions but it is much harder to take into account all nuances
to make it reliable.

[1]
http://stackoverflow.com/questions/348630/how-can-i-download-all-emails-with-attachments-from-gmail


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to look up historical time zones by date and location

[Akira Li]

luofeiyu  writes:

> http://www.thefreedictionary.com/time+zone
>
> time zone Any of the 24 divisions of the Earth's surface used to
> determine the local time for any given locality.
> Each zone is roughly 15° of longitude in width, with local variations
> for economic and political convenience.
> Local time is one hour ahead for each time zone as one travels east
> and one hour behind for each time zone as one travels west.
>
> Urumqi  's localtime is beijin time ,it is decided  by law .
> Urumqi  's timezone is east 6 ,it is decided by geography.
>
> There is only one localtime in all over the chian,beijin time,but
> there are 5  timezone time in china .
>

In programming, a different timezone defintion is usually used [1] (pytz
library provides access to the timezone database in Python). There are
much more than 24 timezones (e.g., look at pytz.common_timezones) and a
UTC offset used in a place may be unrelated to its longitude. A timezone
is more a political entity than a geographical.

[1] http://www.iana.org/time-zones

For example, here's a shapefile for China [2].  It shows the shape of
Asia/Chongqing, Asia/Harbin, Asia/Kashgar, Asia/Shanghai, Asia/Urumqi
timezones on a map.

[2] http://efele.net/maps/tz/china/

Different timezones may have the same UTC offset at some point. UTC
offset for the same timezone may be different at different times e.g.,
due to DST transitions.

There are multiple ways to get a timezone name from location [3].

[3]
http://stackoverflow.com/questions/16505501/get-timezone-from-city-in-python-django


--
Akira



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Efficiency, threading, and concurrent.futures

[Akira Li]

Rob Gaddi  writes:

> I've got a situation where I'll be asking an I/O bound process to do
> some work (querying an RS-232 device) while my main code is off
> running a sleep() bound process.  Everyone always talks about how
> expensive thread creation is, so I figured I'd test it out in an
> IPython notebook. 
>
> #
>
> import threading
> from concurent.futures import ThreadPoolExecutor as TPE
> from time import sleep
>
> def fn():
>   sleep(0.001)
>
> %%timeit -r 50 -n 1000
> thr = threading.Thread(target=fn)
> thr.start()
> thr.join()
> 1000 loops, best of 5: 1.24 ms per loop
>
> %%timeit -r 50 -n 1000 ex=TPE(1)
> fut=ex.submit(fn)
> fut.result()
> 1000 loops, best of 5: 1.26 ms per loop
>
> #
>
> Now, my understanding is that the ThreadPoolExecutor spawns all its
> threads at the outset, then stuffs requests into one queue and
> fishes results out of another, which should be substantially faster than
> having create new threads each time.  And yet those were pretty dead on
> even. Any idea what I'm seeing here?

To see any difference, you should submit more than one job per worker to
ThreadPoolExecutor and avoid waiting for the each result synchronously.

I don't know whether ThreadPoolExecutor starts all workers at once in
the current CPython implementation. The name max_workers suggests that
it may start them as needed.


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Media Conversion Using Python - Converting MP3 to Other Formats

[Akira Li]

Mark Lawrence  writes:

> On 25/08/2014 16:28, Parth Trivedi wrote:
>> Dear All,
>>
>> I need some help of yours. I want to convert audio in MP3 format to
>> other formats including uncompressed raw format, WAV etc. and I am using
>> python 2.7. Is there any built-in module I can use or any third party
>> modules available ?
>>
>> Please help me on this. I would be very grateful.
>>
>> Thanks and Regards,
>> Parth T.
>>
>
> There are a number of extremely useful third party pieces of software
> available to help with this.  They are called "search engines".  What
> have you tried with them, what problems did you have, what else do you
> need to know about them?
>
> If you'd have tried these rather obscure beasts, you might have found
> https://docs.python.org/2/py-modindex.html or
> https://pypi.python.org/pypi
>
> You can do the rest from there.

Curated lists [1] might be useful. If you don't know what you are searching
for then a search engine might return a wrong answer that you won't be
able to recognize as one (Catch-22).

[1] https://github.com/vinta/awesome-python#audio

Everybody needs to start somewhere.

Having said that, a search query: “python mp3” [2] returns useful
results for me.

[2] https://duckduckgo.com/?q=python+mp3


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Reading from sys.stdin reads the whole file in

[Akira Li]

Chris Angelico  writes:

> On Wed, Aug 27, 2014 at 4:37 PM, Steven D'Aprano  wrote:
>> On Wed, 27 Aug 2014 08:29:20 +0300, Marko Rauhamaa wrote:
>>
>>> Try flushing after each print.
>>
>> Doesn't help.
>
> It does, but insufficiently. If slurp.py is run under Py3, it works
> fine; or take Naoki's suggestion (although without the parens):
>
> import sys
> import time
>
> for line in iter(sys.stdin.readline, ''):
> print "Time of input:", time.ctime(), line
> sys.stdin.flush()
> sys.stdout.flush()
>
> Then it works.
>
> ChrisA

It looks like this bug http://bugs.python.org/issue3907

`python -u out.py | python -u slurp.py`  could be used to avoid .flush()
calls everywhere.

Or reassign `sys.stdin = io.open(sys.stdin.fileno(), 'r', 1)` inside the
script.

http://stackoverflow.com/questions/107705/python-output-buffering


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: subprocess module usage

[Akira Li]

Earl Lapus  writes:

> Hi,
>
> I made simple test program using the subprocess module (see attached:
> exec_cmd.py). I ran it passing variations of 'ls' command options.
>
> I encounter exceptions every time I use '-l' options. Example runs
> where exception occurs:
> # ./exec_cmd.py ls -al
> # ./exec_cmd.py ls -l
>
> However, if I pass 'ls' and arguments as one argument, like so:
> #./exec_cmd.py 'ls -al'
> exception does not occur.
>
> I logged output (see ls_test.out).
>
> So, what could be causing this behavior? Is this expected or is there
> something wrong with how I'm using the subprocess module?

You shouldn't use shell=True with a list argument (sys.argv[1:] is a
list). Specify the command as a string or use shell=False.
See http://bugs.python.org/issue21347


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CSV methodology

[Akira Li]

je...@newsguy.com writes:

> Hello.  Back in the '80s, I wrote a fractal generator, which, over the years,
> I've modified/etc to run under Windows.  I've been an Assembly Language
> programmer for decades.  Recently, I decided to learn a new language,
> and decided on Python, and I just love it, and the various IDEs.
>
> Anyway, something I thought would be interesting, would be to export
> some data from my fractal program (I call it MXP), and write something
> in Python and its various scientific data analysis and plotting modules,
> and... well, see what's in there.
>

Tools that are worth mentioning: ipython notebook, pandas

For example,

http://nbviewer.ipython.org/github/twiecki/financial-analysis-python-tutorial/blob/master/1.%20Pandas%20Basics.ipynb


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: GCD in Fractions

[Akira Li]

Mark Lawrence  writes:

> On 24/09/2014 12:14, Mark Dickinson wrote:
>> Mark Lawrence  yahoo.co.uk> writes:
>>> Somebody got there first http://bugs.python.org/issue22477
>>
>> I think there's good reason to suspect that Brian Gladman and
>> blindanagram are one and the same. :-)
>>
>>>>> sorted("BrianGladman".lower()) == sorted("blindanagram")
>> True
>>
>
> Is that considered proof that I still reign supreme as the most
> unobservant person on the planet? :)

It might mean that you are *too observant* because I've noticed that
the order of the letters is different just now.

http://english.stackexchange.com/questions/8628/is-it-true-that-only-the-positions-of-the-first-and-last-letter-in-a-word-matter

[spoiler: it is not true in general but brains can do a lot of
"error-correction" subconsciously]


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Timezones

[Akira Li]

Seymore4Head  writes:

> This is not a new video, but it is new to me.
> https://www.youtube.com/watch?v=-5wpm-gesOY
>
> Any links to some easy to follow time zone math?

The point of the video is that you should not do it yourself, use
already written tools.

It is quite comprehensive video. Here's how the mentioned issues could
be resolved:

- erratic UTC offset changes by politicians in different countries --
  *use UTC timezone* for calculations, in addition (if necessary) store
  both the local time and timezone info. *Use pytz module* that provides
  access to the tz database to convert between timezones (many people
  get it wrong but it is not a rocket science). As said in the video, it
  won't help if the rules change a day before the DST transition but at
  least most [1] systems will be consistent. It also doesn't support
  some exotic timezone rules such as in Saudi Arabia (sunrise and/or
  sunset at the fixed local time every day [2]) or many timezones before
  1970.

- julian, gregorian calendars, proleptic UT (past dates): *use proleptic
  gregorian calendar (datetime module)* unless you know that you need
  otherwise (then you could investigate what JD means in a particular
  document given multiple possible interpretations)

- leap seconds (posix, Mills, Windows(ntp), smear (google, java)),
  "right" timezones, and the difference between UT1, UTC(BIMP), GPS,
  TT(TAI), TCB, etc time scales: *ignore them* unless you know that you
  need otherwise e.g., legally elapsed seconds [3]. Here's some pictures
  to illustrate the difference between time scales [4].

[1] http://www.iana.org/time-zones/repository/tz-link.html
[2] https://github.com/eggert/tz/blob/master/asia#L2439
[3] http://www.ucolick.org/~sla/leapsecs/epochtime.html
[4] http://www.ucolick.org/~sla/leapsecs/deltat.html


--
Akira

-- 
https://mail.python.org/mailman/listinfo/python-list

Pythonic list/tuple/dict layout?

[Akira Kitada]

Hi,

There is more than one way to write a list/tuple/dict in Python,
and actually different styles are used in standard library.
As a hobgoblin of little minds, I rather like to know which style is
considered "Pythonic"
in the community.

I collected common layout from existing code and pasted them below.
My vote would go to d1. How about yours?

If there is consensus on this, that might be worth being included in PEP 8.

Thanks,

"""
d1 = {
0: "ham",
1: "jam",
2: "spam",
3: "alot",
4: "knights",
5: "who",
6: "say",
7: "ni",
8: "dead",
9: "parrot",
}

d2 = {
0: "ham",
1: "jam",
2: "spam",
3: "alot",
4: "knights",
5: "who",
6: "say",
7: "ni",
8: "dead",
9: "parrot",}

d3 = {0: "ham",
  1: "jam",
  2: "spam",
  3: "alot",
  4: "knights",
  5: "who",
  6: "say",
  7: "ni",
  8: "dead",
  9: "parrot",
  }

d4 = {0: "ham",
  1: "jam",
  2: "spam",
  3: "alot",
  4: "knights",
  5: "who",
  6: "say",
  7: "ni",
  8: "dead",
  9: "parrot",}

d5 = {0: "ham",
  1: "jam",
  2: "spam",
  3: "alot",
  4: "knights",
  5: "who",
  6: "say",
  7: "ni",
  8: "dead",
  9: "parrot",
}
"""
--
http://mail.python.org/mailman/listinfo/python-list

Re: Pythonic list/tuple/dict layout?

[Akira Kitada]

> Wow!  A Python debate over curly brace placement!  Imagine that!

PEP8 even deals with tabs vs spaces, where to put a blank line, etc :)
--
http://mail.python.org/mailman/listinfo/python-list

Re: Pythonic list/tuple/dict layout?

[Akira Kitada]

> BTW, there's no need to use such large examples. Three items per dict
> would be sufficient to illustrate the styles, using ten items doesn't add
> anything useful to the discussion.

I worried to be told
'you can make it in a line like {"ham": "jam", "spam": "alot"}'
;)
--
http://mail.python.org/mailman/listinfo/python-list

Re: Pythonic list/tuple/dict layout?

[Akira Kitada]

> These are the only two that follow PEP 8; the others don't have
> four-space indent levels.

In those examples, the following sentence in PEP 8 would be applied.

"Make sure to indent the continued line appropriately."

> I actually use this style:
>
>foo = {
>0: 'spam',
>1: 'eggs',
>2: 'beans',
>}
>
> because that makes it clear that *all* the indented lines are a
> continuation of the same statement, just like a suite of statements
> are all uniformly indented under (e.g.) a function definition.
>
> It seems that the author of the Python editing mode in Emacs agrees
> with me too (the style, at least, if not the reasoning), which makes
> my programming life easier.

Yes, it does, but people around me tend to prefer d1 style to that one.
One purpose of this silly thread is to figure out which is most popular one...
--
http://mail.python.org/mailman/listinfo/python-list

Re: *.python.org broken?

[Akira Kitada]

http://downforeveryoneorjustme.com/

On Sun, Jan 25, 2009 at 10:06 AM,   wrote:
> Hi all,
>
> Is anybody else having trouble accessing sites (including www, docs,
> wiki) in the python.org tree, or is it just me? (Or just .au?)
>
> Cheers,
>
> Tim
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
http://mail.python.org/mailman/listinfo/python-list

Re: python3 tutorial for newbie

[Akira Kitada]

http://wiki.python.org/moin/Python3.0Tutorials

On Wed, Feb 11, 2009 at 3:22 AM, Gary Wood  wrote:
> Can someone recommend a good tutorial for Python 3, ideally that has tasks
> or assignments at the end of each chapter.
> Please,
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>
--
http://mail.python.org/mailman/listinfo/python-list

New book "The Python Programming Language" by Guido van Rossum

[Akira Kitada]

The Python Programming Language by Guido van Rossum, Raymond Hettinger,
Jack Diedrich, David Beazley, David Mertz, Nicholas Coghlan to be published.
http://www.amazon.co.uk/Python-Programming-Language-Guido-Rossum/dp/0132299690

Anyone found the TOC of this?

Thanks,
--
http://mail.python.org/mailman/listinfo/python-list

Cannot build _multiprocessing, math, mmap and readline of Python 2.6 on FreeBSD 4.11 w/ gcc 2.95.4

[Akira Kitada]

 function `math_asinh':
/usr/home/build/dev/Python-2.6/Modules/mathmodule.c:276: `asinh'
undeclared (first use in this function)
/usr/home/build/dev/Python-2.6/Modules/mathmodule.c: In function `math_atanh':
/usr/home/build/dev/Python-2.6/Modules/mathmodule.c:283: `atanh'
undeclared (first use in this function)
/usr/home/build/dev/Python-2.6/Modules/mathmodule.c: In function
`math_copysign':
/usr/home/build/dev/Python-2.6/Modules/mathmodule.c:288: `copysign'
undeclared (first use in this function)
/usr/home/build/dev/Python-2.6/Modules/mathmodule.c: In function `math_log1p':
/usr/home/build/dev/Python-2.6/Modules/mathmodule.c:301: `log1p'
undeclared (first use in this function)
/usr/home/build/dev/Python-2.6/Modules/mathmodule.c: In function `math_ldexp':
/usr/home/build/dev/Python-2.6/Modules/mathmodule.c:666: `copysign'
used prior to declaration
/usr/home/build/dev/Python-2.6/Modules/mathmodule.c:670: `copysign'
used prior to declaration
/usr/home/build/dev/Python-2.6/Modules/mathmodule.c: In function `math_modf':
/usr/home/build/dev/Python-2.6/Modules/mathmodule.c:699: `copysign'
used prior to declaration
/usr/home/build/dev/Python-2.6/Modules/mathmodule.c: In function `math_pow':
/usr/home/build/dev/Python-2.6/Modules/mathmodule.c:901: `copysign'
used prior to declaration
"""

"""
building 'mmap' extension
gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -O3 -Wall
-Wstrict-prototypes -I. -I/usr/home/build/dev/Python-2.6/./Include -I.
-IInclude -I./Include -I/usr/local/include
-I/usr/home/build/dev/Python-2.6/I
nclude -I/usr/home/build/dev/Python-2.6 -c
/usr/home/build/dev/Python-2.6/Modules/mmapmodule.c -o
build/temp.freebsd-4.11-RELEASE-i386-2.6/usr/home/build/dev/Python-2.6/Modules/mmapmodule.o
/usr/home/build/dev/Python-2.6/Modules/mmapmodule.c: In function `initmmap':
/usr/home/build/dev/Python-2.6/Modules/mmapmodule.c:1440: warning:
implicit declaration of function `my_getallocationgranularity'
gcc -shared 
build/temp.freebsd-4.11-RELEASE-i386-2.6/usr/home/build/dev/Python-2.6/Modules/mmapmodule.o
-L/usr/local/lib -o build/lib.freebsd-4.11-RELEASE-i386-2.6/mmap.so
*** WARNING: renaming "mmap" since importing it failed:
build/lib.freebsd-4.11-RELEASE-i386-2.6/mmap.so: Undefined symbol
"my_getallocationgranularity"
"""

"""
building 'readline' extension
gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -O3 -Wall
-Wstrict-prototypes -I. -I/usr/home/build/dev/Python-2.6/./Include -I.
-IInclude -I./Include -I/usr/local/include
-I/usr/home/build/dev/Python-2.6/I
nclude -I/usr/home/build/dev/Python-2.6 -c
/usr/home/build/dev/Python-2.6/Modules/readline.c -o
build/temp.freebsd-4.11-RELEASE-i386-2.6/usr/home/build/dev/Python-2.6/Modules/readline.o
In file included from /usr/include/readline/readline.h:37,
 from /usr/home/build/dev/Python-2.6/Modules/readline.c:31:
/usr/include/readline/keymaps.h:40: warning: function declaration
isn't a prototype
/usr/include/readline/keymaps.h:41: warning: function declaration
isn't a prototype
/usr/include/readline/keymaps.h:42: warning: function declaration
isn't a prototype
/usr/include/readline/keymaps.h:43: warning: function declaration
isn't a prototype
In file included from /usr/home/build/dev/Python-2.6/Modules/readline.c:31:
/usr/include/readline/readline.h:343: warning: function declaration
isn't a prototype
/usr/home/build/dev/Python-2.6/Modules/readline.c:38: syntax error
before `rl_compentry_func_t'
/usr/home/build/dev/Python-2.6/Modules/readline.c:38: warning:
function declaration isn't a prototype
/usr/home/build/dev/Python-2.6/Modules/readline.c: In function
`set_completion_display_matches_hook':
/usr/home/build/dev/Python-2.6/Modules/readline.c:216:
`rl_compdisp_func_t' undeclared (first use in this function)
/usr/home/build/dev/Python-2.6/Modules/readline.c:216: (Each
undeclared identifier is reported only once
/usr/home/build/dev/Python-2.6/Modules/readline.c:216: for each
function it appears in.)
/usr/home/build/dev/Python-2.6/Modules/readline.c:216: syntax error before `)'
/usr/home/build/dev/Python-2.6/Modules/readline.c: At top level:
/usr/home/build/dev/Python-2.6/Modules/readline.c:669: warning:
`on_completion_display_matches_hook' defined but not used
"""

Because FreeBSD is not listed on
http://www.python.org/dev/peps/pep-0011/, I suppose it's still a
supported platform.
Any help, suggestions would be appreciated.

Thanks,

Akira
--
http://mail.python.org/mailman/listinfo/python-list

Re: Cannot build _multiprocessing, math, mmap and readline of Python 2.6 on FreeBSD 4.11 w/ gcc 2.95.4

[Akira Kitada]

Hi Marc-Andre,

Thanks for the suggestion.
I opened a ticket for this issue: http://bugs.python.org/issue4204

Now I understand the state of the multiprocessing module,
but it's too bad to see math, mmap and readline modules, that worked
fine before,
cannot be built anymore.

As for FreeBSD4, yeah it's really dated and I understand newer FreeBSD should
make my life easier, but I would rather want Python continue to
support old system like this
as long as it's not getting very hard to maintain the clean code base.

Thanks,

On Sat, Oct 25, 2008 at 10:53 PM, M.-A. Lemburg <[EMAIL PROTECTED]> wrote:
> On 2008-10-25 08:39, Akira Kitada wrote:
>> Hi list,
>>
>> I was trying to build Python 2.6 on FreeBSD 4.11 and found it failed
>> to build some of the modules.
>>
>> """
>> Failed to find the necessary bits to build these modules:
>> _bsddb _sqlite3   _tkinter
>> gdbm   linuxaudiodev  spwd
>> sunaudiodev
>> To find the necessary bits, look in setup.py in detect_modules() for
>> the module's name.
>>
>>
>> Failed to build these modules:
>> _multiprocessing   math   mmap
>> readline
>> """
>>
>> Because I don't have Berkeley DB, SQLite3 tk, GDBM installed on the
>> system and running FreeBSD,
>> there is no wonder it failed to build  _bsddb, _sqlite3, _tkinter,
>> gdbm, linuxaudiodev, spwd and sunaudiodev.
>>
>> The problem is it failed to build _multiprocessing, math, mmap and readline.
>
> Please post a bug report on python.org about these failures.
>
> The multiprocessing module is still fairly new and obviously needs
> more fine tuning for the large set of platforms on which Python
> can run. However, please also note that FreeBSD4 is a rather old
> version of that OS. FWIW: Python 2.6 compiles just fine on FreeBSD6.
>
> Thanks.
>
>> Here are the outputs of each build failure.
>>
>> """
>> building '_multiprocessing' extension
>> creating 
>> build/temp.freebsd-4.11-RELEASE-i386-2.6/usr/home/build/dev/Python-2.6/Modules/_multiprocessing
>> gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -O3 -Wall
>> -Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1
>> -DHAVE_SEM_TIMEDWAIT=1 -IModules/_multiprocessing -I.
>> -I/usr/home/build/dev/Python-2.6/./
>> Include -I. -IInclude -I./Include -I/usr/local/include
>> -I/usr/home/build/dev/Python-2.6/Include
>> -I/usr/home/build/dev/Python-2.6 -c
>> /usr/home/build/dev/Python-2.6/Modules/_multiprocessing/multiprocessing.c
>> -o b
>> uild/temp.freebsd-4.11-RELEASE-i386-2.6/usr/home/build/dev/Python-2.6/Modules/_multiprocessing/multiprocessing.o
>> In file included from
>> /usr/home/build/dev/Python-2.6/Modules/_multiprocessing/multiprocessing.h:24,
>>  from
>> /usr/home/build/dev/Python-2.6/Modules/_multiprocessing/multiprocessing.c:9:
>> /usr/include/arpa/inet.h:89: warning: parameter has incomplete type
>> /usr/include/arpa/inet.h:92: warning: parameter has incomplete type
>> /usr/include/arpa/inet.h:96: warning: parameter has incomplete type
>> /usr/home/build/dev/Python-2.6/Modules/_multiprocessing/multiprocessing.c:
>> In function `multiprocessing_sendfd':
>> /usr/home/build/dev/Python-2.6/Modules/_multiprocessing/multiprocessing.c:102:
>> storage size of `dummy_iov' isn't known
>> /usr/home/build/dev/Python-2.6/Modules/_multiprocessing/multiprocessing.c:102:
>> warning: unused variable `dummy_iov'
>> /usr/home/build/dev/Python-2.6/Modules/_multiprocessing/multiprocessing.c:
>> In function `multiprocessing_recvfd':
>> /usr/home/build/dev/Python-2.6/Modules/_multiprocessing/multiprocessing.c:137:
>> storage size of `dummy_iov' isn't known
>> /usr/home/build/dev/Python-2.6/Modules/_multiprocessing/multiprocessing.c:137:
>> warning: unused variable `dummy_iov'
>> """
>>
>> """
>> building 'cmath' extension
>> gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -O3 -Wall
>> -Wstrict-prototypes -I. -I/usr/home/build/dev/Python-2.6/./Include -I.
>> -IInclude -I./Include -I/usr/local/include
>> -I/usr/home/build/dev/Python-2.6/I
>> nclude -I/usr/home/build/dev/Python-2.6 -c
>> /usr/home/build/dev/Python-2.6/Modules/cmathmodule.c -o
>> build/temp.freebsd-4.11-RELEASE-i386-2.6/usr/home/build/dev/Python-2.6/Modules/cmathmodule.o
>> /usr/home/build/dev/Python-2.6/Modules/cmathmodule.c: In function
>> `special_type':
>> /usr/home/build/dev/Python-2.6/Modules/cma

Can I build Python with lthread instead of pthread?

[Akira Kitada]

Hi,

I'm running Python 2.5 on FreeBSD 4.
pthread on FreeBSD 4 has some problems so I would like to build
python with lthread (linuxthreads) instead of BSD's pthread.
So I looked at configure options but couldn't find any options for it.
Python support lthread? and how can I build python with it?

Thanks in advance,

Akira
--
http://mail.python.org/mailman/listinfo/python-list

Re: Performance of Python 3

[Akira Kitada]

Is this what you are looking for?
http://shootout.alioth.debian.org/u32q/benchmark.php?test=all&lang=python3&lang2=yarv&box=1


On Sun, Mar 1, 2009 at 10:04 PM, Kless  wrote:
> Does anybody has seen the performance of Python 3?
> Respect to speed it's the last language together to Ruby 1.8, but Ruby
> 1.9 has a lot of better performance. :(
>
>
> http://shootout.alioth.debian.org/u32q/benchmark.php?test=all&lang=all
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
http://mail.python.org/mailman/listinfo/python-list

Why is it that *dbm modules don't provide an iterator? (Language design question)

[Akira Kitada]

Hi,

I was wondering why *dbm modules in Python do not give us an iterable interface?
Take a look at an example below

"""
# Python 2.6
>>> import gdbm
>>> d = gdbm.open("spam.db", "n")
>>> d["key1"] = "ham"
>>> d["key2"] = "spam"
>>>
>>> for k in d:
... print k
...
Traceback (most recent call last):
  File "", line 1, in 
TypeError: 'gdbm.gdbm' object is not iterable
"""

The loop has to be:
"""
>>> k = d.firstkey()
>>> while k != None:
...print k
...k = d.nextkey(k)
key2
key1
"""

I would like to know the background of this design decision.
Some research on this led me to this ticket:

http://bugs.python.org/issue662923

It looks I'm not the only one who wondered on this.

Thanks in advance.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Why is it that *dbm modules don't provide an iterator? (Language design question)

[Akira Kitada]

keys() returns a list and my question was not about "how to" but more
like "why"...
I assumed there were some decisions behind this, rather than it's just
not implemented yet.

Best,
On Friday, April 10, 2009, Joshua Kugler  wrote:
> Akira Kitada wrote:
>
>> The loop has to be:
>> """
>>>>> k = d.firstkey()
>>>>> while k != None:
>> ...    print k
>> ...    k = d.nextkey(k)
>> key2
>> key1
>> """
>
> Why not
>
> for key in d.keys():
>     print key
>
> That worked for me.
>
> j
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
http://mail.python.org/mailman/listinfo/python-list

Re: Why is it that *dbm modules don't provide an iterator? (Language design question)

[Akira Kitada]

Opened a ticket for this and attached a patch. (experimental)
http://bugs.python.org/issue5736

On Fri, Apr 10, 2009 at 8:39 AM, "Martin v. Löwis"  wrote:
 I assumed there were some decisions behind this, rather than it's just
 not implemented yet.
>>> I believe this assumption is wrong - it's really that no code has been
>>> contributed to do that.
>>
>> But doesn't the issue at http://bugs.python.org/issue662923 imply that
>> there *was* suitable code (for 2.4)?
>
> Yes, for bsddb.
>
>> So is this a regression?
>
> No. bsddb still supports iteration in 2.6.
>
>> And should that issue be re-opened?
>
> I don't think that would matter much. Actually contributing code might.
>
> Regards,
> Martin
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
http://mail.python.org/mailman/listinfo/python-list

Re: Top Python Interview Questions

[Akira Li]

Terry Reedy  writes:

> On 5/26/2017 1:03 AM, Aarusha wrote:
>> PYTHON INTERVIEW QUESTIONS
>>
>> Mindmajix has compiled Python Interview questions which would
>> benefit the learners to attend the Python interviews.
>>
>> Q. How is Python executed?
>
> It depends on the implementation (interpreter or compiler).
>
>> Python files are compiled to bytecode.
>
...
> Cython and others compiles to C (or C++) which is compiled to machine code.

Cython generates C/C++ code that depends on CPython API such as
PyNumber_TrueDivide(). For comparison, RPython (a restricted subset of
Python used by Pypy) can be statically compiled.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Transitioning from Linux to Windows

[Akira Li]

chitt...@uah.edu writes:

> ...
> Ideally, I would like to set up the user on their Windows 7/10 system
> so that they can "login" to the ubuntu system (say putty) - change
> working directory (to where desired) - run the script (on the ubuntu
> system) - and scp the file back to the windows desktop.
>
> ("porting" the ubuntu script to anaconda(3) on the windows desktop IS
> possible (but it has not been as easy as I had hoped!) (django and
> programs like that do seem to provide a "GUI" to have python scripts
> run on ubuntu/systems - but the setup looks mysterious/complicated (to
> me anyway))
>
> I stumbled onto "paramiko" - is that a possible answer?  
> ...


You could use "fabric" http://www.fabfile.org/ to automate running shell
commands (such as python scripts) via ssh.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is An Element of a Sequence an Object?

[Akira Li]

Jon Forrest  writes:

> I'm learning about Python. A book I'm reading about it
> says "...
> a string in Python is a sequence.

correct.

> A sequence is an ordered collection of objects".

correct. https://docs.python.org/3/glossary.html#term-sequence

> This implies that each character in a string
> is itself an object.

Everything in Python is an object. If *s* is a name that refers to a str
object then *s[i]* returns i-th Unicode code point from the string as a
str object of length 1:

  >>> s = "ab"
  >>> s[0]
  'a'

It does not matter how *s* is represented internally on a chosen Python
implementation: *s[0]* may return an existing object or it may create a
new one on-the-fly.

Here's a perfectly valid sequence of ten squares:

  >>> class Squares:
  ... def __getitem__(self, i):
  ... if not (-len(self) <= i < len(self)):
  ... raise IndexError(i)
  ... if i < 0:
  ... i += len(self)
  ... return i * i
  ...
  ... def __len__(self):
  ... return 10
  >>> s = Squares()
  >>> s[9]
  81

All 10 squares are generated on-the-fly (though all int objects already
exist due to the small int caching on CPython).

-- 
https://mail.python.org/mailman/listinfo/python-list