Re: UNABLE TO GET IDLE TO RUN
In a message of Fri, 23 Oct 2015 11:42:54 +0200, Peter Otten writes: >I tried it out: > >$ mkdir test >$ cd test >$ touch string.py >$ idle3 >Traceback (most recent call last): > File "", line 1, in > File "/usr/lib/python3.4/idlelib/run.py", line 12, in >from idlelib import CallTips > File "/usr/lib/python3.4/idlelib/CallTips.py", line 16, in >from idlelib.HyperParser import HyperParser > File "/usr/lib/python3.4/idlelib/HyperParser.py", line 14, in >_ASCII_ID_CHARS = frozenset(string.ascii_letters + string.digits + "_") >AttributeError: 'module' object has no attribute 'ascii_letters' > >Then idle shows the message and quits after you hit OK. > >Is there a bug report? I didn't make one at the time. I am not finding one on the tracker that looks like it. Laura -- https://mail.python.org/mailman/listinfo/python-list
Best way to do background calculations?
tl;dr: I've been using the multiprocessing module to run some calculations in the background of my CherryPy web app, but apparently this process sometimes gets stuck, causing problems with open sockets piling up and blocking the app. Is there a better way?The (rather wordy) details:I have a moderately busy web app written in python using the CherryPy framework (CherryPy v 3.8.0 with ws4py v 0.3.4 and Python 2.7.6) . One of the primary purposes of this web app is to track user-entered flight logs, and keep a running tally of hours/cycles/landings for each aircraft. To that end, whenever a user enters or modifies a log, I "recalculate" the totals for that aircraft, and update all records with the new totals. There are probably ways to optimize this process, but so far I haven't seen a need to spend the time.Ideally, this recalculation process would happen in the background. There is no need for the user to wait around while the system crunches numbers - they should be able to move on with entering another log or whatever else they need to do. To that end, I implemented the call to the recalc function using the multiprocessing module, so it could start in the background and the main process move on.Lately, though, I've been running into a problem where, when looking at the process list on my server (Mac OS X 10.10.5), I'll see two or more "copies" of my server process running - one master and one or more child processes. As the above described process is the only place I am using the multiprocessing module, I am making the assumption that this is what these additional processes are. If they were only there for a few minutes I would think this is normal, and it wouldn't be a problem. However, what I am seeing is that from time to time (once or twice every couple of days) these additional processes will get "stuck", and when that happens sockets opened by the web app don't get properly closed and start piling up. Looking at a list of open sockets on the server when I have one of these "hung" processes shows a steadily increasing number of sockets in a "CLOSE_WAIT" state (normally I see none in that state). Killing off the hung process(es) clears out these sockets, but if I don't catch it quickly enough these sockets can build up to the point that I am unable to open any more, and the server starts rejecting connections.I'm told this happens because the process retains a reference to all open files/sockets from the parent process, thus preventing the sockets from closing until the process terminates. Regardless of the reason, it can cause a loss of service if I don't catch it quickly enough. As such, I'm wondering if there is a better way. Should I be looking at using the threading library rather than the multiprocessing library? My understanding is that the GIL would prevent that approach from being of any real benefit for a calculation intensive type task, but maybe since the rest of the application is CherryPy threads, it would still work well?. Or perhaps there is a way to not give the child process any references to the parent's files/sockets - although that may not help with the process hanging? Maybe there is a way to "monitor" the process, and automatically kill it if it stops responding? Or am I totally barking up the wrong tree here?Thanks for any insight anyone can provide! ---Israel BrewsterSystems Analyst IIRavn Alaska5245 Airport Industrial RdFairbanks, AK 99709(907) 450-7293---BEGIN:VCARD VERSION:3.0 N:Brewster;Israel;;; FN:Israel Brewster ORG:Frontier Flying Service;MIS TITLE:PC Support Tech II EMAIL;type=INTERNET;type=WORK;type=pref:[email protected] TEL;type=WORK;type=pref:907-450-7293 item1.ADR;type=WORK;type=pref:;;5245 Airport Industrial Wy;Fairbanks;AK;99701; item1.X-ABADR:us CATEGORIES:General X-ABUID:36305438-95EA-4410-91AB-45D16CABCDDC\:ABPerson END:VCARD -- https://mail.python.org/mailman/listinfo/python-list
OT snake_case
I just discovered today that writing names in lowercase with underscores is sometimes called "snake case". Seems kind of apt for Python! :-) Apparently, it was coined in the Ruby community... -- https://mail.python.org/mailman/listinfo/python-list
Re: Best way to do background calculations?
On 2015-10-23 17:35, Israel Brewster wrote: tl;dr: I've been using the multiprocessing module to run some calculations in the background of my CherryPy web app, but apparently this process sometimes gets stuck, causing problems with open sockets piling up and blocking the app. Is there a better way? The (rather wordy) details: I have a moderately busy web app written in python using the CherryPy framework (CherryPy v 3.8.0 with ws4py v 0.3.4 and Python 2.7.6) . One of the primary purposes of this web app is to track user-entered flight logs, and keep a running tally of hours/cycles/landings for each aircraft. To that end, whenever a user enters or modifies a log, I "recalculate" the totals for that aircraft, and update all records with the new totals. There are probably ways to optimize this process, but so far I haven't seen a need to spend the time. Ideally, this recalculation process would happen in the background. There is no need for the user to wait around while the system crunches numbers - they should be able to move on with entering another log or whatever else they need to do. To that end, I implemented the call to the recalc function using the multiprocessing module, so it could start in the background and the main process move on. Lately, though, I've been running into a problem where, when looking at the process list on my server (Mac OS X 10.10.5), I'll see two or more "copies" of my server process running - one master and one or more child processes. As the above described process is the only place I am using the multiprocessing module, I am making the assumption that this is what these additional processes are. If they were only there for a few minutes I would think this is normal, and it wouldn't be a problem. However, what I am seeing is that from time to time (once or twice every couple of days) these additional processes will get "stuck", and when that happens sockets opened by the web app don't get properly closed and start piling up. Looking at a list of open sockets on the server when I have one of these "hung" processes shows a steadily increasing number of sockets in a "CLOSE_WAIT" state (normally I see none in that state). Killing off the hung process(es) clears out these sockets, but if I don't catch it quickly enough these sockets can build up to the point that I am unable to open any more, and the server starts rejecting connections. I'm told this happens because the process retains a reference to all open files/sockets from the parent process, thus preventing the sockets from closing until the process terminates. Regardless of the reason, it can cause a loss of service if I don't catch it quickly enough. As such, I'm wondering if there is a better way. Should I be looking at using the threading library rather than the multiprocessing library? My understanding is that the GIL would prevent that approach from being of any real benefit for a calculation intensive type task, but maybe since the rest of the application is CherryPy threads, it would still work well?. Or perhaps there is a way to not give the child process any references to the parent's files/sockets - although that may not help with the process hanging? Maybe there is a way to "monitor" the process, and automatically kill it if it stops responding? Or am I totally barking up the wrong tree here? It sounds like the multiprocessing module is forking the new process, which inherits the handles. Python 3.4 added the ability to spawn the new process, which won't inherit the handles. It's unfortunate that you're using Python 2.7.6! Could you start the background process early, before any of those sockets have been opened, and then communicate with it via queues? -- https://mail.python.org/mailman/listinfo/python-list
Detection of a specific sound
I'm looking for a library that will allow Python to listen for the shriek of a smoke alarm. Once it detects this shriek, it is to notify someone. Ideally, specificity can be adjusted for the user's environment. For example, I expect to need moderate specificity as I live in a quiet neighborhood, but an apartment dweller might need more. I'm thinking of recording a smoke alarm and having the program try to find the recorded sound in the stream from the microphone. Any help is greatly appreciated! -- https://mail.python.org/mailman/listinfo/python-list
ANN: lfm v3.0
Hi all,
after many years, new version of lfm; now compatible with Python 3.4+.
Description
===
Last File Manager is a powerful file manager for the UNIX console.
It has a curses interface and it's written in Python v3.4+.
Licensed under GNU Public License version 3 or later.
Some of the features you can find in lfm:
- console-based file manager for UNIX platforms
- 1-pane or 2-pane view
- tabs
- files filters
- bookmarks
- history
- VFS for compressed files
- tree view
- dialogs with entry completion
- PowerCLI, a command line interface with advanced features
- fast access to the shell
- direct integration of find/grep, df and other tools
- color files by extension [Andrey Skvortsov]
- fully customizable themes (colors)
- fully customizable key bindings
- support for filenames with wide chars, f.e. East Asian
- ...and many others
Home page: https://inigo.katxi.org/devel/lfm/
Pypi: https://pypi.python.org/pypi/lfm
Code repository: https://bitbucket.org/inigoserna/lfm3
Changes since last version
==
Version 3.0 ("Only you") - 2015/10/23:
+ About the code
- almost completely rewritten from scratch
. it hasn't been tested as much as lfm v2.x series on non-linux OS
- requires Python v3.4+
- pyview, the file viewer, has been removed from lfm package
- configuration location has changed to a new directory and files:
~/.config/lfm/{lfm.ini, lfm.keys, lfm.theme, lfm.history}
+ New features
- fully customizable themes (colors)
- fully customizable key bindings
. allow Alt-key shorcuts (A-)
. only for main window (not for dialogs)
- files filters (using globs)
. information in pane frame: ".F" => show dotfiles, active filters
. filters are a property of a tab, they remain active even when chdir
. Ctrl-f: edit current filter
. Some examples:
. "*.png,*.jpg" => hide all PNG and JPEG files
. "*.jpg,!*shot*" => hide all JPEG files except those with 'shot'
in the name
. "*,!*py" => hide all except python source files
- up to 35 bookmarks (0-9, a-z)
. b: go to bookmark, B: set bookmark, C-d: select bookmark
. fix: don't delete bookmark at start if path does not exist
- nested archive handling (vfs inside vfs) works now
- added optional support for filenames with wide chars, f.e. East Asian
. to enable, set 'use_wide_chars' flag in configuration or use -w
command line flag
. it's disabled by default for performance
. it's not perfect, but it mostly works
- there are 2 different versions of move_file to chose from in the key
bindings file:
. move_file: old implementation
. move_file2: alternative version using shtutil.move instead of
copy & delete. Faster but less control of errors
- new action: redraw screen (default key A-r)
+ Changes & improvements (vs v2.x):
- chmod & chown/chgrp are 2 different actions now
- cursor_goto_file (C-s): find text pattern (no regex or glob) in the whole
file name, not at the beginning as v2.x
- cursor_goto_file_1char (A-s): go to file by 1st letter of name
(old C-s behaviour)
- bookmarks have new key bindings:
b: go to bookmark, B: set bookmark, C-d: select bookmark
- PowerCLI:
. ending command with % must be $ now
. added new date variables: dm, dc, da, dn
. ending command with % must be $ now
. added new date variables: dm, dc, da, dn
. old $d variable is $p now
- find/grep: panelize = create vfs with matched files
. if rebuild: all files modifications or deletions are translated to
original directory so be careful!
- pyview, the file viewer, has been removed from lfm package
+ Fixes in v3.0 (vs v2.x):
- nested archive handling (vfs inside vfs) works now
- find & grep with spaces in file name
- wide chars file names support (f.e. Eastern languages) => lfm -w
- sort by size after show dirs size
- move_file: not overwritten files in destination are not deleted
Of course, all comments, suggestions, etc. are welcome.
Best regards,
Iñigo Serna
--
https://mail.python.org/mailman/listinfo/python-list
Ignore stderr and use return code
Hi Teamm
In the below code, we found that the command i.e cmd = "mount
/filesystem1" succeeded. But the test failed due to the weaker stderr
def mount_me():
cmd = "mount /filesystem1"
proc = subprocess.Popen(shlex.split(cmd),
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = proc.communicate()
if err != "":
logging.error("Can't run %s got %s!" % (cmd, err))
return False
Verification:
>> print err
10/25/2015 12:10:28 PM ERROR:Can't run mount /filesystem1 got :
mount_err: Reading GUID from 1213: No such file or directory
#df -kh
FilesystemSize Used Avail Use% Mounted on
Filesystem 38G 5.5G 30G 16% /filesystem1
- To handle this case, Iam planning to use return code and modify the
above code as below ( Any other suggestions please let me know)
def mount_me():
cmd = ("mount /filesystem1")
out, err, ret = run(cmd, timeout=60) # run is the wrapper,
returns (stdout, stderr, returncode)
if ret != 0: # zero means succeeded
logging.error("Can't run %s got %s (%d)!" % (cmd, err, ret))
return False
- Do I need to add more check to ensure the mount actually succeeds,
may be a function?
if ret != 0 and check_df_output():
logging.error("Can't run %s got %s (%d)!" % (cmd, err, ret))
Iam using python 2.7 on Linux
Regards,
Ganesh
--
https://mail.python.org/mailman/listinfo/python-list
Please post “text/plain” message body (was: Best way to do background calculations?)
Israel Brewster writes: [no text] Please ensure your email message body is “text/plain” (and preferably not HTML) when posting to unfamiliar recipients — which is always, on a public forum like this. -- \“Pinky, are you pondering what I'm pondering?” “Umm, I think | `\ so, Brain, but three men in a tub? Ooh, that's unsanitary!” | _o__) —_Pinky and The Brain_ | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: UNABLE TO GET IDLE TO RUN
On 10/23/2015 5:42 AM, Peter Otten wrote: Laura Creighton wrote: In a message of Fri, 23 Oct 2015 00:19:42 -0400, Terry Reedy writes: On 10/21/2015 11:24 AM, Terry Alexander via Python-list wrote: I have tried installing both Python 2.7 and 3.5, and in both cases I cannot get IDLE to work. I received the following message both times: What OS? Windows? which version? How did you start IDLE? Start menu icon? Command line? IDLE’s subprocess didn’t make connection.Either IDLE can’t start a subprocess or personal firewall software is blocking the connection. I am running Norton, and disabled it, but still IDLE will not run. Any suggestions? Don't shout with ALL CAPS in the subject line. It usually indicates spam. I already know that this problem is very frustrating. Firewalls are seldom the problems anymore. I occasionally saw this on Win 7 when restarting, but never on startup, and never more than once or twice in a session. What's left is misconfiguration of your network interface that prevents a loopback connection. There might be answers on Stackoverflow that would help, depending on your OS. In the meanwhile, you can start IDLE with the -n option. Either use a command line or create an 'IDLE -n' icon. Again, details depend on exact OS. Terry Jan Reedy You can also get this message if you run idle in directory where you have your own python file whose name shadows something in the standard library (that idle is interested in). I think it was a file named 'string.py' that did this to a student of mine a few years ago. Laura I can imagine that a bad socket module might have this effect. With verification that a user file could have this effect, I would augment the message. I tried it out: $ mkdir test $ cd test $ touch string.py $ idle3 Traceback (most recent call last): File "", line 1, in File "/usr/lib/python3.4/idlelib/run.py", line 12, in from idlelib import CallTips File "/usr/lib/python3.4/idlelib/CallTips.py", line 16, in from idlelib.HyperParser import HyperParser File "/usr/lib/python3.4/idlelib/HyperParser.py", line 14, in _ASCII_ID_CHARS = frozenset(string.ascii_letters + string.digits + "_") AttributeError: 'module' object has no attribute 'ascii_letters' Then idle shows the message and quits after you hit OK. Is there a bug report? No. Quitting because a stdlib module cannot be imported (or corrupted) is not a bug. AttributeError is a typical symptom. A user file called random.py is probably more common. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: OT snake_case
On 23/10/2015 17:53, MRAB wrote: I just discovered today that writing names in lowercase with underscores is sometimes called "snake case". Seems kind of apt for Python! :-) Apparently, it was coined in the Ruby community... So when I get around to forking Python to produce 2.8 the name will have to be Camel :) Or is that already taken? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: OT snake_case
On Fri, Oct 23, 2015 at 12:53 PM, MRAB wrote: > I just discovered today that writing names in lowercase with > underscores is sometimes called "snake case". > > Seems kind of apt for Python! :-) > > Apparently, it was coined in the Ruby community... > -- > https://mail.python.org/mailman/listinfo/python-list > I just got this message delivered at 10 pm Sunday. But it is an interesting piece of info. -- Joel Goldstick http://joelgoldstick.com/stats/birthdays -- https://mail.python.org/mailman/listinfo/python-list
Re: OT snake_case
MRAB writes: > I just discovered today that writing names in lowercase with > underscores is sometimes called "snake case". > > Seems kind of apt for Python! :-) Cute! > Apparently, it was coined in the Ruby community... Do you have references for tracing the origin of that term? -- \ “You can be a theist, and you can be a skeptic. But if you're | `\ both, you're not very good at one of them.” —Dave Silverman, | _o__) 2011-11-19 | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: how to get python socket to use a specific interface
On Friday 23 October 2015 05:03:41 Robin Becker wrote:
> I need to run lynx on a ubuntu headless server with a specific IP
> address in order to try and cancel an email blacklist.
>
> Lynx doesn't provide a way to do that so I thought to use a local
> proxy. I tried pymiproxy and hacked the connection code so it looked
> like this
>
>
> proxy.py
>
>
> > from socket import socket, SOL_SOCKET
> > ...
> > import os
> > BIND_ADDRESS = os.environ.get('PYMIPROXY_BIND_ADDRESS',None)
> > BIND_DEVICE = os.environ.get('PYMIPROXY_BIND_DEVICE',None)
> > .
> > # Connect to destination
> > sock = self._proxy_sock = socket()
> > sock.settimeout(10)
> > if BIND_ADDRESS:
> > sock.bind((BIND_ADDRESS, 0))
> > if BIND_DEVICE:
> > sock.setsockopt(SOL_SOCKET, 25, BIND_DEVICE)
> > sock.connect((self.hostname, int(self.port)))
>
> 25 is derived from /usr/include/asm-generic/socket.h
>
> > #define SO_BINDTODEVICE 25
>
> This works if I export BIND_DEVICE='eth0' and run as root, but my
> desired interface is actually called 'eth0:0' and when I run with that
> exported I get a device error. Does anyone know what I should call the
> device? --
> Robin Becker
Using eth0:0 is normally a method to setup eth0 to respond to a 2nd
IPV4/IPV6 address. Have you done the ifconfig steps to enable that? If
its been done, you will see it's 2nd address in an ifconfig query. Man
pages are wunnerful things.
I think you can use it as a normal user, but the ifconfig IIRC has to be
done by root, or sudo.
One of my printers has a cat5 socket, which turns out to be its fastest
interface, and I had to setup an eth0:0 in 192.168.0.## block for a day
or so until I had wandered thru its menus and found I could put it on my
local networks class D block and did, so eth0:0 was no longer required.
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page
--
https://mail.python.org/mailman/listinfo/python-list
Re: Detection of a specific sound
On 2015-10-26 00:17, Montana Burr wrote: I'm looking for a library that will allow Python to listen for the shriek of a smoke alarm. Once it detects this shriek, it is to notify someone. Ideally, specificity can be adjusted for the user's environment. For example, I expect to need moderate specificity as I live in a quiet neighborhood, but an apartment dweller might need more. I'm thinking of recording a smoke alarm and having the program try to find the recorded sound in the stream from the microphone. Any help is greatly appreciated! Here are a couple of questions from stackoverflow that might be of help: Python frequency detection http://stackoverflow.com/questions/2648151/python-frequency-detection Recognising tone of the audio http://stackoverflow.com/questions/1797631/recognising-tone-of-the-audio -- https://mail.python.org/mailman/listinfo/python-list
Re: Ignore stderr and use return code
On 25Oct2015 11:35, Ganesh Pal wrote:
In the below code, we found that the command i.e cmd = "mount
/filesystem1" succeeded. But the test failed due to the weaker stderr
[...]
out, err = proc.communicate()
if err != "":
Checking stderr is never the correct way to do this.
[...]
- To handle this case, Iam planning to use return code and modify the
above code as below ( Any other suggestions please let me know)
def mount_me():
cmd = ("mount /filesystem1")
out, err, ret = run(cmd, timeout=60) # run is the wrapper,
returns (stdout, stderr, returncode)
if ret != 0: # zero means succeeded
This is the correct approach.
logging.error("Can't run %s got %s (%d)!" % (cmd, err, ret))
return False
- Do I need to add more check to ensure the mount actually succeeds,
may be a function?
No. Mount should have a zero status if it succeeds and anon-zero exit status if
it fails. There is no grey area here.
if ret != 0 and check_df_output():
logging.error("Can't run %s got %s (%d)!" % (cmd, err, ret))
Rather than looking at the stdout or stderr of df, consider calling
os.statvfs() directly from Python:
https://docs.python.org/3/library/os.html#os.statvfs
Iam using python 2.7 on Linux
It is the same in Python 2.
Cheers,
Cameron Simpson
--
https://mail.python.org/mailman/listinfo/python-list
Re: Best way to do background calculations?
On Sat, Oct 24, 2015 at 3:35 AM, Israel Brewster wrote: > > Ideally, this recalculation process would happen in the background. There is > no need for the user to wait around while the system crunches numbers - they > should be able to move on with entering another log or whatever else they > need to do. To that end, I implemented the call to the recalc function using > the multiprocessing module, so it could start in the background and the main > process move on. One way to get around this would be to separate the processes completely, and simply alert the other process (maybe via a socket) to ask it to do the recalculation. That way, the background process would never have any of the main process's sockets, and can't affect them in any way. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: OT snake_case
On 2015-10-25 23:23, Ben Finney wrote: MRAB writes: I just discovered today that writing names in lowercase with underscores is sometimes called "snake case". Seems kind of apt for Python! :-) Cute! Apparently, it was coined in the Ruby community... Do you have references for tracing the origin of that term? https://groups.google.com/forum/#!msg/comp.lang.ruby/ra5WDzNGwGU/oe_mnvRVclUJ -- https://mail.python.org/mailman/listinfo/python-list
Re: Ignore stderr and use return code
Ganesh Pal writes:
> In the below code, we found that the command i.e cmd = "mount
> /filesystem1" succeeded. But the test failed due to the weaker stderr
I don't know what you mean by stderr being “weaker”.
> def mount_me():
> cmd = "mount /filesystem1"
> proc = subprocess.Popen(shlex.split(cmd),
> stdout=subprocess.PIPE, stderr=subprocess.PIPE)
> out, err = proc.communicate()
> if err != "":
> logging.error("Can't run %s got %s!" % (cmd, err))
> return False
The test ‘if err != ""’ implies you are checking *only* whether the value
is an empty string. It might be clearer if you write ‘if err’.
The error message implies the process *could not* run, but that's not
the case. If that line is reached, the process did run to completion.
The presence of output on stderr does not imply the process failed.
If instead you want to check whether the process failed, check its exit
status. The “exit status” of a process, indicating success or failure,
is “returned” by the subprocess back to Python and is available as
‘Popen.returncode’::
EXIT_STATUS_SUCCESS = 0
if proc.returncode != EXIT_STATUS_SUCCESS:
logging.error(
"Subprocess {cmd} failed with code {code:d},"
" error output {err}".format(
cmd=cmd, code=proc.returncode, err=err))
> Verification:
I don't quite understand the claim of “weaker”, so I don't know what is
verified here.
> - To handle this case, Iam planning to use return code and modify the
> above code as below ( Any other suggestions please let me know)
You may also want to think about whether raising an exception is
appropriate (rather than just returning False).
> - Do I need to add more check to ensure the mount actually succeeds,
> may be a function?
I'd suggest relying on the exit status from the command, unless you have
good evidence it is unreliable.
--
\ “When people believe that they have absolute knowledge, with no |
`\ test in reality, this [the Auschwitz crematorium] is how they |
_o__) behave.” —Jacob Bronowski, _The Ascent of Man_, 1973 |
Ben Finney
--
https://mail.python.org/mailman/listinfo/python-list
