python - handling HTTP requests asynchronously
Hi everyone, I need to generate a PDF report for each entry of a django queryset. There'll be between between 30k and 40k entries. The PDF is generated through an external API. Since currently is generated on demand, this is handled synchronously via an HTTP request/response. That will be different for this task, since I think I'll use a django management command to loop through the queryset and perform the PDF generation. Which approach should I follow for this task? I thought about 3 possibile solutions, although are technologies that I never used: 1) Celery: assign a task (http request with a different payload) to a worker, then retrieve it once it's done. 2) request-futures: using requests in a non-blocking way. 3) multiprocessing module, with e.g. 10 as workers limit. the goal is to use the API concurrently (e.g. send 10 or 100 http requests simultaneously, depending on how many concurrent requests the API can handle). Anybody here that handled a similar task and can give advices on how to proceed on this? -- https://mail.python.org/mailman/listinfo/python-list
Re: Whittle it on down
DFS wrote:
> On 5/5/2016 1:39 AM, Stephen Hansen wrote:
>
>> Given:
>>
> input = [u'Espa\xf1ol', 'Health & Fitness Clubs (36)', 'Health Clubs &
> Gymnasiums (42)', 'Health Fitness Clubs', 'Name', 'Atlanta city
> guide', 'edit address', 'Tweet', 'PHYSICAL FITNESS CONSULTANTS &
> TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS',
> 'www.custombuiltpt.com/', 'RACQUETBALL COURTS PRIVATE',
> 'www.lafitness.com', 'GYMNASIUMS', 'HEALTH & FITNESS CLUBS',
> 'www.lafitness.com', 'HEALTH & FITNESS CLUBS', 'www.lafitness.com',
> 'PERSONAL FITNESS TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'EXERCISE &
> PHYSICAL FITNESS PROGRAMS', 'FITNESS CENTERS', 'HEALTH CLUBS &
> GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS', 'PERSONAL FITNESS TRAINERS',
> '5', '4', '3', '2', '1', 'Yellow Pages', 'About Us', 'Contact Us',
> 'Support', 'Terms of Use', 'Privacy Policy', 'Advertise With Us', 'Add
> /Update Listing', 'Business Profile Login', 'F.A.Q.']
>>
>> Then:
>>
> pattern = re.compile(r"^[A-Z\s&]+$")
> output = [x for x in list if pattern.match(x)]
> output
>
>> ['PHYSICAL FITNESS CONSULTANTS & TRAINERS', 'HEALTH CLUBS & GYMNASIUMS',
>> 'HEALTH CLUBS & GYMNASIUMS', 'RACQUETBALL COURTS PRIVATE', 'GYMNASIUMS',
>> 'HEALTH & FITNESS CLUBS', 'HEALTH & FITNESS CLUBS', 'PERSONAL FITNESS
>> TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'EXERCISE & PHYSICAL FITNESS
>> PROGRAMS', 'FITNESS CENTERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH CLUBS
>> & GYMNASIUMS', 'PERSONAL FITNESS TRAINERS']
>
>
> Should've looked earlier. Their master list of categories
> http://www.usdirectory.com/cat/g0 shows a few commas, a bunch of dashes,
> and the ampersands we talked about.
>
> "OFFICE SERVICES, SUPPLIES & EQUIPMENT" gets removed because of the comma.
>
> "AUTOMOBILE - DEALERS" gets removed because of the dash.
>
> I updated your regex and it seems to have fixed it.
>
> orig: (r"^[A-Z\s&]+$")
> new : (r"^[A-Z\s&,-]+$")
>
>
> Thanks again.
If there is a "master list" compare your candidates against it instead of
using a heuristic, i. e.
categories = set(master_list)
output = [category for category in input if category in categories]
You can find the categories with
>>> import urllib.request
>>> import bs4
>>> soup =
bs4.BeautifulSoup(urllib.request.urlopen("http://www.usdirectory.com/cat/g0";).read())
>>> categories = set()
>>> for li in soup.find_all("li"):
... assert li.parent.parent["class"][0].startswith("category_items")
... categories.add(li.text)
...
>>> print("\n".join(sorted(categories)[:10]))
Accounting & Bookkeeping Services
Adoption Services
Adult Entertainment
Advertising
Agricultural Equipment & Supplies
Agricultural Production
Agricultural Services
Aids Resources
Aircraft Charters & Rentals
Aircraft Dealers & Services
--
https://mail.python.org/mailman/listinfo/python-list
Re: After a year using Node.js, the prodigal son returns
Chris Angelico wrote: https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript This video seems to be broken. It stops about 1/3 of the way through for me and says "No video with supported format and MIME type found". -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: After a year using Node.js, the prodigal son returns
On Fri, May 6, 2016 at 6:51 PM, Gregory Ewing wrote: > Chris Angelico wrote: > >> https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript > > > This video seems to be broken. It stops about 1/3 of the way > through for me and says "No video with supported format and > MIME type found". Ouch. I think the site's gone down. Maybe being posted on python-list slashdotted it? [1] ChrisA [1] Nah, I doubt it. -- https://mail.python.org/mailman/listinfo/python-list
Re: Whittle it on down
On Thu, 05 May 2016 19:31:33 -0400, DFS wrote: > On 5/5/2016 1:39 AM, Stephen Hansen wrote: > >> Given: >> > input = [u'Espa\xf1ol', 'Health & Fitness Clubs (36)', 'Health Clubs > & Gymnasiums (42)', 'Health Fitness Clubs', 'Name', 'Atlanta city > guide', 'edit address', 'Tweet', 'PHYSICAL FITNESS CONSULTANTS & > TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS', > 'www.custombuiltpt.com/', 'RACQUETBALL COURTS PRIVATE', > 'www.lafitness.com', 'GYMNASIUMS', 'HEALTH & FITNESS CLUBS', > 'www.lafitness.com', 'HEALTH & FITNESS CLUBS', 'www.lafitness.com', > 'PERSONAL FITNESS TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'EXERCISE > & PHYSICAL FITNESS PROGRAMS', 'FITNESS CENTERS', 'HEALTH CLUBS & > GYMNASIUMS', 'HEALTH CLUBS & GYMNASIUMS', 'PERSONAL FITNESS > TRAINERS', '5', '4', '3', '2', '1', 'Yellow Pages', 'About Us', > 'Contact Us', 'Support', 'Terms of Use', 'Privacy Policy', > 'Advertise With Us', 'Add/Update Listing', 'Business Profile Login', > 'F.A.Q.'] >> >> Then: >> > pattern = re.compile(r"^[A-Z\s&]+$") > output = [x for x in list if pattern.match(x)] > output > >> ['PHYSICAL FITNESS CONSULTANTS & TRAINERS', 'HEALTH CLUBS & >> GYMNASIUMS', >> 'HEALTH CLUBS & GYMNASIUMS', 'RACQUETBALL COURTS PRIVATE', >> 'GYMNASIUMS', >> 'HEALTH & FITNESS CLUBS', 'HEALTH & FITNESS CLUBS', 'PERSONAL FITNESS >> TRAINERS', 'HEALTH CLUBS & GYMNASIUMS', 'EXERCISE & PHYSICAL FITNESS >> PROGRAMS', 'FITNESS CENTERS', 'HEALTH CLUBS & GYMNASIUMS', 'HEALTH >> CLUBS & GYMNASIUMS', 'PERSONAL FITNESS TRAINERS'] > > > Should've looked earlier. Their master list of categories > http://www.usdirectory.com/cat/g0 shows a few commas, a bunch of dashes, > and the ampersands we talked about. > > "OFFICE SERVICES, SUPPLIES & EQUIPMENT" gets removed because of the > comma. > > "AUTOMOBILE - DEALERS" gets removed because of the dash. > > I updated your regex and it seems to have fixed it. > > orig: (r"^[A-Z\s&]+$") > new : (r"^[A-Z\s&,-]+$") > > > Thanks again. it looks to me like this system is trying to prevent SQL injection attacks by blacklisting certain characters. this is not the correct way to block such attacks & is probably not a good indicator to the quality of the rest of the application. -- When love is gone, there's always justice. And when justice is gone, there's always force. And when force is gone, there's always Mom. Hi, Mom! -- Laurie Anderson -- https://mail.python.org/mailman/listinfo/python-list
Re: Pylint prefers list comprehension over filter...
On Fri, 6 May 2016 12:57 pm, Stephen Hansen wrote: > On Thu, May 5, 2016, at 07:46 PM, Dan Sommers wrote: >> On Thu, 05 May 2016 18:37:11 -0700, Stephen Hansen wrote: >> >> > ''.join(x for x in string if x.isupper()) >> >> > The difference is, both filter and your list comprehension *build a >> > list* which is not needed, and wasteful. The above skips building a >> > list, instead returning a generator ... >> >> filter used to build a list, but now it doesn't (where "used to" means >> Python 2.7 and "now" means Python 3.5; I'm too lazy to track down the >> exact point(s) at which it changed): > > Oh, didn't know that. Then again the OP was converting the output of > filter *into* a list, which wasted a list either way. In Python 2.7, run `from future_builtins import *` to get most of the Python 3 behaviour: py> filter(None, []) [] py> from future_builtins import * py> filter(None, []) -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: python, ctypes and GetIconInfo issue
Hello eryk sun,
first of all thank you very much, really appreciate your help.
Please be informed that I'm a python beginner, so forgive me if
my following questions sound stupid (also I'm not a native speaker).
> Please avoid windll. It caches the loaded library, which in turn
> caches function pointers. So all packages that use windll.user32 are
> potentially stepping on each others' toes with mutually incompatible
> function prototypes. It also doesn't allow configuring
> use_last_error=True to enable ctypes.get_last_error() for WinAPI
> function calls.
I assume you are referring to this block of code
GetIconInfo = windll.user32.GetIconInfo
GetIconInfo.argtypes = [HICON, POINTER(ICONINFO)]
GetIconInfo.restype= BOOL
GetIconInfo.errcheck = ErrorIfZero
where as you use
user32 = ctypes.WinDLL('user32', use_last_error=True)
user32.GetIconInfoExW.errcheck = check_bool
user32.GetIconInfoExW.restype = wintypes.BOOL
user32.GetIconInfoExW.argtypes = (
wintypes.HICON, # _In_ hIcon
PICONINFOEX,) # _Out_ piconinfoex
I've checked ctype docu included in python but don't find any hint about your
concerns.
May I ask you, do you know additional documents/sites which I can use to get a
better understanding
about caching issue? Or did I miss something from used documentation?
> The attribute name is "_fields_", not "__fields__", so you haven't
> actually defined any fields and sizeof(ICONINFO) is 0. When you pass
> this empty struct to GetIconInfo, it potentially overwrites and
> corrupts existing data on the heap that can lead to a crash later on.
Ahhh, typically me. Thank you for pointing to it.
> Here's the setup I created to test GetIconInfo and GetIconInfoEx.
> Maybe you can reuse some of this code, but if you're using XP this
> won't work as written because GetIconInfoEx was added in Vista.
> Note the use of a __del__ finalizer to call DeleteObject on the
> bitmaps. Otherwise, in a real application, calling GetIconInfo would
> leak memory.
Oh, you answered already an upcoming question I guess, thank you ;-)
> Using __del__ is convenient, but note that you can't
> reuse an instance without manually calling DeleteObject on the
> bitmaps.
Don't understand this. Isn't this covered by your example in base class?
class ICONINFO_BASE(ctypes.Structure):
def __del__(self, gdi32=gdi32):
if self.hbmMask:
gdi32.DeleteObject(self.hbmMask)
self.hbmMask = None
if self.hbmColor:
gdi32.DeleteObject(self.hbmColor)
self.hbmColor = None
If I would do somthing like
iconinfoex = ICONINFOEX()
iconinfoex = None
DeleteObject would be called once None gets assigned, wouldn't it?
Again, thank you very much.
Hubert
--
https://mail.python.org/mailman/listinfo/python-list
Re: After a year using Node.js, the prodigal son returns
On 2016-05-06, Chris Angelico wrote: > On Fri, May 6, 2016 at 12:49 PM, Michael Torrie wrote: >> On 05/04/2016 02:59 AM, Steven D'Aprano wrote: >>> A year ago, Gavin Vickery decided to move away from Python and give >>> Javascript with Node.js a try. Twelve months later, he has written about his >>> experiences: >>> >>> >>> http://geekforbrains.com/post/after-a-year-of-nodejs-in-production >> >> Very interesting. Frankly Javascript sounds awful. Even on the front end. > > https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript > > JavaScript is terrible. Really, really bad. And because of that, it > has the potential to sweep the world. If your reasoning is correct, it'll never be able to overtake PHP. I've never written anything over a hundred or two lines in JavaScript, but for small stuff it seems OK -- though as others have noted there are some oddly missing batteries that result in use of a lot of small external libraries for things that any C, PHP, or Python user would have expected to be in the standard library. -- Grant Edwards grant.b.edwardsYow! I'm wearing PAMPERS!! at gmail.com -- https://mail.python.org/mailman/listinfo/python-list
Re: After a year using Node.js, the prodigal son returns
On Fri, May 6, 2016 at 11:45 PM, Grant Edwards wrote: >> JavaScript is terrible. Really, really bad. And because of that, it >> has the potential to sweep the world. > > If your reasoning is correct, it'll never be able to overtake PHP. > > I've never written anything over a hundred or two lines in JavaScript, > but for small stuff it seems OK -- though as others have noted there > are some oddly missing batteries that result in use of a lot of small > external libraries for things that any C, PHP, or Python user would > have expected to be in the standard library. Except that it's pretty easy to switch out PHP for Python, or anything else. JavaScript is what it is because it's hard to just use a different language. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Whittle it on down
On 5/6/2016 3:45 AM, Peter Otten wrote:
DFS wrote:
Should've looked earlier. Their master list of categories
http://www.usdirectory.com/cat/g0 shows a few commas, a bunch of dashes,
and the ampersands we talked about.
"OFFICE SERVICES, SUPPLIES & EQUIPMENT" gets removed because of the comma.
"AUTOMOBILE - DEALERS" gets removed because of the dash.
I updated your regex and it seems to have fixed it.
orig: (r"^[A-Z\s&]+$")
new : (r"^[A-Z\s&,-]+$")
Thanks again.
If there is a "master list" compare your candidates against it instead of
using a heuristic, i. e.
categories = set(master_list)
output = [category for category in input if category in categories]
You can find the categories with
import urllib.request
import bs4
soup =
bs4.BeautifulSoup(urllib.request.urlopen("http://www.usdirectory.com/cat/g0";).read())
categories = set()
for li in soup.find_all("li"):
... assert li.parent.parent["class"][0].startswith("category_items")
... categories.add(li.text)
...
print("\n".join(sorted(categories)[:10]))
"import urllib.request
ImportError: No module named request"
I'm on python 2.7.11
Accounting & Bookkeeping Services
Adoption Services
Adult Entertainment
Advertising
Agricultural Equipment & Supplies
Agricultural Production
Agricultural Services
Aids Resources
Aircraft Charters & Rentals
Aircraft Dealers & Services
Yeah, I actually did something like that last night. Was trying to get
their full tree structure, which goes 4 levels deep: ie
Arts & Entertainment
Newpapers
News Dealers
Prepess Services
What I referred to as their 'master list' is actually just 2 levels
deep. My bad.
So far I haven't come across one that had anything in it but letters,
dashes, commas or ampersands.
Thanks
--
https://mail.python.org/mailman/listinfo/python-list
Slight problems with python in Windows
I'm trying to install Python under Windows 7 so that I can use git-review and have found a few niggling issues. 1) Apparently (according to the git-review pages) pip has a problem with directories with spaces in their names. Python's default installation directory is under Program Files. I agree that this is a pip issue rather than a Python one, but maybe a warning message would help? 2) According to the Programs and Files section of the Windows Control Panel, installing Python also installs something called the Python Launcher. When I try to remove this (so I can reinstall Python in a better directory) is comes up with an error message: Error opening installation log file. Verify that the specified log file location exists and is writable. After reinstalling I now have 2 copies of the launcher I hope it doesn't give me any problems. 3) After uninstalling Python the installation directory is still there with a few files in it (possibly connected with the previous issue). Can I just delete it? Regards, Peter mailto:[email protected] www.ptoye.com -- https://mail.python.org/mailman/listinfo/python-list
Re: Whittle it on down
On 5/6/2016 9:58 AM, DFS wrote:
On 5/6/2016 3:45 AM, Peter Otten wrote:
DFS wrote:
Should've looked earlier. Their master list of categories
http://www.usdirectory.com/cat/g0 shows a few commas, a bunch of dashes,
and the ampersands we talked about.
"OFFICE SERVICES, SUPPLIES & EQUIPMENT" gets removed because of the
comma.
"AUTOMOBILE - DEALERS" gets removed because of the dash.
I updated your regex and it seems to have fixed it.
orig: (r"^[A-Z\s&]+$")
new : (r"^[A-Z\s&,-]+$")
Thanks again.
If there is a "master list" compare your candidates against it instead of
using a heuristic, i. e.
categories = set(master_list)
output = [category for category in input if category in categories]
You can find the categories with
import urllib.request
import bs4
soup =
bs4.BeautifulSoup(urllib.request.urlopen("http://www.usdirectory.com/cat/g0";).read())
categories = set()
for li in soup.find_all("li"):
... assert li.parent.parent["class"][0].startswith("category_items")
... categories.add(li.text)
...
print("\n".join(sorted(categories)[:10]))
"import urllib.request
ImportError: No module named request"
Figured it out using urllib2. Your code returns 411 categories from
that first page.
There are up to 4 levels of categorization:
Level 1: Arts & Entertainment
Level 2: Newspapers
Level 3: Newspaper Brokers
Level 3: Newspaper Dealers Back Number
Level 3: Newspaper Delivery
Level 3: Newspaper Distributors
Level 3: Newsracks
Level 3: Printers Newspapers
Level 3: Newspaper Dealers
Level 3: News Dealers
Level 4: News Dealers Wholesale
Level 4: Shoppers News Publications
Level 3: News Service
Level 4: Newspaper Feature Syndicates
Level 4: Prepress Services
http://www.usdirectory.com/cat/g0 shows 21 Level 1 categories, and 390
Level 2. To get the Level 3 and 4 you have to drill-down using the
hyperlinks.
How to do it in python code is beyond my skills at this point. Get the
hrefs and load them and parse, then get the next level and load them and
parse, etc.?
--
https://mail.python.org/mailman/listinfo/python-list
Re: python, ctypes and GetIconInfo issue
A further question if you don't mind.
In your example you used a base class
and ICONINFO well as ICONINFOEX inherit it.
As the members of ICONINFO are part of ICONINFOEX
couldn't we do something like
class ICONINFO_BASE(ctypes.Structure):
def __del__(self, ):
if self.hbmMask:
gdi32.DeleteObject(self.hbmMask)
self.hbmMask = None
if self.hbmColor:
gdi32.DeleteObject(self.hbmColor)
self.hbmColor = None
class ICONINFO(ICONINFO_BASE):
_fields_ = (('fIcon',wintypes.BOOL),
('xHotspot', wintypes.DWORD),
('yHotspot', wintypes.DWORD),
('hbmMask', wintypes.HBITMAP),
('hbmColor', wintypes.HBITMAP))
class ICONINFOEX(ICONINFO):
_fields_ = (('cbSize',wintypes.DWORD),
# ('fIcon', wintypes.BOOL),
# ('xHotspot', wintypes.DWORD),
# ('yHotspot', wintypes.DWORD),
# ('hbmMask', wintypes.HBITMAP),
# ('hbmColor', wintypes.HBITMAP),
('wResID',wintypes.WORD),
('szModName', wintypes.WCHAR * MAX_PATH),
('szResName', wintypes.WCHAR * MAX_PATH))
def __init__(self, *args, **kwds):
super(ICONINFOEX, self).__init__(*args, **kwds)
self.cbSize = ctypes.sizeof(self)
A dir on instance of that class indicates that it should be possible
as it contains all needed members and size seems to be correct as well,
but a call to GetIconInfoExW fails with:
WindowsError: [Error 87] The parameter is incorrect.
info = ICONINFOEX()
info.cbSize = ctypes.sizeof(info)
print dir(info)
print info.cbSize
user32.GetIconInfoExW(hIcon, ctypes.byref(info))
Thank you
Hubert
--
https://mail.python.org/mailman/listinfo/python-list
Re: python - handling HTTP requests asynchronously
On Thu, May 5, 2016 at 11:56 PM, wrote: > Hi everyone, > I need to generate a PDF report for each entry of a django queryset. > There'll be between between 30k and 40k entries. > > The PDF is generated through an external API. Since currently is generated > on demand, this is handled synchronously via an HTTP request/response. That > will be different for this task, since I think I'll use a django management > command to loop through the queryset and perform the PDF generation. > > Which approach should I follow for this task? I thought about 3 possibile > solutions, although are technologies that I never used: > > 1) Celery: assign a task (http request with a different payload) to a > worker, then retrieve it once it's done. > > 2) request-futures: using requests in a non-blocking way. > > 3) multiprocessing module, with e.g. 10 as workers limit. > > > the goal is to use the API concurrently (e.g. send 10 or 100 http requests > simultaneously, depending on how many concurrent requests the API can > handle). > > Anybody here that handled a similar task and can give advices on how to > proceed on this? > -- > https://mail.python.org/mailman/listinfo/python-list > Have you tried channels: https://github.com/andrewgodwin/channels ? If it's an asyncronous request/response cycle you're looking for, it should work well. Essentially, you have worker processes that receive a message over a websocket connection and then send the new message back to the "group". I would recommend using the redis in memory transaction layer if you go this route as it is the fastest and most efficient. The best part about channels is that you can still run a normal django app alongside it. You can have only one app use websockets while the rest use standard http.. -- https://mail.python.org/mailman/listinfo/python-list
Re: Whittle it on down
DFS wrote: > There are up to 4 levels of categorization: > http://www.usdirectory.com/cat/g0 shows 21 Level 1 categories, and 390 > Level 2. To get the Level 3 and 4 you have to drill-down using the > hyperlinks. > > How to do it in python code is beyond my skills at this point. Get the > hrefs and load them and parse, then get the next level and load them and > parse, etc.? Yes, that should work ;) -- https://mail.python.org/mailman/listinfo/python-list
Re: Slight problems with python in Windows
On 6.5.2016 14:22, Peter Toye wrote: I'm trying to install Python under Windows 7 so that I can use git-review and have found a few niggling issues. 1) Apparently (according to the git-review pages) pip has a problem with directories with spaces in their names. Python's default installation directory is under Program Files. I agree that this is a pip issue rather than a Python one, but maybe a warning message would help? Select install to all users and it should install to root \PythonXY directory where XY is major version number. -- https://mail.python.org/mailman/listinfo/python-list
Re: Slight problems with python in Windows
Hi Peter, On Fri, May 6, 2016 at 6:22 AM, Peter Toye wrote: > I'm trying to install Python under Windows 7 so that I can use git-review and > have found a few niggling issues. > > 1) Apparently (according to the git-review pages) pip has a problem with > directories with spaces in their names. Python's default installation > directory is under Program Files. I agree that this is a pip issue rather > than a Python one, but maybe a warning message would help? I don't believe this is true anymore, I've successfully used pip with 3.5 installed in Program Files, and also just now in a test venv named "test venv". Do note that with installation in Program Files, you get the benefits of the install directory being writable only to administrators, but also the drawbacks: only administrators can use pip to install to the global site-packages. You can use either 'pip --user', or create a venv in a directory writable to you and use it. Also note that you can't use "pip.exe" to upgrade pip itself since it can't overwrite "pip.exe" while it's in use; use 'python -m pip' instead. > 2) According to the Programs and Files section of the Windows Control Panel, > installing Python also installs something called the Python Launcher. When I > try to remove this (so I can reinstall Python in a better directory) is comes > up with an error message: The Python Launcher is a very handy tool called 'py.exe' which makes it much easier to use more than one version of Python on a Windows machine. In an all users install, py.exe is installed to C:\Windows and is thus always available on PATH, so you can invoke Python 3.5 by calling 'py -3.5' without having to adjust your PATH. The error message is odd, though, would you mind trying to reproduce it and opening a bug at bugs.python.org? > Error opening installation log file. Verify that the specified log file > location exists and is writable. > > After reinstalling I now have 2 copies of the launcher I hope it doesn't > give me any problems. It shouldn't. The launcher only installs 2 files, py.exe and pyw.exe (counterpart to pythonw.exe), both in C:\Windows. > 3) After uninstalling Python the installation directory is still there with a > few files in it (possibly connected with the previous issue). Can I just > delete it? Yes, that should be fine. I would guess it's still there due to pip artifacts in Lib\ and Scripts\. Hope this helps, -- Zach -- https://mail.python.org/mailman/listinfo/python-list
Re: After a year using Node.js, the prodigal son returns
On Friday, May 6, 2016 at 8:23:27 AM UTC+5:30, Chris Angelico wrote: > On Fri, May 6, 2016 at 12:49 PM, Michael Torrie wrote: > > On 05/04/2016 02:59 AM, Steven D'Aprano wrote > >> A year ago, Gavin Vickery decided to move away from Python and give > >> Javascript with Node.js a try. Twelve months later, he has written about > >> his > >> experiences: > >> > >> > >> http://geekforbrains.com/post/after-a-year-of-nodejs-in-production > > > > Very interesting. Frankly Javascript sounds awful. Even on the front end. > > https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript > > JavaScript is terrible. Really, really bad. And because of that, it > has the potential to sweep the world. If python community is passionate in hating javascript it may wish to look at webassembly: https://en.wikipedia.org/wiki/WebAssembly https://medium.com/javascript-scene/what-is-webassembly-the-dawn-of-a-new-era-61256ec5a8f6 -- https://mail.python.org/mailman/listinfo/python-list
A fun python CLI program for all to enjoy!
getAddresses.py
Scrapes addresses from www.usdirectory.com and stores them in a SQLite
database, or writes them to text files for mailing labels, etc
Now, just by typing 'fast food Taco Bell 10 db all' you can find
out how many Taco Bells are within 10 miles of you, and store all the
addresses in your own address database.
No more convoluted Googling, or hitting the 'Next Page' button, or
fumbling with the Yellow Pages...
Note: the db structure is flat on purpose, and the .csv files aren't
quote delimited.
Put the program in its own directory. It creates the SQLite database
there, and writes files there, too.
Reviews of code, bug reports, criticisms, suggestions for improvement,
etc are all welcome.
Enjoy!
#getAddresses.py
import os, sys, requests, time, datetime
from lxml import html
import pyodbc, sqlite3, re
#show values of variables, HTML content, etc
#set it to False for short/concise program output
verbose = False
if verbose == True:
print "The verbose setting is turned On."
print ""
#check if address is unique
addrCheck = []
def addrUnique(addr):
if addr not in addrCheck:
x = True
addrCheck.append(addr)
else: x = False
return x
#validate and parse command line
def showHelp():
print ""
print " Enter search word(s), city or zip, state, miles to search, txt
or csv or db, # addresses to save (no commas)"
print ""
print " eg: restaurant Knoxville TN 10 txt 50"
print " search for restaurants within 10 miles of Knoxville TN, and
write"
print " the first 50 address to a txt file"
print ""
print " eg: furniture 30303 GA 20 csv all"
print " search for furniture within 20 miles of zip 30303 GA,"
print " and write all results to a csv file"
print ""
print " eg: boxing gyms Detroit MI 10 db 5"
print " search for boxing gyms within 10 miles of Detroit MI, and
store"
print " the first 5 results in a database"
print ""
print " All entries are case-insensitive (ie TX or tx are acceptable)"
exit(0)
argCnt = len(sys.argv)
if argCnt < 7: showHelp()
if verbose == True:
print ""
print str(argCnt) + " arguments"
keyw = "" #eg restaurant,
boxing gym
if argCnt == 7: keyw = sys.argv[1] #one search word
if argCnt > 7: #multiple search words
for i in range(1,argCnt-5):
keyw = keyw + sys.argv[i] + "+"
keyw = keyw[:-1]#drop trailing + sign
cityzip = sys.argv[argCnt-5] #eg Atlanta or 30339
state= sys.argv[argCnt-4] #eg GA
miles= sys.argv[argCnt-3] #eg 5,10,20,30,50 (website allows max 30)
store= sys.argv[argCnt-2] #write address to file or database
addrWant = sys.argv[argCnt-1] #eg save All or number >0
if addrWant.lower() != "all": #how many addresses to save
if addrWant.isdigit() == False: showHelp()
if addrWant == "0": showHelp()
addrWant = int(addrWant)
elif addrWant.lower() == "all": addrWant = addrWant.lower()
else: addrWant = int(addrWant)
if store != "csv" and store != "txt" and store != "db": showHelp()
#begin timing the code
startTime = time.clock()
#website, SQLite db, search string, current date/time for use with db
datasrc = "www.usdirectory.com"
dbName = "addresses.sqlite"
search = keyw + " " + str(cityzip) + " " + state + " " + str(miles) + "
" + str(addrWant)
loaddt = datetime.datetime.now()
#write addresses to file
#each time the same search is done, the file is deleted and recreated
if store == "csv" or store == "txt":
#csv will write in .csv format - header and 1 line per address
#txt will write out 3 lines per address, then blank before next address
webfile = "usdirectory.com_"+keyw+"_"+cityzip+"_"+state+"."+store
f = open(webfile,"w")
if store == "csv": f.write("Name,Address,CityStateZip\n")
f.close
#store addresses in database
cSQL = ""
if store == "db":
#creates a SQLite database that Access 2003 can't read
#conn = sqlite3.connect(dbName)
#also creates a SQLite database that Access 2003 can't read
conn = pyodbc.connect('Driver={SQLite3 ODBC Driver};Database=' + dbName)
db = conn.cursor()
cSQL = "CREATE TABLE If Not Exists ADDRESSES "
cSQL += "(datasrc, search, category, name, street, city, state, zip,
loaddt, "
cSQL += "PRIMARY KEY (datasrc, search, name, street));"
db.execute(cSQL)
# cSQL = "CREATE TABLE If Not Exists CATEGORIES "
# cSQL += "(catID INTEGER PRIMARY KEY, catDesc);"
# db.execute(cSQL)
# db.execute("CREATE UNIQUE INDEX If Not Exists UIDX_CATDESC ON
CA
Re: A fun python CLI program for all to enjoy!
On 2016-05-06 20:10, DFS wrote:
getAddresses.py
Scrapes addresses from www.usdirectory.com and stores them in a SQLite
database, or writes them to text files for mailing labels, etc
Now, just by typing 'fast food Taco Bell 10 db all' you can find
out how many Taco Bells are within 10 miles of you, and store all the
addresses in your own address database.
No more convoluted Googling, or hitting the 'Next Page' button, or
fumbling with the Yellow Pages...
Note: the db structure is flat on purpose, and the .csv files aren't
quote delimited.
Put the program in its own directory. It creates the SQLite database
there, and writes files there, too.
Reviews of code, bug reports, criticisms, suggestions for improvement,
etc are all welcome.
OK, you asked for it... :-)
1. It's shorter and clearer not to compare with True or False:
if verbose:
and:
if not dupeRow:
2. You can print a blank line with an empty print statement:
print
3. When looking for unique items, a set is a better choice than a list:
addrCheck = set()
def addrUnique(addr):
if addr not in addrCheck:
x = True
addrCheck.add(addr)
else:
x = False
return x
4. Try string formatting instead multiple concatenation:
print "%s arguments" % argCnt
5. Strings have a .join method, and when you combine it with string slicing:
keyw = "+".join(sys.argv[1 : argCnt - 5])
6. Another example of string formatting:
search = "%s %s %s %s %s" % (keyw, cityzip, state, miles, addrWant)
7. It's recommended to use the 'with' statement when handling files:
with open(webfile, "w") as f:
if store == "csv":
f.write("Name,Address,CityStateZip\n")
If you don't want to use the 'with' statement, note that closing the
file is:
f.close()
It needs the "()"!
8. When using SQL, you shouldn't try to insert the values yourself; you
should use parametrised queries:
cSQL = "DELETE FROM addresses WHERE datasrc = ? AND search = ?;"
if verbose:
print cSQL
db.execute(cSQL, (datasrc, search))
conn.commit()
It'll insert the values where the "?" are and will do any necessary
quoting itself. (Actually, some drivers use "?", others use "%s", so if
it doesn't work with one, try the other.)
The way you wrote it, it would fail if a value contained a "'".
It's that kind of thing that leads to SQL injection attacks.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python is an Equal Opportunity Programming Language
On Thursday, May 5, 2016 at 3:00:01 PM UTC-4, Terry Reedy wrote: > https://motherboard.vice.com/blog/python-is-an-equal-opportunity-programming-language > > from an 'Intel(R) Software Evangelist' > -- > Terry Jan Reedy >From the link: MB: What is it about Python that makes it friendly to women? Is it something about the actual language itself? Or is it more of a subcultural thing within the community? DS: One thing that I think causes this is the founder of the Python project, a guy named Guido van Rossum. He's referred to as the "BDFL"--the Benevolent Dictator for Life. The way to think of him is like Linus Torvalds of Linux. Most of his keynote at that conference was answering questions from the people who had attended. And he actually said, "Let's alternate between men and women asking questions."On the second day of the conference, he was wearing a shirt from PyLadies, another nonprofit like Django Girls that helps women learn how to program on Python. * This not "equal opportunity". It is a quota system. It's my impression that in the U.S., Asians are over-represented among programmers relative to their share of the population and that whites and especially blacks are under-represented. Should we impose racial quotas on questions at conferences and call that "equal opportunity" as well? -- https://mail.python.org/mailman/listinfo/python-list
Re: Python is an Equal Opportunity Programming Language
On 05/06/2016 01:35 PM, beliavsky--- via Python-list wrote: Most of [Guido's] keynote at that conference was answering questions from > the people who had attended. And he actually said, "Let's alternate between > men and women asking questions."On the second day of the conference, he was > wearing a shirt from PyLadies, another nonprofit like Django Girls that helps > women learn how to program on Python. * This not "equal opportunity". It is a quota system. It's a corrective action, a way of getting men accustomed to listening to women and hearing good ideas and questions from them, and a way to accustom women to speaking in (currently) male dominated groups. And it is far more equal opportunity than having 25 males ask questions and only one or two females. -- ~Ethan~ -- https://mail.python.org/mailman/listinfo/python-list
Re: Python is an Equal Opportunity Programming Language
On Friday, May 6, 2016 at 5:07:28 PM UTC-4, Ethan Furman wrote: > On 05/06/2016 01:35 PM, beliavsky--- via Python-list wrote: > > > Most of [Guido's] keynote at that conference was answering questions from > > the people who had attended. And he actually said, "Let's alternate > between > > men and women asking questions."On the second day of the conference, > he was > > wearing a shirt from PyLadies, another nonprofit like Django Girls > that helps > > women learn how to program on Python. > > > > * > > > > This not "equal opportunity". It is a quota system. > > It's a corrective action, a way of getting men accustomed to listening > to women and hearing good ideas and questions from them, and a way to > accustom women to speaking in (currently) male dominated groups. It's silly to say that just because a group is over-represented that it "dominates". If a conference has more Asians than whites does that necessarily make it Asian-dominated? > And it is far more equal opportunity than having 25 males ask questions > and only one or two females. Not if there are 25 males with questions and only one or two females with questions. Among the people who have questions, you could choose randomly. You and Terry Reedy misuse the term "equal opportunity". -- https://mail.python.org/mailman/listinfo/python-list
Re: Python is an Equal Opportunity Programming Language
It seems like it would be equal opportunity between sexes. 1:1 opportunity to ask based on apparent sex. It is not equal representation necessarily. On May 6, 2016 5:53 PM, "beliavsky--- via Python-list" < [email protected]> wrote: > On Friday, May 6, 2016 at 5:07:28 PM UTC-4, Ethan Furman wrote: > > On 05/06/2016 01:35 PM, beliavsky--- via Python-list wrote: > > > > > Most of [Guido's] keynote at that conference was answering questions > from > > > the people who had attended. And he actually said, "Let's alternate > > between > > > men and women asking questions."On the second day of the conference, > > he was > > > wearing a shirt from PyLadies, another nonprofit like Django Girls > > that helps > > > women learn how to program on Python. > > > > > > * > > > > > > This not "equal opportunity". It is a quota system. > > > > It's a corrective action, a way of getting men accustomed to listening > > to women and hearing good ideas and questions from them, and a way to > > accustom women to speaking in (currently) male dominated groups. > > It's silly to say that just because a group is over-represented that it > "dominates". If a conference has more Asians than whites does that > necessarily make it Asian-dominated? > > > And it is far more equal opportunity than having 25 males ask questions > > and only one or two females. > > Not if there are 25 males with questions and only one or two females with > questions. Among the people who have questions, you could choose randomly. > You and Terry Reedy misuse the term "equal opportunity". > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Whittle it on down
On 5/6/2016 11:44 AM, Peter Otten wrote: DFS wrote: There are up to 4 levels of categorization: http://www.usdirectory.com/cat/g0 shows 21 Level 1 categories, and 390 Level 2. To get the Level 3 and 4 you have to drill-down using the hyperlinks. How to do it in python code is beyond my skills at this point. Get the hrefs and load them and parse, then get the next level and load them and parse, etc.? Yes, that should work ;) How about you do it, and I'll tell you if you did it right? ha! -- https://mail.python.org/mailman/listinfo/python-list
Re: A fun python CLI program for all to enjoy!
On 5/6/2016 4:30 PM, MRAB wrote:
On 2016-05-06 20:10, DFS wrote:
getAddresses.py
Scrapes addresses from www.usdirectory.com and stores them in a SQLite
database, or writes them to text files for mailing labels, etc
Now, just by typing 'fast food Taco Bell 10 db all' you can find
out how many Taco Bells are within 10 miles of you, and store all the
addresses in your own address database.
No more convoluted Googling, or hitting the 'Next Page' button, or
fumbling with the Yellow Pages...
Note: the db structure is flat on purpose, and the .csv files aren't
quote delimited.
Put the program in its own directory. It creates the SQLite database
there, and writes files there, too.
Reviews of code, bug reports, criticisms, suggestions for improvement,
etc are all welcome.
OK, you asked for it... :-)
1. It's shorter and clearer not to compare with True or False:
if verbose:
and:
if not dupeRow:
Done. It will take some getting used to, though. I like that it's
shorter, but I could do the same in VBA and almost always chose not to.
2. You can print a blank line with an empty print statement:
print
Done. I actually like the way print looks better than print ""
3. When looking for unique items, a set is a better choice than a list:
addrCheck = set()
def addrUnique(addr):
if addr not in addrCheck:
x = True
addrCheck.add(addr)
else:
x = False
return x
Done.
I researched this just now on StackOverflow:
"Sets are significantly faster when it comes to determining if an object
is present in the set"
and
"lists are very nice to sort and have order while sets are nice to use
when you don't want duplicates and don't care about order."
The speed difference won't matter here in my little app, but it's better
to use the right construct for the job.
4. Try string formatting instead multiple concatenation:
print "%s arguments" % argCnt
You're referring to this line:
print str(argCnt) + " arguments"
Is there a real benefit of using string formatting here? (other than
the required str() conversion)
5. Strings have a .join method, and when you combine it with string
slicing:
keyw = "+".join(sys.argv[1 : argCnt - 5])
Slick. Works a treat, and saved 2 lines of code. String handling is
another area in which python shines compared to VB.
6. Another example of string formatting:
search = "%s %s %s %s %s" % (keyw, cityzip, state, miles, addrWant)
Done. It's shorter, and doesn't require the str() conversion I had to
do on several of the items.
If I can remember to use it, it should eliminate these:
"TypeError: cannot concatenate 'str' and 'int' objects"
7. It's recommended to use the 'with' statement when handling files:
with open(webfile, "w") as f:
if store == "csv":
f.write("Name,Address,CityStateZip\n")
Done. I read that using 'with' means Python closes the file even if an
exception occurs. So a definite benefit.
If you don't want to use the 'with' statement, note that closing the
file is:
f.close()
It needs the "()"!
I used close() in 1 place, but close without parens in 2 other places.
So it works either way. Good catch.
(it's moot now: all 'f.open()/f.close()' replaced by 'with open()')
8. When using SQL, you shouldn't try to insert the values yourself; you
should use parametrised queries:
cSQL = "DELETE FROM addresses WHERE datasrc = ? AND search = ?;"
if verbose:
print cSQL
db.execute(cSQL, (datasrc, search))
conn.commit()
It'll insert the values where the "?" are and will do any necessary
quoting itself. (Actually, some drivers use "?", others use "%s", so if
it doesn't work with one, try the other.)
The way you wrote it, it would fail if a value contained a "'". It's
that kind of thing that leads to SQL injection attacks.
Fixed.
You'll notice later on in the code I used the parameterized method for
INSERTS. I hate the look of that method, but it does make dealing with
apostrophes easier, and makes it safer as you say.
Thanks for the code review, RMAB. Good improvements.
--
https://mail.python.org/mailman/listinfo/python-list
Re: A fun python CLI program for all to enjoy!
On 05/06/2016 04:12 PM, DFS wrote: On 5/6/2016 4:30 PM, MRAB wrote: If you don't want to use the 'with' statement, note that closing the file is: f.close() It needs the "()"! I used close() in 1 place, but close without parens in 2 other places. So it works either way. Good catch. No, it doesn't. `f.close` simple returns the close function, it doesn't call it. The "it works" was simply because Python closed the files for you later. Not a big deal in a small program like this, but still a mistake. -- ~Ethan~ -- https://mail.python.org/mailman/listinfo/python-list
Re: python, ctypes and GetIconInfo issue
On Fri, May 6, 2016 at 9:49 AM, wrote:
>
> In your example you used a base class
> and ICONINFO well as ICONINFOEX inherit it.
> As the members of ICONINFO are part of ICONINFOEX
> couldn't we do something like
>
> class ICONINFO_BASE(ctypes.Structure):
> def __del__(self, ):
> if self.hbmMask:
> gdi32.DeleteObject(self.hbmMask)
> self.hbmMask = None
> if self.hbmColor:
> gdi32.DeleteObject(self.hbmColor)
> self.hbmColor = None
>
> class ICONINFO(ICONINFO_BASE):
> _fields_ = (('fIcon',wintypes.BOOL),
> ('xHotspot', wintypes.DWORD),
> ('yHotspot', wintypes.DWORD),
> ('hbmMask', wintypes.HBITMAP),
> ('hbmColor', wintypes.HBITMAP))
>
> class ICONINFOEX(ICONINFO):
> _fields_ = (('cbSize',wintypes.DWORD),
> # ('fIcon', wintypes.BOOL),
> # ('xHotspot', wintypes.DWORD),
> # ('yHotspot', wintypes.DWORD),
> # ('hbmMask', wintypes.HBITMAP),
> # ('hbmColor', wintypes.HBITMAP),
> ('wResID',wintypes.WORD),
> ('szModName', wintypes.WCHAR * MAX_PATH),
> ('szResName', wintypes.WCHAR * MAX_PATH))
In this case, cbSize field will be offset after hbmColor:
>>> ICONINFOEX.hbmColor.offset
24
>>> ICONINFOEX.cbSize.offset
32
A struct subclass appends its fields to the base class fields. In
theory, you can do this in some cases, but in practice I don't
recommend it (see below).
For example, look at SHARE_INFO_0 [1], SHARE_INFO_1 [2], and
SHARE_INFO_2 [3], which are used to query different levels of
information about network shares.
[1]: https://msdn.microsoft.com/en-us/library/bb525402
[2]: https://msdn.microsoft.com/en-us/library/bb525407
[3]: https://msdn.microsoft.com/en-us/library/bb525408
It can help to maintain a consistent type hierarchy, such as in the
following answer that I wrote to list network shares on Windows:
http://stackoverflow.com/a/36848031/205580
When ctypes checks the type of a pointer argument, it first checks
whether its _type_ is a subclass of the _type_ of the corresponding
pointer type in argtypes. If not, it falls back on checking whether
the pointer argument itself is an instance of the argtypes pointer
type. Similarly, for a byref() argument it checks whether the referent
is an instance of the _type_ of the argtypes pointer type. Maintaining
a consistent type hierarchy provides type safety without having to
tediously cast pointers.
However, I don't recommend subclassing to append _fields_ because it
has a bug. The code that updates the StgDictObject (i.e. the subclass
of dict used by ctypes types for FFI storgage info) for structs and
union types doesn't doesn't properly initialize the ffi_type elements
array. The length field of the stgdict needs to be the total number of
fields, inclusive of all base classes, in order to copy the entire
ffi_type elements array from the base class. However, storing the
total length in the stgdict's length field would require rewriting the
way an instance of a struct is recursively initialized over the base
classes.
This bug affects passing structs by value. This isn't common (passing
unions by value isn't even supported), so I haven't bothered to submit
a patch for this. Here's an example crash on 64-bit Linux:
test.c:
#include
typedef struct _data_t {
int x, y, z;
} data_t;
int test(data_t d) {
printf("%d, %d, %d\n", d.x, d.y, d.z);
return 0;
}
test.py:
from ctypes import *
lib = CDLL('./test.so')
class A(Structure):
_fields_ = (('a', c_int), ('b', c_int), ('c', c_int),)
class B(Structure):
_fields_ = (('a', c_int),)
class C(B):
_fields_ = (('b', c_int),)
class D(C):
_fields_ = (('c', c_int),)
print('test A')
lib.test(A(42, 84, 168))
print('test D')
lib.test(D(42, 84, 168))
output:
test A
42, 84, 168
test D
Aborted
--
https://mail.python.org/mailman/listinfo/python-list
Re: python, ctypes and GetIconInfo issue
On Fri, May 6, 2016 at 8:36 AM, wrote:
>
>> Please avoid windll. It caches the loaded library, which in turn
>> caches function pointers. So all packages that use windll.user32 are
>> potentially stepping on each others' toes with mutually incompatible
>> function prototypes. It also doesn't allow configuring
>> use_last_error=True to enable ctypes.get_last_error() for WinAPI
>> function calls.
> I assume you are referring to this block of code
>
> GetIconInfo = windll.user32.GetIconInfo
> GetIconInfo.argtypes = [HICON, POINTER(ICONINFO)]
> GetIconInfo.restype= BOOL
> GetIconInfo.errcheck = ErrorIfZero
>
> where as you use
>
> user32 = ctypes.WinDLL('user32', use_last_error=True)
> user32.GetIconInfoExW.errcheck = check_bool
> user32.GetIconInfoExW.restype = wintypes.BOOL
> user32.GetIconInfoExW.argtypes = (
> wintypes.HICON, # _In_ hIcon
> PICONINFOEX,) # _Out_ piconinfoex
>
> I've checked ctype docu included in python but don't find any hint about your
> concerns.
> May I ask you, do you know additional documents/sites which I can use to get
> a better
> understanding about caching issue? Or did I miss something from used
> documentation?
You haven't missed anything in the documentation. The ctypes docs need
work, and some of the examples are bad, if not wrong. For example, the
GetModuleHandleA examples incorrectly handle the pointer result
because they were never updated for 64-bit Windows. One can't use a
Python function as the restype with a C function that returns a
pointer because it will be truncated to a C int. Whoever wrote the
GetModuleHandleA examples either doesn't know how this feature is
implemented in ctypes (probably not, since I think Thomas Heller wrote
the example), or doesn't know that a Windows HMODULE is a pointer to
the module's base address, or was just writing sloppy code in the era
of 32-bit Windows.
In this case, look at the CDLL [1] and LibraryLoader [2] classes. Note
how CDLL.__getattr__ caches function pointers using setattr(self,
name, func). Note how LibraryLoader.__getattr__ caches libraries
using setattr(self, name, dll), and how it instantiates the library
using self._dlltype(name), with no way to specify use_last_error=True.
This has caused real problems for projects such as colorama (fixed)
and pyreadline (still broken), and I've seen potential problems in
several other projects that naively copy the cdll and windll examples
from the docs. It's not their fault. The docs are just bad on this
subject.
[1]: https://hg.python.org/cpython/file/v3.5.1/Lib/ctypes/__init__.py#l314
[2]: https://hg.python.org/cpython/file/v3.5.1/Lib/ctypes/__init__.py#l410
>> Using __del__ is convenient, but note that you can't
>> reuse an instance without manually calling DeleteObject on the
>> bitmaps.
>
> Don't understand this. Isn't this covered by your example in base class?
I'm talking about reusing an instance, to avoid the cost of repeated
allocation and deallocation. For example:
info = ICONINFOEX()
for hIcon in hIcons:
user32.GetIconInfoExW(hIcon, ctypes.byref(info))
print('fIcon: %d' % info.fIcon)
print('wResID : %d' % info.wResID)
print('szModName: %s' % info.szModName)
gdi32.DeleteObject(info.hbmMask)
gdi32.DeleteObject(info.hbmColor)
--
https://mail.python.org/mailman/listinfo/python-list
Re: A fun python CLI program for all to enjoy!
On 5/6/2016 7:29 PM, Ethan Furman wrote: On 05/06/2016 04:12 PM, DFS wrote: On 5/6/2016 4:30 PM, MRAB wrote: If you don't want to use the 'with' statement, note that closing the file is: f.close() It needs the "()"! I used close() in 1 place, but close without parens in 2 other places. So it works either way. Good catch. No, it doesn't. `f.close` simple returns the close function, it doesn't call it. The "it works" was simply because Python closed the files for you later. Not a big deal in a small program like this, but still a mistake. Yes. Check out the answer by 'unutbu' here: http://stackoverflow.com/questions/1832528/is-close-necessary-when-using-iterator-on-a-python-file-object He says "I...checked /proc/PID/fd for when the file descriptor was closed. It appears that when you break out of the for loop, the file is closed for you." Improper f.close didn't seem to affect any of the files my program wrote - and I checked a lot of them when I was writing the code. Maybe it worked because the last time the file was written to was in a for loop, so I got lucky and the files weren't truncated? Don't know. Did you notice any other gotchas in the program? -- https://mail.python.org/mailman/listinfo/python-list
Re: A fun python CLI program for all to enjoy!
On 2016-05-07 00:58, DFS wrote: On 5/6/2016 7:29 PM, Ethan Furman wrote: On 05/06/2016 04:12 PM, DFS wrote: On 5/6/2016 4:30 PM, MRAB wrote: If you don't want to use the 'with' statement, note that closing the file is: f.close() It needs the "()"! I used close() in 1 place, but close without parens in 2 other places. So it works either way. Good catch. No, it doesn't. `f.close` simple returns the close function, it doesn't call it. The "it works" was simply because Python closed the files for you later. Not a big deal in a small program like this, but still a mistake. Yes. Check out the answer by 'unutbu' here: http://stackoverflow.com/questions/1832528/is-close-necessary-when-using-iterator-on-a-python-file-object He says "I...checked /proc/PID/fd for when the file descriptor was closed. It appears that when you break out of the for loop, the file is closed for you." If you read the comments for that answer, you'll find the explanation. Improper f.close didn't seem to affect any of the files my program wrote - and I checked a lot of them when I was writing the code. Maybe it worked because the last time the file was written to was in a for loop, so I got lucky and the files weren't truncated? Don't know. Did you notice any other gotchas in the program? -- https://mail.python.org/mailman/listinfo/python-list
Re: Python is an Equal Opportunity Programming Language
On Sat, 7 May 2016 06:35 am, [email protected] wrote: > This not "equal opportunity". It is a quota system. I must ask, what do you think the phrase "quota system" means? Who is setting and enforcing this quota, and given that only about 1 in 20 Python programmers is a woman, do you think men are seriously missing out on any opportunities? > It's my > impression that in the U.S., Asians are over-represented among programmers > relative to their share of the population and that whites and especially > blacks are under-represented. Should we impose racial quotas on questions > at conferences and call that "equal opportunity" as well? I don't know. Are there systematic social forces that discourage whites or blacks from taking up programming? With an AOL email address, you're probably in the USA, and with an email username like "beliavsky" I'm guessing you're probably Chinese. Nah just kidding, you're probably of Eastern European or Russian ancestry, and probably very white indeed. - Do you feel systematically excluded and biased against because of your skin colour? - Do white-fellas like yourself find yourself repeatedly missing out on opportunities because employers and managers bypass you as soon as they realise you are white? - When you do manage to find a job, do you feel that employers and managers consistently hold you to a higher standard than your Asian colleagues, expecting you to work twice as hard to get half the recognition? - Do you get patronised by your colleagues because you're just a whitey? - Do you find that there is a systematic and repeating assumption that white-fellas like you can't program? Do people review your code with "It's not bad, for a whitey"? - Do you find that even when you are on an hourly rate, not a salary, you consistently get offered lower pay for the same work as your Asian colleagues? - During staff meetings and conferences, do you find that your Asian colleagues form cliques that exclude you, preventing you from establishing the sort of networks that a professional needs? If you can answer "Yes" to four or more of those questions, then perhaps there is a case for something to combat the overwhelming anti-white racism that you're suffering from. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Python is an Equal Opportunity Programming Language
On Sat, May 7, 2016 at 1:33 PM, Steven D'Aprano wrote: > On Sat, 7 May 2016 06:35 am, [email protected] wrote: > >> This not "equal opportunity". It is a quota system. > > I must ask, what do you think the phrase "quota system" means? > > Who is setting and enforcing this quota, and given that only about 1 in 20 > Python programmers is a woman, do you think men are seriously missing out > on any opportunities? > The problem with quotas isn't "women don't deserve to be heard" (because they most assuredly do!), but that a restriction can sometimes force awkwardnesses that weren't there to start with. It's unlikely to be an issue at PyCon, but the same problem has come up in other contexts. A great summary comes from the TV show "Yes, Minister" [1], in which the eponymous Minster wishes to promote a woman, and aiming for 25% women in senior positions (a quota, exactly on par with "alternating questions from men and women"). In that case, the "quota-promoted" woman objected, specifically because she didn't want to be part of some 25%, she wanted to go somewhere that would respect her for her accomplishments. So it's possible to disagree with the quota system without disagreeing with the goal it's trying to accomplish (or, conversely, without agreeing with the imbalance that it's trying to address). It's a sensitive matter that has to be handled carefully. In the case of PyCon questions, I fully agree with it; there were enough women present that it wasn't a ridiculous suggestion, and it encourages people to speak up who might otherwise have kept quiet. But just because that worked well, it doesn't mean we should automatically enact quotas everywhere, as some sort of "gender/race/culture imbalance panacea", because it isn't. ChrisA [1] https://en.wikipedia.org/wiki/Equal_Opportunities_(Yes_Minister) -- https://mail.python.org/mailman/listinfo/python-list
Re: Python is an Equal Opportunity Programming Language
On Sat, May 7, 2016 at 12:04 AM, Chris Angelico wrote: > > > In the case of PyCon questions, I fully agree with it; there were > enough women present that it wasn't a ridiculous suggestion, and it > encourages people to speak up who might otherwise have kept quiet. But > just because that worked well, it doesn't mean we should automatically > enact quotas everywhere, as some sort of "gender/race/culture > imbalance panacea", because it isn't. > I think it is a good call to ask if non-male attendees would be interested in asking question. I didn't attend those PyCons so I don't know how many male and female attendees lined up awaiting to ask Guido questions. If there were 25 male and 1 female standing in the line, while I do admire Guido (or just about anybody) encouraging more non-male to speak, is it worth asking whether we place pressure on the females attendees if were to say "hey look, we got a lot of male attendees asking, please more female attendees." I totally understand there is a long history of females being treated as inferior (even in America here!), but too much encouragement or too eager to seek more females speaking is almost like saying females are shy and can't speak up without the presence of a heroic voice. I am a male and I am Asian so I am usually regarded as majority in the tech world so I don't always feel underrepresented and can be biased here. Recently I went to some company's website and on the career page I found a banner photo full of white males and maybe 3-4 females in the pictures, holding beers having a great smile posing for a group picture. It could be really genuine, but I felt so uncomfortable immediately because (1) the ratio of male:female is so out balanced, (2) I felt the company was selling the "equal opportunity" sloan too hard. What I am saying is don't try so hard, people will apply job if they want the job, regardless of gender and ethnicity. Similarly, if females attendees want to ask questions, they will. We shouldn't broadcast every single time "we gotta have more females speaking, or more underrepresented people speaking." When I am hanging out with my friends, whether they are male or female, I don't really think of he/she. I think of them as friends, as human being, no need to differentiate whether they are Mexican or Black or Asian. Just human being. Sexual assault laws in some countries are pretty stupid in the sense that female sexual assault offender would receive light punishment compared to female offender. While social and history would justify such law (because again, males historically dominated women), we still treat people inferior by gender and ethnicity. I don't know, this is a sensitive issue. People are either coerced to believe in one kind of response, or perceive as anti-X if given a different kind of response. Thanks. John -- https://mail.python.org/mailman/listinfo/python-list
Re: A fun python CLI program for all to enjoy!
On Fri, May 6, 2016, at 04:58 PM, DFS wrote:
> Improper f.close didn't seem to affect any of the files my program wrote
> - and I checked a lot of them when I was writing the code.
To be clear, its not an "improper" f.close. That command is simply not
closing the file. Period. "f.close" is how you get the 'close' function
from the 'f' object, and then... you do nothing with it.
If you removed "f.close" entirely, you'd get the exact same behavior as
you have now. The "f.close" does nothing.
That said, in CPython semantics, closing a file explicitly is often not
required. CPython is reference-counted. Once the references to an object
reaches 0, CPython deletes the object. This is an implementation detail
of the CPython and not a guarantee of the Python language itself, which
is why explicit close calls are preferred.
So while 'f.close' does nothing, CPython might be closing the file
*anyways*, and it might work... but that 'might' is hard to reason about
without a deeper understanding, so using explicit closing mechanics
(either via f.close() or with or something else) is strongly
recommended.
For example, if you were to do:
for item in sequence:
f = open(item, 'wb')
f.write("blah")
It probably works fine. The first time through, 'f' is bound to a file
object, and you write to it. The second time through, 'f' is bound to a
*new file object*, and the original file object now has 0 references, so
is automatically deleted.
The last sequence through, f is not closed: the 'for loop' is not a
scope which deletes its internal name bindings when its done. So that
'f' will likely remain open until the very end of the current function,
which may be an issue for you.
Implicit closing actually works in a large number of situations in
CPython, but it isn't a good thing to rely on. It only works in simple
operations where you aren't accidentally storing a reference somewhere
else. You have to keep track of the references in your head to make sure
things will get closed at proper times.
The 'with' statement clearly defines when resources should be closed, so
its preferred (As I see you've adopted from other responses). But its
also needed in other Python implementations which might not follow
CPython's reference counting scheme.
I'm not giving further feedback because MRAB caught everything I thought
was an issue.
--
Stephen Hansen
m e @ i x o k a i . i o
--
https://mail.python.org/mailman/listinfo/python-list
Re: A fun python CLI program for all to enjoy!
DFS wrote: Maybe it worked because the last time the file was written to was in a for loop, so I got lucky and the files weren't truncated? Don't know. It "works" because CPython disposes of objects as soon as they are not referenced anywhere. Other implementations of Python (e.g. Jython, PyPy) might not do that. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Why do these statements evaluate the way they do?
I'm trying to figure out why the following statements evaluate the way they do and I'm not grasping it for some reason. I'm hoping someone can help me. 40+2 is 42 #evaluates to True But 2**32 is 2**32 #evaluates to False This is an example taken from a Microsoft blog on the topic. They say the reason is because the return is based on identity and not value but, to me, these statements are fairly equal. Can someone clue me in? Anthony -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python is an Equal Opportunity Programming Language
Steven D'Aprano wrote: Who is setting and enforcing this quota, and given that only about 1 in 20 Python programmers is a woman, do you think men are seriously missing out on any opportunities? Suppose there are 100 people wanting to ask questions, and there is only time to answer 10 questions. If the 1 in 20 ratio holds, then 5 of those people are women and the other 95 are men. Alternating between men and women means that all of the women get their questions answered, and only 5/95 of the men. So in this example, if you're a woman you have a 100% chance of getting answered, and if you're a man you only have a 5.26% chance. Whether you think this is a good strategy or not, beliavsky is right that it's not "equal". -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Why do these statements evaluate the way they do?
On Sat, May 7, 2016 at 4:36 PM, Anthony Papillion
wrote:
> I'm trying to figure out why the following statements evaluate the way they
> do and I'm not grasping it for some reason. I'm hoping someone can help me.
>
> 40+2 is 42 #evaluates to True
> But
> 2**32 is 2**32 #evaluates to False
>
> This is an example taken from a Microsoft blog on the topic. They say the
> reason is because the return is based on identity and not value but, to me,
> these statements are fairly equal.
Frankly, you shouldn't care. Integers and strings should always be
compared for equality ("=="), not identity ("is"). Everything else is
an interpreter optimization; what you'll find is that some small
integers are cached deep within CPython, so the integer 7 is always
the same object. The exact set that's cached varies from version to
version of CPython, and other interpreters mightn't do that at all -
or might not even have actual in-memory objects for *any* integers in
the machine word range, using their values directly. Or anything at
all.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
