Draw rectangle on a Window DC
Hi： I want to Draw rectangle on Dc when gived a position. Can you teach me? Let me view your code? Thanks. -- http://mail.python.org/mailman/listinfo/python-list
sed to python: replace Q
For some reason I'm unable to grok Python's string.replace() function.
Just trying to parse a simple IP address, wrapped in square brackets,
from Postfix logs. In sed this is straightforward given:
line = "date process text [ip] more text"
sed -e 's/^.*\[//' -e 's/].*$//'
yet the following Python code does nothing:
line = line.replace('^.*\[', '', 1)
line = line.replace('].*$', '')
Is there a decent description of string.replace() somewhere?
Raymond
--
http://mail.python.org/mailman/listinfo/python-list
Re: sed to python: replace Q
>Another approach is to use the split() function in "re" module. Ah ha, thar's the disconnect. Thanks for all the pointers, my def is now working. Still don't understand the logic behind this design though. I mean why would any programming language have separate search or find functions, one for regex and and another for non-regex based pattern matching? Aren't sed, awk, grep, and perl the reference implementations of search and replace? They don't have non-regex functions, why does Python? Wouldn't it be a lot simpler to use a flag, like grep's '-f', to change the meaning of a search string to be literal? My other gripe is with the kludgy object-oriented regex functions. Couldn't these be better implemented in-line? Why should I, as a coder, have to 're.compile()' when all the reference languages do this at compile time, from a much more straightforward and easy to read in-line function... Raymon -- http://mail.python.org/mailman/listinfo/python-list
Re: New Science Discovery: Perl Detractors Remain Idiots After A Decade
Shmuel (Seymour J.) Metz writes: > In , on 03/12/2012 >at 11:27 AM, Albert van der Horst said: > >>You're confused. > > No, s/h/it is just an acephalic troll with delusions of adequacy. Another way to put it is to say that Xah is a legend in his own mind. -- http://mail.python.org/mailman/listinfo/python-list
Re: Standard Deviation One-liner
On Jun 3, 10:55 am, Billy Mays wrote: > I'm trying to shorten a one-liner I have for calculating the standard > deviation of a list of numbers. I have something so far, but I was > wondering if it could be made any shorter (without imports). > > Here's my function: > > a=lambda d:(sum((x-1.*sum(d)/len(d))**2 for x in d)/(1.*(len(d)-1)))**.5 > > The functions is invoked as follows: > > >>> a([1,2,3,4]) > 1.2909944487358056 Besides trying to do it one line, it is also interesting to write an one-pass version with incremental results: http://mathcentral.uregina.ca/QQ/database/QQ.09.06/h/murtaza2.html Another interesting avenue to is aim for highest possible accuracy. Consider using math.fsum() to avoid rounding errors in the summation of large numbers of nearly equal values. Raymond - follow my python tips on twitter: @raymondh -- http://mail.python.org/mailman/listinfo/python-list
Bloom Filter in 22 lines of Python (updated)
Thanks for all the feedback on the earlier post. I've updated the recipe to use a cleaner API, simpler code, more easily subclassable, and with optional optimizations for better cache utilization and speed: http://code.activestate.com/recipes/577684-bloom-filter/ Raymond -- follow my python tips and recipes on twitter: @raymondh -- http://mail.python.org/mailman/listinfo/python-list
Re: Bloom Filter in 22 lines of Python (updated)
On Jun 6, 10:47 am, geremy condra wrote: > On Fri, Jun 3, 2011 at 1:17 PM, Raymond Hettinger wrote: > > Thanks for all the feedback on the earlier post. > > > I've updated the recipe to use a cleaner API, simpler code, > > more easily subclassable, and with optional optimizations > > for better cache utilization and speed: > > > http://code.activestate.com/recipes/577684-bloom-filter/ > > Any chance of this landing in collections? I would be happy to add something like this to collections once the API matures. Right now, the constructor expects the user to know the desired memory size and number of probes. Those could be computed from the maximum number of unique keys and the tolerable error false positive rate. The most convenient constructor could be: b = BloomFilter(myseq) where len(myseq) is presumed indicate the maximum number of unique entries and there is a default tolerable false positive rate of 1 in 10,000. The API already lets the user substitute different methods for get_probes(). It should also allow the bytearray to be easily replaced by other objects with indexable access (array objects, mmap objects, etc). We could also provide union, intersection, and difference operations but these would be a little awkward for general purpose use because the result is only meaningful when the sizes, probe functions, and input domain are all the same. Fortunately, there is a lot of prior art, so the API design can be informed by what has worked well for others. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there any advantage or disadvantage to using sets over list comps to ensure a list of unique entries?
On Jun 20, 9:43 pm, deathweaselx86 wrote:
> Howdy guys, I am new.
>
> I've been converting lists to sets, then back to lists again to get
> unique lists.
> e.g
>
> Python 2.5.2 (r252:60911, Jan 20 2010, 21:48:48)
> [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.>>> foo
> = ['1','2','3']
> >>> bar = ['2','5']
> >>> foo.extend(bar)
> >>> foo = list(set(foo))
That works great and is very clear.
If you want to also preserve order, use an OrderedDict:
>>> list(OrderedDict.fromkeys('abracadabra'))
['a', 'b', 'r', 'c', 'd']
Raymond
--
http://mail.python.org/mailman/listinfo/python-list
Re: Dictionaries and incrementing keys
On Jun 14, 12:57 pm, Steve Crook wrote: > Today I spotted an alternative: > > dict[key] = dict.get(key, 0) + 1 > > Whilst certainly more compact, I'd be interested in views on how > pythonesque this method is. It is very pythonesque in the it was the traditional one way to do it (also one of the fastest ways). Now we have collections.Counter which simplifies the code to: c = Counter() ... c[key] += 1 For existing keys, it is as fast as a regular dictionary (because it is a dict subclass and it does not override or extend either __getitem__ or __setitem__). For new keys, it is a little slower because it calls the __missing__ method which returns zero. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: blist question
On Jul 7, 5:08 am, dmitrey wrote: > hi all, > I feel lack of native Python lists operations (e.g. taking N greatest > elements with the involved key function and O(n) speed) Take a look at heapq.nlargest()... > and > occasionally found blisthttp://pypi.python.org/pypi/blist/ > Its entry says it is better than Python list. It should say: better in some respects and worse in others Do you honestly think that python's list implementation as a simple array of pointers can be beaten in every category of performance? > Did anyone ensure? > Will it ever be merged into Python source code? It was rejected as a replacement for the existing list implementation. There is some chance it will get accepted into the collections module someday. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: a little parsing challenge ☺
On Jul 17, 12:47 am, Xah Lee wrote: > i hope you'll participate. Just post solution here. Thanks. http://pastebin.com/7hU20NNL Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: a little parsing challenge ☺
On Jul 17, 7:15 am, Thomas 'PointedEars' Lahn wrote: > Did you notice the excessive crosspost? Please do not feed the troll. IMO, this was a legitimate cross post since it is for a multi-language programming challenge and everyone can learn from comparing the results. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: a little parsing challenge ☺
On Jul 17, 8:49 am, Thomas Boell wrote: > But why do you enumerate with start=1? Shouldn't you start with index 0? The problem specification says that the the char number should match the emacs goto-char function which is indexed from one, not from zero. This is testable by taking the output of the program and running it through emacs to see that the cursor gets moved exactly to the location of the mismatched delimiter. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: list(), tuple() should not place at "Built-in functions" in documentation
On Jul 14, 6:21 pm, Inside wrote: > As telling in the subject,because "list" and "tuple" aren't functions,they > are types.Is that right? list() and tuple() are in the right place in the documentation because they would be harder to find if listed elsewhere. Tools like str(), int(), list(), tuple() etc tend to be used like functions, so the current location in the docs is where they have been for years. A simple fact of documentation that is that tools don't always fall cleanly into distinct categories. Accordingly, the Library Reference includes str,int,list,tuple, etc in both Section 2 for builtin functions and Section 5 for builtin types. 'nuff said, Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Mar 26, 4:39 am, Mark Dickinson wrote: > On Mar 25, 2:00 pm, Stefan Behnel wrote: > > > > > > > > > > > Westley Martínez, 25.03.2011 14:39: > > > > On Fri, 2011-03-25 at 07:11 +0100, Stefan Behnel wrote: > > >> Steven D'Aprano, 25.03.2011 06:46: > > >>> On Thu, 24 Mar 2011 18:32:11 -0700, Carl Banks wrote: > > > >>>> It's probably the least justified builtin other than pow. > > > >>> I don't know about that. Correctly, efficiently and *quickly* > > >>> implementing the three-argument version of pow is exactly the sort of > > >>> thing that should be in the built-ins, or at least the standard library. > > > >> I think that touches it already. We have a "math" module, so why is > > >> there a > > >> *builtin* for doing special math? How much code is there really that only > > >> uses pow() and does not import "math"? > > > > pow() and math.pow() are different. > > > I don't find that a good excuse for pow() being a builtin. Why not just > > rename it to "math.powmod" instead? > > +1 (modulo bikeshedding about the name). I've long thought that three- > argument pow belongs in the standard library rather than the core. > And then the existence of the ** operator obviates the need for two- > argument pow. IMO, there is far too much willingness to churn long-standing APIs for nearly zero benefit. In my decade as a core Python developer, I've come view deprecations as things that cause unnecessary pain for users and is partially to blame for so many organizations staying with older, unmaintained versions of Python. When it comes to moving pow() to another module or changing the signature for the __pow__ magic method, there's not much of a win to be had, but it will affect tons of existing code (including examples in printed books where pow() is commonly used in examples). I think we (as a group) need to develop a strong aversion to breaking existing APIs and instead focus on making existing APIs more robust or giving them useful extensions. The Python language is not like a personal library where changing APIs has limited (and predictable costs). Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Understanding the Pystone measurement
On Mar 24, 9:21 pm, "[email protected]" wrote: > Hi there, > > Is there a good writeup of what the pystone measurement actually > means? I'm working on benchmarking of some Python code at work, and > I'm interested in how Pystone might be relevant to me. I've tried > googling, but I can't find any introductory / definitional > descriptions of what this module/measurement means. Pystone provides an estimate of the relative performance of different pieces of hardware running single threaded Python apps. If your old laptop scores 12000 pystones and your new shiny macbook scores 84000, then you can expect your non-i/o bound single threaded python apps to run about seven times faster on the new machine. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Understanding the Pystone measurement
I forgot to mention that PyStone is just a Python translation from C of the venerable Dhrystone benchmark. See http://en.wikipedia.org/wiki/Dhrystone for a short write-up on the history, purposes, and limitations of the benchmark. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: FW: ciao
On Mar 26, 11:34 am, MRAB wrote: > On 26/03/2011 18:07, bledar seferi wrote: > > > 3.Scrivere unafunsioncheprende comeargomentouna lista > > diinterierestituisce uninsieme contenentequei numerichesono2 o più > > voltenellalista fornita.Per esempio,seservecomelista di > > input=[1,2,3,3,4,4,5,6,7,7,7],quindila funzionerestituiràilset([3,4,7]) > > > Mi puoi aiutare a fare questo programma a python > > > Grazie > > I'd use a defaultdict to count them. Or use itertools.groupby() to group consecutive values, count the length of each group, and emit the value whenever the count is more than one: >>> s = [1,2,3,3,4,4,5,6,7,7,7] >>> [k for k, g in groupby(s) if len(list(g)) > 1] [3, 4, 7] Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Mar 28, 8:43 am, Steven D'Aprano wrote: > Thank you for spending the time to get some hard data, but I can't > replicate your results since you haven't shown your code. Rather than > attempt to guess what you did and duplicate it, I instead came up with my > own timing measurements. Results are shown here, my code follows below: > > [steve@sylar ~]$ python2.7 sort_test.py > Comparison function: 12.1256039143 > Key function: 3.51603388786 > Double sort: 2.33165812492 > cmp_to_key: 28.1129128933 > > By far the best result comes from two calls to sort. Not far off is the > key function solution. (Admittedly, coming up with a key function for non- > numeric data would be challenging.) The comparison function, and the > cmp_to_key solution, are clearly *much* slower. On Mar 28, 8:43 am, Steven D'Aprano wrote: > Thank you for spending the time to get some hard data, but I can't > replicate your results since you haven't shown your code. Rather than > attempt to guess what you did and duplicate it, I instead came up with my > own timing measurements. Results are shown here, my code follows below: > > [steve@sylar ~]$ python2.7 sort_test.py > Comparison function: 12.1256039143 > Key function: 3.51603388786 > Double sort: 2.33165812492 > cmp_to_key: 28.1129128933 > > By far the best result comes from two calls to sort. Not far off is the > key function solution. (Admittedly, coming up with a key function for non- > numeric data would be challenging.) The comparison function, and the > cmp_to_key solution, are clearly *much* slower. There is less here than meets the eye. The cmp-function dispatch in cmp_to_key() is written in pure Python. Essentially, the timings are showing the already well-known fact that call forwarding is faster in C than in pure python. Each of the O(n log n) comparisons is run through pure Python code that like this: def __lt__(self, other): # mycmp is nested scope variable pointing to # the user-defined cmp-function return mycmp(self.obj, other.obj) < 0 I intend to add a C version of cmp_to_key() so that no trips around the eval-loop are needed. It should be about as efficient as bound_method dispatch (very fast), leaving the user-supplied cmp- function as the dominant cost in the handful of cases where the superfast O(n) key-function approach can't be used for some reason. The memory overhead should either stay the same or drop slightly (currently at 14 bytes per element on a 32-bit build and 28 bytes per element on a 64-bit build). Also note that running timings on Py2.7 instead of Py3.2 disadvantages key-functions. In 2.7, key-functions use sortwrapper objects which were removed in 3.2 (saving memory and reducing dispatch time considerably). Also, in Py2.7 cmp logic is called even when a key- function is defined (because key-function logic was wrapped around the already existing sort with cmp-logic) so you pay the price of both (i.e. creating a 2-tuple for cmp arguments on every call). In Py3.x the cmp logic is gone, making the remaining key-function logic even faster. IOW, key-function logic is way faster and more space efficient in 3.2. One of the implications of the last paragraph is that if 2.7 style cmp- logic were reinstated in 3.3, not only would it clutter the API with two-ways-to-do-it, it would also disadvantage make the common case (using key-functions) run much more slowly. In other words, the performance mavens will have shot themselves (and us) in the foot. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Fun python 3.2 one-liner
from collections import Counter
from itertools import product
print('\n'.join('*'*(c//2000) for _,c in sorted(Counter(map(sum,
product(range(6), repeat=8))).items(
almost-normally-yours,
Raymond
--
http://mail.python.org/mailman/listinfo/python-list
Re: Fun python 3.2 one-liner
>>> print('\n'.join('*'*(c//2000) for _,c in sorted(Counter(map(sum,
>>> product(range(6), repeat=8))).items(
*
***
*
**
*
*
*
**
**
***
**
**
*
*
*
**
*
***
*
Re-posting (forgot to attach the output).
Besides the interesting output, other
interesting virtues include very low
memory usage and that the inner-loop
pipeline runs entirely in C.
Raymond
--
http://mail.python.org/mailman/listinfo/python-list
Re: delete namespaces
On Mar 29, 6:14 pm, monkeys paw wrote: > How do i delete a module namespace once it has been imported? . . . > Then i make a modification to banner.py. When i import it again, > the new changes are not reflected. Is there a global variable i can > modify? In Python2.x, you can use the reload() function: Help on built-in function reload in module __builtin__: reload(...) reload(module) -> module Reload the module. The module must have been successfully imported before. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Dictionary Descriptors
On the python-ideas list, someone made a wild proposal to add
descriptors to dictionaries.
None of the respondents seemed to realize that you could (not should,
just could) already implement this using hooks already present in the
language. I'm posting an example here because I thought you all might
find it to be both interesting and educational.
For more details on how it works and how it relates to descriptors,
see http://mail.python.org/pipermail/python-ideas/2011-March/009657.html
Raymond
sample code
class MyDict(object):
def __init__(self, mapping):
self.mapping = mapping
def __getitem__(self, key):
value = self.mapping[key]
if hasattr(value, '__get__'):
print('Invoking descriptor on', key)
return value.__get__(key)
print('Getting', key)
return value
def __setitem__(self, key, value):
self.mapping[key] = value
class Property:
def __init__(self, getter):
self.getter = getter
def __get__(self, key):
return self.getter(key)
if __name__ == '__main__':
md = MyDict({})
md['x'] = 10
md['_y'] = 20
md['y'] = Property(lambda key: md['_'+key])
print(eval('x+y+1', {}, md))
output
Getting x
Invoking descriptor on y
Getting _y
31
--
http://mail.python.org/mailman/listinfo/python-list
Re: Fun python 3.2 one-liner
On Mar 30, 2:19 am, Martin De Kauwe wrote:
> what is the character limit on a one liner :P. Very interesting
> jesting apart, any more?
Sure, here are three one-liners using itertools.groupby() to emulate
some Unix pipelines:
sort letters | uniq # list unique values
sort letters | uniq -c# count unique values
sort letters | uniq -d# find duplicates
>>> from itertools import groupby
>>> [k for k, g in groupby(sorted('abracadabra'))]
['a', 'b', 'c', 'd', 'r']
>>> [(k, len(list(g))) for k, g in groupby(sorted('abracadabra'))]
[('a', 5), ('b', 2), ('c', 1), ('d', 1), ('r', 2)]
>>> [k for k, g in groupby(sorted('abracadabra')) if len(list(g)) > 1]
['a', 'b', 'r']
Raymond
P.S. Of course, there are many ways to do this.
>>> sorted(set('abracadabra'))
['a', 'b', 'c', 'd', 'r']
>>> sorted(Counter('abracadabra').items())
[('a', 5), ('b', 2), ('c', 1), ('d', 1), ('r', 2)]
>>> sorted(k for k,c in Counter('abracadabra').items() if c > 1)
['a', 'b', 'r']
--
http://mail.python.org/mailman/listinfo/python-list
Re: learn python the hard way exercise 42 help
On Mar 30, 6:48 am, neil harper wrote: > http://pastie.org/1735028 > hey guys play is confusing me, i get how next gets the first room, which > is passed when the instance of Game() is created, but how does it get > the next room? It might help show calling patterns if you added print statements to the while loop: def play(self): next = self.start while True: room = getattr(self, next) print "--- Calling the method:", room, "---" next = room() print "--- That method returned:", next, "---" Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: popular programs made in python?
On Mar 29, 7:32 am, Neil Alt wrote: > i mean made with python only, not just a small part of python. BitTorrent was a huge success. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: delete namespaces
[monkeys paw] > > How do i delete a module namespace once it has been imported? . . . > > Then i make a modification to banner.py. When i import it again, > > the new changes are not reflected. [Terry Reedy] > The best thing, if possible, is to restart the program. > If you develop banner.py with adequate tests, you will want to restart > the test anyway, and you should not need to modify much thereafter. This is excellent advice. You're much better-off starting fresh each time. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Directly Executable Files in Python
On Mar 28, 8:37 pm, Jordan Meyer wrote: > Is it possible to make a directly executable (such as .exe on Windows) file > from scripts written in Python? So as to prevent the end-user from having to > download an interpreter to run the program. http://docs.python.org/faq/programming.html#how-can-i-create-a-stand-alone-binary-from-a-python-script Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Why aren't copy and deepcopy in __builtins__?
On Mar 27, 8:29 pm, John Ladasky wrote: > Simple question. I use these functions much more frequently than many > others which are included in __builtins__. I don't know if my > programming needs are atypical, but my experience has led me to wonder > why I have to import these functions. I asked Guido about this once and he said that he didn't consider them to be part of the core. He worried that they would be overused by beginners and they would be a distraction from learning plain, simple Python which doesn't often need either copy() or deepcopy(). AFAICT, he was right. I've seen large projects where deepcopy and copy where not used even once. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: best python games?
On Mar 25, 7:39 pm, sogeking99 wrote: > hey guys, what are some of the best games made in python? free games > really. like pygames stuff. i want to see what python is capable of. > > cant see any good one on pygames site really, though they have nothing > like sort by rating or most downloaded as far as i can tell At Pycon, I saw some impressive looking games written during the PyWeek, Python Game Programming Challenge http://www.pyweek.org/ I think they're fine examples of what Python is capable of. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Alias for an attribute defined in a superclass
On Mar 31, 3:14 pm, Ben Finney wrote: > Howdy all, > > I want to inherit from a class, and define aliases for many of its > attributes. How can I refer to “the attribute that will be available by > name ‘spam’ once this class is defined”? > > class Foo(object): > def spam(self): > pass > > def eggs(self): > pass > > class Bar(Foo): > beans = Foo.spam > mash = Foo.eggs > > Is that the right way to do it? For methods, that will work just fine. For attributes, you will need to make @property accessors that get and set the underlying attribute. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: a better way to invert a list?
[Ian Kelly] > Which is O(n). If that is too verbose, you could also use a dictionary: > > def invert(p): > return dict(map(reversed, enumerate(p))) def inv(p): return dict(zip(p, itertools.count())) Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Fun python 3.2 one-liner
On Apr 5, 6:38 am, Daniel Fetchinson wrote: > >> what is the character limit on a one liner :P. > > > For PEP 8 compliance, 80 characters. :-) > > Yeah, but we don't live in the 80's or 90's anymore and our screens > can support xterms (or let alone IDE widows) much wider than 80 > characters. I'm using 140 for python these days. Seriously, who would > want to limit him/herself to 80 characters in 2011? I wonder how many people will shorten otherwise clear variable names just to get their lines to fit in 80 characters? Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Why is __root checked for in OrderedDict?
On Apr 7, 4:13 am, andrew cooke wrote:
> If you look at the code
> inhttp://hg.python.org/cpython/file/6adbf5f3dafb/Lib/collections/__init...the
> attribute __root is checked for, and only created if missing. Why?
>
> I ask because, from what I understand, the __init__ method will only be
> called when the object is first being created, so __root will always be
> missing.
First of all, three cheers for reading the source code!
A user can call __init__() even after the OrderedDict instance has
already been created. If so, the __root attribute will already exist
and the whole operation becomes equivalent to an update().
You can see the same behavior with regular dictionaries:
>>> d = dict(a=1, b=2)
>>> d.__init__(b=4, c=5)
>>> d
{'a': 1, 'c': 5, 'b': 4}
Raymond
--
http://mail.python.org/mailman/listinfo/python-list
Re: Why is __root checked for in OrderedDict?
On Apr 7, 2:40 pm, andrew cooke wrote: > Is that normal? I mean, OK, it's possible (and yes I forgot it could be > called directly), but is there any usual reason to do so? It's common for subclasses to call their parent's __init__ method, so that should emulate dict as nearly as possible to help avoid surprises. For the most part, client code won't use __init__ directly because update() already provides equivalent functionality (but with a better name). For list.__init__(), the behavior is different. It clears the list, so someone may be using an __init__ call when it isn't practical to use the syntactic equivalents "del a[:]" or "a[:] = []". Outside those cases, I don't it is "normal", but who knows what future programmers will need to do? > I guess what I'm asking is: if I'm writing library code should I be this > careful? For the standard library, it pays to really think out the corner cases because you never know what people are going to rely on. The Liskov substitution principle asks us to make OrderedDict behave as much like dict as possible (so that OD's can be used a drop-in substitute when needed). Also subclassers tend to stress the API in ways that wouldn't be common for the base class. For standard library code, I think all the little things matter: * int(), bool(), list() etc all return a "zero" value when called with no arguments * the ordered dict code internally calls __update() instead of update() so that a subclass can override update() without breaking the constructor * eval(repr(obj)) should round-trip whereever possible * containers should behave reasonable well even if someone makes the container reference itself: a=[]; a.append(a) * classes should try to be pickable, copyable, and deepcopyable, etc ... Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Tips on Speeding up Python Execution
On Apr 8, 12:25 am, Chris Angelico wrote: > On Fri, Apr 8, 2011 at 5:04 PM, Abhijeet Mahagaonkar > > wrote: > > I was able to isolate that major chunk of run time is eaten up in opening a > > webpages, reading from them and extracting text. > > I wanted to know if there is a way to concurrently calling the functions. > > So, to clarify: you have code that's loading lots of separate pages, > and the time is spent waiting for the internet? If you're saturating > your connection, then this won't help, but if they're all small pages > and they're coming over the internet, then yes, you certainly CAN > fetch them concurrently. As the Perl folks say, There's More Than One > Way To Do It; one is to spawn a thread for each request, then collect > up all the results at the end. Look up the 'threading' module for > details: > > http://docs.python.org/library/threading.html The docs for Python3.2 have a nice example for downloading multiple webpages in parallel: http://docs.python.org/py3k/library/concurrent.futures.html#threadpoolexecutor-example Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Generators and propagation of exceptions
On Apr 8, 8:55 am, r wrote: > I had a problem for which I've already found a "satisfactory" > work-around, but I'd like to ask you if there is a better/nicer > looking solution. Perhaps I'm missing something obvious. > > The code looks like this: > > stream-of-tokens = token-generator(stream-of-characters) > stream-of-parsed-expressions = parser-generator(stream-of-tokens) > stream-of-results = evaluator-generator(stream-of-parsed-expressions) > > each of the above functions consumes and implements a generator: > > def evaluator-generator(stream-of-tokens): > for token in stream-of-tokens: > try: > yield token.evaluate() # evaluate() returns a Result > except Exception as exception: > yield ErrorResult(exception) # ErrorResult is a subclass of Result > > The problem is that, when I use the above mechanism, the errors > propagate to the output embedded in the data streams. This means, I > have to make them look like real data (in the example above I'm > wrapping the exception with an ErrorExpression object) and raise them > and intercept them again at each level until they finally trickle down > to the output. It feels a bit like writing in C (checking error codes > and propagating them to the caller). You're right, checking and propagating at each level doesn't feel right. If you need an exception to break out of the loops and close the generators, then it seems you should just ignore the exception on every level except for the one where you really want to handle the exception (possibly in an outer control-loop): while True: try: for result in evaluator-generator(stream-of-parsed- expressions): let the exception float up to > > OTOH, if I don't intercept the exception inside the loop, it will > break the for loop and close the generator. So the error no longer > affects a single token/expression but it kills the whole session. I > guess that's because the direction flow of control is sort of > orthogonal to the direction of flow of data. > > Any idea for a working and elegant solution? > > Thanks, > > -r -- http://mail.python.org/mailman/listinfo/python-list
Re: Generators and propagation of exceptions
On Apr 8, 8:55 am, r wrote: > I had a problem for which I've already found a "satisfactory" > work-around, but I'd like to ask you if there is a better/nicer > looking solution. Perhaps I'm missing something obvious. > > The code looks like this: > > stream-of-tokens = token-generator(stream-of-characters) > stream-of-parsed-expressions = parser-generator(stream-of-tokens) > stream-of-results = evaluator-generator(stream-of-parsed-expressions) > > each of the above functions consumes and implements a generator: > > def evaluator-generator(stream-of-tokens): > for token in stream-of-tokens: > try: > yield token.evaluate() # evaluate() returns a Result > except Exception as exception: > yield ErrorResult(exception) # ErrorResult is a subclass of Result > > The problem is that, when I use the above mechanism, the errors > propagate to the output embedded in the data streams. This means, I > have to make them look like real data (in the example above I'm > wrapping the exception with an ErrorExpression object) and raise them > and intercept them again at each level until they finally trickle down > to the output. It feels a bit like writing in C (checking error codes > and propagating them to the caller). You could just let the exception go up to an outermost control-loop without handling it at all on a lower level. That is what exceptions for you: terminate all the loops, unwind the stacks, and propagate up to some level where the exception is caught: while 1: try: results = evaluator-generator(stream-of-parsed-expressions) for result in results: print(result) except Exception as e: handle_the_exception(e) OTOH, If you want to catch the exception at the lowest level and wrap it up as data (the ErrorResult in your example), there is a way to make it more convenient. Give the ErrorResult object some identify methods that correspond to the methods being called by upper levels. This will let the object float through without you cluttering each level with detect-and-reraise logic. class ErrorResult: def __iter__(self): # pass through an enclosing iterator yield self Here's a simple demo of how the pass through works: >>> from itertools import * >>> list(chain([1,2,3], ErrorResult(), [4,5,6])) [1, 2, 3, <__main__.ErrorResult object at 0x2250f70>, 4, 5, 6] > Any idea for a working and elegant solution? Hope these ideas have helped. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Generators and propagation of exceptions
On Apr 8, 12:47 pm, r wrote: > Anyway, thank you all for helping me out and bringing some ideas to > the table. I was hoping there might be some pattern specifically > designed for thiskind of job (exception generators anyone?), which > I've overlooked. If not anything else, knowing that this isn't the > case, makes me feel better about the solution I've chosen. Sounds like you've gathered a bunch of good ideas and can now be pretty sure of your design choices. While it doesn't help your current use case, you might be interested in the iter_except() recipe in http://docs.python.org/py3k/library/itertools.html#itertools-recipes Raymond twitter: @raymondh -- http://mail.python.org/mailman/listinfo/python-list
Re: Creating unit tests on the fly
On Apr 8, 12:10 pm, Roy Smith wrote:
> I can even create new test cases from these on the fly with something
> like:
>
> newClass = type("newClass", (BaseSmokeTest,), {'route': '/my/newly/
> discovered/anchor'})
>
> (credit
> tohttp://jjinux.blogspot.com/2005/03/python-create-new-class-on-fly.html
> for that neat little trick). The only thing I don't see is how I can
> now get unittest.main(), which is already running, to notice that a
> new test case has been created and add it to the list of test cases to
> run. Any ideas on how to do that?
The basic unittest.main() runner isn't well suited to this task. It
flows in a pipeline of discovery -> test_suite -> test_runner.
I think you're going to need a queue of tests, with your own test
runner consuming the queue, and your on-the-fly test creator running
as a producer thread.
Writing your own test runner isn't difficult. 1) wait on the queue
for a new test case. 2) invoke test_case.run() with a TestResult
object to hold the result 3) accumulate or report the results 4)
repeat forever.
Raymond
twitter: @raymondh
--
http://mail.python.org/mailman/listinfo/python-list
Re: python on iPad (PyPad)
On Apr 8, 10:13 pm, Jon Dowdall wrote: > Hi All, > > Sorry for the blatant advertising but hope some of you may be interested > to know that I've created an iPad application containing the python > interpreter and a simple execution environment. It's available in iTunes > athttp://itunes.apple.com/us/app/pypad/id428928902?mt=8# > > I wanted to have python available 'on the go' without carrying a laptop. > The current implementation is based on my need to test simple python > functions in an isolated environment. I hope to add more iOS specific > capabilities if there is enough interest. I'm definitely interested in your adding more capabilities. Nice work so far. I'm surprised at the number of things that you got work in the first version: long integers, simple classes, a number of modules successfully import (math and itertools for example). I don't now how difficult the work is, but it would be nice to have a way to save multiple scripts, have an interactive prompt, and get more modules to work (random, collections, functools). Thanks again for your work. Raymond twitter: @raymondh -- http://mail.python.org/mailman/listinfo/python-list
Re: dict.setdefault()
On Apr 11, 2:35 pm, rantingrick wrote: > setdefault should take **kw args in the case of needing to set > multiple defaults at one time. I would even settle for an *arg list if > i had to. Anything is better than... > > d.setdefault(blah, blah) > d.setdefault(blah, blah) > d.setdefault(blah, blah) > d.setdefault(blah, blah) > if blah is not blah: > d.setdefault(blah, blah) I hate to feed a troll, but I'm curious when you see that pattern of calls. The typical use case always makes immediate use of the value returned by setdefault(): d.setdefault(key, []).append() The pattern you've shown would more typically be expressed like this: for key in multiple_defaults.items(): if key, default_value not in d: d[key] = default_value Or people sometimes use updates to get the same effect: result = multiple_defaults.copy() result.update(d) Another approach is to use something like ChainMap() which has been added to Python 3.3. http://docs.python.org/dev/library/collections.html#collections.ChainMap result = ChainMap(d, multiple_defaults) Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: dict.setdefault()
On Apr 11, 4:25 pm, Tim Chase wrote: > Finally, if it were added, I'd call it something like merge() Guido rejected merge() a long time ago. Anyway, there is a new ChainMap() tool in the collections module for Py3.3 that should address a number of use cases for handling default values. Raymond twitter: @raymondh -- http://mail.python.org/mailman/listinfo/python-list
Re: Questions about GIL and web services from a n00b
> > Is the limiting factor CPU? > > > If it isn't (i.e. you're blocking on IO to/from a web service) then the > > GIL won't get in your way. > > > If it is, then run as many parallel *processes* as you have cores/CPUs > > (assuming you're designing an application that can have multiple > > instances running in parallel so that you can run over multiple servers > > anyway). > > Great question. At this point, there isn't a limiting factor, but yes > the concern is around CPU in the future with lots of threads handling > many simultaneous transactions. In the Python world, the usual solution to high transaction loads is to use event-driven processing (using an async library such as Twisted) rather than using multi-threading which doesn't scale well in any language. Also, the usual way to take advantage of multiple-cores is to run multiple pythons in separate processes. Threading is really only an answer if you need to share data between threads, if you only have limited scaling needs, and are I/O bound rather than CPU bound Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Equivalent code to the bool() built-in function
On Apr 16, 1:24 pm, candide wrote:
> Consider the following code :
>
> # --
> def bool_equivalent(x):
> return True if x else False
It's faster to write:
def bool_equivalent(x):
return not not x
Raymond
--
http://mail.python.org/mailman/listinfo/python-list
Re: suggestions, comments on an "is_subdict" test
On Apr 22, 8:18 am, MRAB wrote:
> On 22/04/2011 15:57, Irmen de Jong wrote:
>
>
>
>
>
>
>
> > On 22-4-2011 15:55, Vlastimil Brom wrote:
> >> Hi all,
> >> I'd like to ask for comments or advice on a simple code for testing a
> >> "subdict", i.e. check whether all items of a given dictionary are
> >> present in a reference dictionary.
> >> Sofar I have:
>
> >> def is_subdict(test_dct, base_dct):
> >> """Test whether all the items of test_dct are present in base_dct."""
> >> unique_obj = object()
> >> for key, value in test_dct.items():
> >> if not base_dct.get(key, unique_obj) == value:
> >> return False
> >> return True
>
> >> I'd like to ask for possibly more idiomatic solutions, or more obvious
> >> ways to do this. Did I maybe missed some builtin possibility?
>
> > I would use:
>
> > test_dct.items()<= base_dct.items()
>
> In Python 2:
>
> >>> test_dct = {"foo": 0, "bar": 1}
> >>> base_dct = {"foo": 0, "bar": 1, "baz": 2}
> >>>
> >>> test_dct.items() <= base_dct.items()
> False
>
> In Python 3:
>
> >>> test_dct = {"foo": 0, "bar": 1}
> >>> base_dct = {"foo": 0, "bar": 1, "baz": 2}
> >>> test_dct.items() <= base_dct.items()
> True
>
> YMMV
That is because it is spelled differently in Python 2.7:
>>> test_dct = {"foo": 0, "bar": 1}
>>> base_dct = {"foo": 0, "bar": 1, "baz": 2}
>>> test_dct.viewitems() <= base_dct.viewitems()
True
Raymond
--
http://mail.python.org/mailman/listinfo/python-list
Simple map/reduce utility function for data analysis
Here's a handy utility function for you guys to play with: http://code.activestate.com/recipes/577676/ Raymond twitter: @raymondh -- http://mail.python.org/mailman/listinfo/python-list
Re: Simple map/reduce utility function for data analysis
On Apr 25, 7:42 pm, Paul Rubin wrote: > Raymond Hettinger writes: > > Here's a handy utility function for you guys to play with: > > http://code.activestate.com/recipes/577676/ > > Cute, but why not use collections.defaultdict for the return dict? > Untested: My first draft had a defaultdict but that implementation detail would get exposed to the user unless the return value was first coerced to a regular dict. Also, I avoided modern python features so the code would run well on psyco and so that it would make sense to beginning users. > Untested: > d = defaultdict(list) > for key,value in ifilter(bool,imap(mapper, data)): > d[key].append(value) > ... Nice use of itertools. FWIW, ifilter() will accept None for the first argument -- that's a bit faster than using bool(). Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Case study: debugging failed assertRaises bug
On Apr 25, 11:05 pm, Steven D'Aprano wrote: > I've just spent two hours banging my head against what I *thought* > (wrongly!) was a spooky action-at-a-distance bug in unittest, so I > thought I'd share it with anyone reading. Thanks for telling your story. I'm sure the lessons learned will be helpful to your readers. Raymond twitter: @raymondh -- http://mail.python.org/mailman/listinfo/python-list
Re: De-tupleizing a list
On Apr 25, 8:28 pm, Gnarlodious wrote:
> I have an SQLite query that returns a list of tuples:
>
> [('0A',), ('1B',), ('2C',), ('3D',),...
>
> What is the most Pythonic way to loop through the list returning a
> list like this?:
>
> ['0A', '1B', '2C', '3D',...
You could unpack the 1-tuple the same way you would with a 2-tuple.
>>> result = [('0A',), ('1B',), ('2C',), ('3D',)]
>>> for elem, in result:
print elem
0A
1B
2C
3D
Raymond
http://twitter.com/raymondh
--
http://mail.python.org/mailman/listinfo/python-list
Have you read the Python docs lately?
A number of developers have been working on adding examples and useful advice to the docs. To sharpen your skills, here are some pieces of recommended reading: http://docs.python.org/dev/library/heapq.html#priority-queue-implementation-notes http://docs.python.org/dev/library/bisect.html#searching-sorted-lists http://docs.python.org/dev/library/re.html#writing-a-tokenizer http://docs.python.org/dev/library/cmd.html#cmd-example http://docs.python.org/dev/library/collections.html#ordereddict-examples-and-recipes http://docs.python.org/dev/howto/logging.html http://docs.python.org/dev/howto/sorting.html http://docs.python.org/dev/library/collections.html#collections.namedtuple Raymond python tips on twitter: @raymondh -- http://mail.python.org/mailman/listinfo/python-list
Re: Have you read the Python docs lately?
On Apr 27, 11:28 pm, Paul Rubin wrote: > Raymond Hettinger writes: > > A number of developers have been working on adding examples and useful > > advice to the docs. To sharpen your skills, here are some pieces of > > recommended reading: > > Thanks, those are nice. The logging one looks especially useful. The > module always looked very confusing to me (too Java-like), and I've > dreaded the day when I might have to figure out how to use it instead of > my own ad-hoc logging. I can sleep better now ;-). Vinay put also put together a logging cookbook: http://docs.python.org/howto/logging-cookbook.html Raymond -- http://mail.python.org/mailman/listinfo/python-list
Coolest Python recipe of all time
I think it is time to give some visibility to some of the instructive and very cool recipes in ActiveState's python cookbook. My vote for the coolest recipe of all time is: http://code.activestate.com/recipes/365013-linear-equations-solver-in-3-lines/ What are your favorites? Raymond twitter: @raymondh -- http://mail.python.org/mailman/listinfo/python-list
Re: Coolest Python recipe of all time
On May 2, 11:29 pm, Gregory Ewing wrote: > Terry Reedy wrote: > > The trick is that replacing x with j and evaluating > > therefore causes (in Python) all the coefficients of x (now j) to be > > added together separately from all the constant terms to reduce the > > linear equation to a*x+b (= 0 implied). > > Hmmm... so if we used quaternions, could we solve systems > of linear equations in 3 variables? Yes :-) The implementation of a Quanternion class and the Quartic equation is left as an exercise for the reader ;-) Raymond @raymondh -- http://mail.python.org/mailman/listinfo/python-list
Re: Coolest Python recipe of all time
On May 2, 10:04 pm, Stefan Behnel wrote: > The bad thing about this recipe is that it requires quite a bit of > background knowledge in order to infer that the code the developer is > looking at is actually correct. At first sight, it looks like an evil hack, > and the lack of documentation doesn't help either. The recipe is cool in the same way that a magic trick is cool. A practical recipe would use a more general purpose method (perhaps using finite differences or continued fractions); it would have documentation and tests; it would accept a regular python function instead of a string; and it would avoid an unsanitized eval(). But then it would completely lose its panache, its flourish, and the pleasant gratification that you get when solving the riddle of how it works. We should have a separate thread for the most practical, best documented, least surprising, and most boring recipe ;-) Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Coolest Python recipe of all time
On May 2, 11:23 pm, Stefan Behnel wrote: > Terry Reedy, 03.05.2011 08:00: > > > On 5/3/2011 1:04 AM, Stefan Behnel wrote: > > >> The bad thing about this recipe is that it requires quite a bit of > >> background knowledge in order to infer that the code the developer is > >> looking at is actually correct. > > > The main math knowledge needed is the trivial fact that if a*x + b = 0, > > then x = -b/a. The other math knowledge needed is that complex numbers add > > componentwise. The trick is that replacing x with j and evaluating > > therefore causes (in Python) all the coefficients of x (now j) to be added > > together separately from all the constant terms to reduce the linear > > equation to a*x+b (= 0 implied). > > As your above paragraph proves, it's the kind of implementation that > requires three lines of executing code and at least 6 lines of additional > comment. Hopefully accompanied by an excuse of the developer. If you found nothing educational, interesting, or amusing about the three-line linear equation solver, then you're *really* going to hate this one: http://groups.google.com/group/comp.lang.python/browse_frm/thread/e46de4596e93188b/ Raymond @raymondh -- http://mail.python.org/mailman/listinfo/python-list
Today's fun and educational Python recipe
Here's a 22-line beauty for a classic and amazing algorithm: http://bit.ly/bloom_filter The wiki article on the algorithm is brief and well-written: http://en.wikipedia.org/wiki/Bloom_filter It turns out that people in the 1970's were pretty smart :-) Raymond --- follow my other python tips and recipes on twitter: @raymondh -- http://mail.python.org/mailman/listinfo/python-list
Re: Today's fun and educational Python recipe
> > It turns out that people in the 1970's were pretty smart :-) > > I think that often, the cleverness of people is inversely proportional > to the amount of CPU power and RAM that they have in their computer. The Google guys have plenty of CPU power *and* plenty of cleverness :-) According to the wikipedia article, Google BigTable uses Bloom filters to reduce the disk lookups for non-existent rows or column. The Google Chrome web browser also uses Bloom filters to speed up its Safe Browsing service. > Also: wasn't there a talk on Pycon in which a bloom filter was mentioned? Yes! As a matter of fact there was: http://www.slideshare.net/c.titus.brown/pycon-2011-talk-ngram-assembly-with-bloom-filters Raymond --- follow my other python tips and recipes on twitter: @raymondh -- http://mail.python.org/mailman/listinfo/python-list
Re: Today's fun and educational Python recipe
On May 4, 12:42 pm, Terry Reedy wrote:
> On 5/4/2011 2:17 PM, Raymond Hettinger wrote:
>
> > Here's a 22-line beauty for a classic and amazing algorithm:
> >http://bit.ly/bloom_filter
>
> > The wiki article on the algorithm is brief and well-written:
> >http://en.wikipedia.org/wiki/Bloom_filter
>
> As I understand the article, the array of num_bits should have
> num_probes (more or less independent) bits set for each key. But as I
> understand the code
>
> for i in range(self.num_probes):
> h, array_index = divmod(h, num_words)
> h, bit_index = divmod(h, 32)
> yield array_index, 1 << bit_index
>
> the same bit is being set or tested num_probes times. The last three
> lines have no dependence on i that I can see, so they appear to do the
> same thing each time. This seems like a bug.
The 512 bits in h are progressively eaten-up between iterations. So
each pass yields a different (array index, bit_mask) pair.
It's easy to use the interactive prompt to show that different probes
are produced on each pass:
>>> bf = BloomFilter(num_bits=1000, num_probes=8)
>>> pprint(list(bf.get_probes('Alabama')))
[(19, 1073741824),
(11, 64),
(9, 134217728),
(25, 1024),
(24, 33554432),
(6, 16),
(7, 16777216),
(22, 1048576)]
The 512 bits are uncorrelated -- otherwise sha512 wouldn't be much of
a cryptographic hash ;)
The fifty state example in the recipe is a reasonable demonstration
that the recipe works as advertised. It successfully finds all fifty
states (the true positives) and it tries 100,000 negatives resulting
in only a handful of false negatives. That should be somewhat
convincing that it all works.
Raymond
---
follow my other python tips and recipes on twitter: @raymondh
--
http://mail.python.org/mailman/listinfo/python-list
Re: Today's fun and educational Python recipe
On May 4, 12:27 pm, Paul Rubin wrote: > Raymond Hettinger writes: > > Here's a 22-line beauty for a classic and amazing algorithm: > >http://bit.ly/bloom_filter > > The use of pickle to serialize the keys is a little bit suspicious if > there might be a reason to dump the filter to disk and re-use it in > another run of the program. Pickle representation might change between > Python releases, for example. It's just supposed to stay interoperable > between versions, not necessarily bitwise-identical. > > Otherwise it's quite nice. I'd suggest adding a .update() operation > that adds keys from a user-supplied iterator. I chose pickle because it solved the problem of turning arbitrary objects into bytes which are needed as inputs to sha512. It seems that this particular choice of hash function is distracting some readers away from the interesting part of how a Bloom filter works. Since the choice of hash functions is completely arbitrary, I'm thinking of substituting something a little more basic. What do you think of this alternative as a way to make it clearer that each successive probe is uncorrelated, yet fully dependent on the key? def get_probes(self, key): hasher = Random(key).randrange num_words = len(self.arr) for _ in range(self.num_probes): array_index = hasher(num_words) bit_index = hasher(32) yield array_index, 1 << bit_index Raymond --- follow my other python tips and recipes on twitter: @raymondh -- http://mail.python.org/mailman/listinfo/python-list
Re: Today's fun and educational Python recipe
On May 4, 5:26 pm, Terry Reedy wrote: > The test would be more convincing to many with 10 other geographic > names (hard to come by, I know), or other english names or words or even > with longer random strings that matched the lengths of the state names. > But an average of 5/10 false positives in 5 runs is good. I've just posted an update with a spell checker using a large dictionary. That should make it easy to validate against some real world text samples. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Coolest Python recipe of all time
[Steven D'Aprano]: > As written, amb is just a brute-force solver using more magic than is > good for any code, but it's fun to play with. With a small change in API, much of the magic isn't needed. from itertools import product def amb(func, *argument_ranges): for args in product(*argument_ranges): if func(*args): print(args) amb(lambda a,b,c,d: 5*a+10*b+20*c+50*d == 45, range(3, 21), # number of 5 cent coins range(11), # number of 10 cent coins range(6), # number of 20 cent coins range(3), # number of 50 cent coins ) def test(a, b, c, d): s = "The %s brown %s jumped over the %s %s." % (a, b, c, d) num_vowels = sum(s.count(c) for c in 'aeiou') return num_vowels in (12, 18, 19) amb(test, ['quick', 'slow', 'hungry', 'wise-old'], ['fox', 'hare', 'turtle', 'kangaroo'], ['lazy', 'stupid', 'sleepy', 'confused'], ['dog', 'aardvark', 'sloth', 'wombat'], ) amb(lambda x, y, z: x*x + y*y == z*z, range(1, 11), range(1, 11), range(1, 11), ) Raymond follow my recipes and tips on twitter: @raymondh -- http://mail.python.org/mailman/listinfo/python-list
Re: checking if a list is empty
On May 5, 11:36 pm, Jabba Laci wrote: > Hi, > > If I want to check if a list is empty, which is the more pythonic way? > > li = [] > > (1) if len(li) == 0: > ... > or > (2) if not li: The Python core developers use the second form. See http://www.python.org/dev/peps/pep-0008/ for the official recommendation. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Coolest Python recipe of all time
On May 7, 1:29 am, Steven D'Aprano wrote: > On Fri, 06 May 2011 12:36:09 -0600, Ian Kelly wrote: > > The amb engine would conceptually execute this function for every > > possible combination of a, b, and c, > > Which pretty much is the definition of "brute-force solver", no? FWIW, here's one of my favorite brute-force solvers: http://code.activestate.com/recipes/576615-alphametics-solver/ Raymond --- follow my recipes and tips on twitter: @raymondh -- http://mail.python.org/mailman/listinfo/python-list
Re: Coolest Python recipe of all time
On May 9, 2:31 am, Trent Nelson wrote:
> > What are your favorites?
>
> I think I've posted this before, but I love my
> 3-lines-if-you-ignore-the-scaffolding language translator. Not because it's
> clever code -- quite the opposite, the code is dead simple -- but because it
> encompasses one of the things I love about Python the most: it gets shit done.
>
> In [1]: from translate import *
>
> In [2]: translate('French', 'The quick brown fox jumped over the lazy
> dog.')
> Le renard brun rapide a sauté par-dessus le chien paresseux.
>
> In [3]: translate('German', 'The quick brown fox jumped over the lazy
> dog.')
> Der schnelle braune Fuchs sprang über den faulen Hund.
>
> In [4]: translate('Spanish', 'The quick brown fox jumped over the lazy
> dog.')
> El zorro marrón rápido saltó sobre el perro perezoso.
>
> translate.py:
>
> import sys
> from urllib import urlopen, urlencode
> from BeautifulSoup import BeautifulSoup
>
> url = 'http://babelfish.altavista.com/tr'
> languages = {
> 'French' : 'en_fr',
> 'German' : 'en_de',
> 'Italian' : 'en_it',
> 'Spanish' : 'en_es',
> 'Russian' : 'en_ru',
> 'Portuguese': 'en_pt',
> 'Dutch' : 'en_nl',
> 'Japanese' : 'en_ja',
> }
>
> def translate(lang, text):
> kwds = { 'trtext' : text, 'lp' : languages[lang]}
> soup = BeautifulSoup(urlopen(url, urlencode(kwds)))
> print soup.find('div', style='padding:10px;').string
>
> if __name__ == '__main__':
> translate(sys.argv[1], sys.argv[2])
That completely rocks! Thanks for the recipe.
Raymond
--
follow my recipes and tips on twitter: @raymondh
--
http://mail.python.org/mailman/listinfo/python-list
Re: string formatting
> Which is the preferred way of string formatting?
>
> (1) "the %s is %s" % ('sky', 'blue')
>
> (2) "the {0} is {1}".format('sky', 'blue')
>
> (3) "the {} is {}".format('sky', 'blue')
>
> As I know (1) is old style. (2) and (3) are new but (3) is only
> supported from Python 2.7+.
>
> Which one should be used?
Sometimes, I use both ;-)
That can save you from ugly escapes such as %%s or {{0}}.
Here's an example from the standard library:
http://hg.python.org/cpython/file/7254c03b7180/Lib/collections.py#l235
Note the template has both {typename} formatting for the first pass
and %r style formatting in the generated code.
Raymond
follow my tips and recipes on twitter: @raymondh
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python 3 dict question
On May 6, 12:40 pm, dmitrey wrote: > hi all, > suppose I have Python dict myDict and I know it's not empty. > I have to get any (key, value) pair from the dict (no matter which > one) and perform some operation. > In Python 2 I used mere > key, val = myDict.items()[0] > but in Python 3 myDict.items() return iterator. > Of course, I could use > for key, val in myDict.items(): > do_something > break > but maybe there is any better way? If your use case allows the item to be removed, then use: key, val = myDict.popitem() Otherwise, use: key, val = next(iter(MyDict.items())) The latter is nice because next() allows you to supply a default argument in case the dictionary is emtpy: key, val = next(iter(MyDict.items()), (None, None)) Raymond - follow my tips and recipes on twitter: @raymondh -- http://mail.python.org/mailman/listinfo/python-list
Re: English Idiom in Unix: Directory Recursively
Hans Georg Schaathun writes: > ["Followup-To:" header set to comp.lang.python.] > On 18 May 2011 09:16:26 -0700, Thomas A. Russ >wrote: > : Well, unless you have a tree with backpointers, you have to keep the > : entire parent chain of nodes visited. Otherwise, you won't be able to > : find the parent node when you need to backtrack. A standard tree > : representation has only directional links. > > The array representation of a binary tree is standard, and the > «back» (parent) pointers are mathematically given. /Some/ > standard tree representation do not have parent pointers. I don't think anybody mentioned *binary* trees. The context was directory traversal, in which case you would have nodes with an arbitrary (almost) number of children. > You are right that I assumed parent pointers of some description; > but it does demonstrate that tree walks can be done iteratively, > without keeping a stack of any sort. Except that the chain of parent pointers *would* constitue a stack. -- http://mail.python.org/mailman/listinfo/python-list
Re: English Idiom in Unix: Directory Recursively
Hans Georg Schaathun writes: > ["Followup-To:" header set to comp.lang.python.] > On Wed, 18 May 2011 20:20:01 +0200, Raymond Wiker >wrote: > : I don't think anybody mentioned *binary* trees. The context was > : directory traversal, in which case you would have nodes with an > : arbitrary (almost) number of children. > > If we are being specific, then directory trees do have parent pointers. > My point was really that «standard tree representations» is not a > well-defined concept, and having parent pointers is as standard as > not having them. I cannot see that going back to the original case (directory traversal) is any more specific than talking about a completely unrelated case (binary trees). Further, even though most(?) hierarchical file systems have parent pointers, this is not necessary. > : Except that the chain of parent pointers *would* constitue a > : stack. > > In the sense that the tree itself is a stack, yes. But if we > consider the tree (or one of its branches) to be a stack, then > the original claim becomes a tautology. No, the tree is not a stack, but the chain of parent pointers from a particular node may be considered as a stack that records the path taken to reach the current node. > But you do have a point. Keeping a stack of nodes on the path > back to root is a great deal simpler and cheaper than a call > stack, and not really a significant expense in context. For this particular operation, possibly. For other tree operations, a single parent pointer may not be sufficient. -- http://mail.python.org/mailman/listinfo/python-list
Re: English Idiom in Unix: Directory Recursively
Hans Georg Schaathun writes: > ["Followup-To:" header set to comp.lang.python.] > On Wed, 18 May 2011 21:09:15 +0200, Raymond Wiker >wrote: > : > In the sense that the tree itself is a stack, yes. But if we > : > consider the tree (or one of its branches) to be a stack, then > : > the original claim becomes a tautology. > : > : No, the tree is not a stack, but the chain of parent pointers > : from a particular node may be considered as a stack that records the > : path taken to reach the current node. > > That is one of its branches, yes, path from root towards leaf. > It is part of the data structure, and you don't travers a data > structure without using the datastructure. > > : > But you do have a point. Keeping a stack of nodes on the path > : > back to root is a great deal simpler and cheaper than a call > : > stack, and not really a significant expense in context. > : > : For this particular operation, possibly. For other tree > : operations, a single parent pointer may not be sufficient. > > Que? What tree operations do you have in mind? We have covered > all the standard textbook tree walks by now. I said tree operations, not tree walks. A tree operation might involve several tree walks. Further, there has been an implicit assumption (I think) in this discussion that the order of children is given, or does not matter - if this is not the case, then you also need to maintain a stack of data structures representing lists (or sets) of children. -- http://mail.python.org/mailman/listinfo/python-list
Re: Faster Recursive Fibonacci Numbers
On May 17, 8:50 am, RJB wrote: > I noticed some discussion of recursion. the trick is to find a > formula where the arguments are divided, not decremented. > I've had a "divide-and-conquer" recursion for the Fibonacci numbers > for a couple of years in C++ but just for fun rewrote it > in Python. It was easy. Enjoy. And tell me how I can improve it! > > def fibo(n): > """A Faster recursive Fibonaci function > Use a formula from Knuth Vol 1 page 80, section 1.2.8: > If F[n] is the n'th Fibonaci number then > F[n+m] = F[m]*F[n+1] + F[m-1]*F[n]. > First set m = n+1 > F[ 2*n+1 ] = F[n+1]**2 + F[n]*2. > > Then put m = n in Knuth's formula, > F[ 2*n ] = F[n]*F[n+1] + F[n-1]* F[n], > and replace F[n+1] by F[n]+F[n-1], > F[ 2*n ] = F[n]*(F[n] + 2*F[n-1]). > """ > if n<=0: > return 0 > elif n<=2: > return 1 > elif n%2==0: > half=n//2 > f1=fibo(half) > f2=fibo(half-1) > return f1*(f1+2*f2) > else: > nearhalf=(n-1)//2 > f1=fibo(nearhalf+1) > f2=fibo(nearhalf) > return f1*f1 + f2*f2 > > RJB the Lurkerhttp://www.csci.csusb.edu/dick/cs320/lab/10.html There are many ways to write this function. The one I like shows-off a general purpose dynamic programming technique while staying *very* close to a common textbook definition of a fibonacci number: @functools.lru_cache() def fibo(n): return 1 if n < 2 else fibo(n-1) + fibo(n-2) Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: super() in class defs?
On May 25, 4:31 pm, Ian Kelly wrote: > Right. It's unnecessary, so why saddle yourself with it? FWIW, I expect to release a blog post tomorrow about the principal use cases for super() and how to use it effectively. With just a little bit of know-how, it can be an important tool in your Python toolkit. If any of the comp.lang.python readers want to review and comment on my latest draft, please email me and I'll send it to you directly. Cheers, Raymond Hettinger my email address is listed at http://users.rcn.com/python/download/Descriptor.htm -- http://mail.python.org/mailman/listinfo/python-list
Re: Parse config file and command-line arguments, to get a single collection of options
On May 25, 9:38 pm, Ben Finney wrote: > Howdy all, > > Python's standard library has modules for configuration file parsing > (configparser) and command-line argument parsing (optparse, argparse). I > want to write a program that does both, but also: > > * Has a cascade of options: default option values, overridden by config > file options, overridden by command-line options. > > * Reads a different, or even additional, configuration file if specified > on the command-line (e.g. --config-file foo.conf) and yet still obeys > the above cascade. > > * Allows a single definition of an option (e.g. logging level) to define > the same option for parsing from configuration files and the command > line. > > * Unifies the parsed options into a single collection for the rest of > the program to access without caring where they came from. > > How can I achieve this with minimum deviation from the Python standard > library? One thought is start with something like ChainMap, http://code.activestate.com/recipes/305268-chained-map-lookups/?in=user-178123 , or some variant to unify multiple mapping objects into a single prioritized collection. A mapping for command line args can be made by using vars() on an argparse namespace to create a dictionary. ConfigParser's mapping is accessible via its get() method. With a ChainMap style object you can add other option sources such as os.environ. This should get you started on your grand unified, do- everything-at-once vision with minimal deviation from the standard library. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Python's super() considered super!
I just posted a tutorial and how-to guide for making effective use of super(). One of the reviewers, David Beazley, said, "Wow, that's really great!I see this becoming the definitive post on the subject" The direct link is: http://rhettinger.wordpress.com/2011/05/26/super-considered-super/ It would also be great if some of you would upvote it on HackerNews. Raymond Hettinger --- follow my python tips on twitter: @raymondh -- http://mail.python.org/mailman/listinfo/python-list
Re: Python's super() considered super!
> It would also be great if some of you would upvote it on HackerNews. Here's a link to the super() how-to-guide and commentary: bit.ly/ iFm8g3 Raymod -- http://mail.python.org/mailman/listinfo/python-list
Re: Python's super() considered super!
On May 26, 6:39 pm, Ben Finney wrote: > We also, though, need *real* URLs. Blind URLs through obfuscation > services have their uses, but surely not in a forum like this. The real > URL is http://news.ycombinator.com/item?id=2588262>. Fair enough. I had copied the link from Jesse's tweet (where shorter is better). Hope you enjoyed the post. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Class decorators might also be super too
David Beazley wrote a class decorator blog post that is worth reading: http://dabeaz.blogspot.com/2011/05/class-decorators-might-also-be-super.html Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Class decorators might also be super too
On May 28, 11:33 pm, Michele Simionato wrote: > He is basically showing that using mixins for implementing logging is not > such a good idea, i.e. you can get the same effect in a better way by making > use of other Python features. I argued the same thing many times in the past. > I even wrote a module once (strait) to reimplement 99% of multiple > inheritance without multiple inheritance, just to show that in can be done in > few lines of code in a language as powerful as Python. More importantly, anyone who reads posts such as David' will walk away with a deeper understanding of class decorators, mixins, and inheritance. That makes the post worthwhile even if someone never ends up using those particular coding technique. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: float("nan") in set or as key
On May 28, 4:41 pm, MRAB wrote:
> Here's a curiosity. float("nan") can occur multiple times in a set or as
> a key in a dict:
Which is by design.
NaNs intentionally have multiple possible instances (some
implementations even include distinct payload values).
Sets and dicts intentionally recognize an instance as being equal to
itself (identity-implies-equality); otherwise, you could put a NaN in
a set/dict but not be able to retrieve it. Basic invariants would
fail -- such as: assert all(elem in container for elem in container).
The interesting thing is that people experimenting with exotic objects
(things with random hash functions, things with unusual equality or
ordering relations, etc) are "surprised" when those objects display
their exotic behaviors.
To me, the "NaN curiousities" are among the least interesting. It's
more fun to break sort algorithms with sets (which override the
ordering relations with subset/superset relations) or with an object
that mutates a list during the sort. Now, that is curious :-)
Also, Dr Mertz wrote a Charming Python article full of these
curiosities:
http://gnosis.cx/publish/programming/charming_python_b25.txt
IMO, equality and ordering are somewhat fundamental concepts. If a
class is written that twists those concepts around a bit, then it
should be no surprise if curious behavior emerges. Heck, I would
venture to guess that something as simple as assuming the speed of
light is constant might yield twin paradoxes and other
curiousities ;-)
Raymond
--
http://mail.python.org/mailman/listinfo/python-list
Re: 3.1.4 release candidate 1
On May 29, 3:44 pm, Benjamin Peterson wrote: > On behalf of the Python development team, I'm happy as a swallow to announce a > release candidate for the fourth bugfix release for the Python 3.1 > series, Python > 3.1.4. The Pi release of Python :-) Raymond P.S. For the most part, if you have a choice, then you are much better off using Py3.2 than any release (even a bugfix release) of Py3.1 -- http://mail.python.org/mailman/listinfo/python-list
Updated blog post on how to use super()
I've tightened the wording a bit, made much better use of keyword arguments instead of kwds.pop(arg), and added a section on defensive programming (protecting a subclass from inadvertently missing an MRO requirement). Also, there is an entry on how to use assertions to validate search order requirements and make them explicit. http://bit.ly/py_super or http://rhettinger.wordpress.com/2011/05/26/super-considered-super/ Any further suggestions are welcome. I'm expecting this to evolve into how-to guide to be included in the regular Python standard documentation. The goal is to serve as a reliable guide to using super and how to design cooperative classes in a way that lets subclasses compose and extent them. Raymond Hettinger follow my python tips on twitter: @raymondh -- http://mail.python.org/mailman/listinfo/python-list
RE: Changing calling sequence
>> I have a function that I use to retrieve daily data from a
>> home-brew database. Its calling sequence is;
>>
>> def TempsOneDay( year, month, date ):
>>
>> After using it (and its friends) for a few years, I've come to
>> realize that there are times where it would be advantageous to
>> invoke it with a datetime.date as its single argument.
>
>You could just use all keyword args:
>
>def TempsOneDay(**kwargs):
>
> if 'date' in kwargs:
> handle_datetime(kwargs['date'])
> elif 'year' in kwargs and 'month' in kwargs and 'day' in kwargs:
> handle_args(kwargs['year'], kwargs['month'], kwargs['day'])
> else:
> raise Exception("Bad keyword args")
>
>TempsOneDay(date=datetime.datetime.now)
>
>TempsOneDay(year=2022, month=11, day=30)
>
Maybe not the prettiest, but you could also define it like this, which also
wouldn't require changing of any existing calls or the main body of the
function past this if block.
def TempsOneDay(*dateComponents):
if len(dateComponents) == 3:
year, month, date = dateComponents
elif len(dateComponents) == 1 and isinstance(dateComponents[0],
datetime.date):
year, month, date = (dateComponents[0].year, dateComponents[0].month,
dateComponents[0].day)
else:
raise Exception("Error message here")
--
https://mail.python.org/mailman/listinfo/python-list
RE: Changing calling sequence
>>def TempsOneDay(*dateComponents):
>>if len(dateComponents) == 3:
>>year, month, date = dateComponents
>>elif len(dateComponents) == 1 and isinstance(dateComponents[0],
>> datetime.date):
>>year, month, date = (dateComponents[0].year, dateComponents[0].month,
>> dateComponents[0].day)
>>else:
>>raise Exception("Error message here")
>
>|>>> help( TempsOneDay )
>|Help on function TempsOneDay in module __main__:
>|
>|TempsOneDay(*dateComponents)
Then just add an appropriate docstring.
>>> def TempsOneDay(*dateComponents):
... """Can be called either with 3 arguments: year, month, day
...or with a single datetime.date object"""
... if len(dateComponents) == 3:
... year, month, date = dateComponents
... elif len(dateComponents) == 1 and isinstance(dateComponents[0],
datetime.date):
... year, month, date = (dateComponents[0].year,
dateComponents[0].month, dateComponents[0].day)
... else:
... raise Exception("Error message here")
...
>>> help(TempsOneDay)
Help on function TempsOneDay in module __main__:
TempsOneDay(*dateComponents)
Can be called either with 3 arguments: year, month, day
or with a single datetime.date object
>>>
--
https://mail.python.org/mailman/listinfo/python-list
Pip upgrade causing issues in 3.10
So after a long while I'm finally getting around to upgrading to 3.10 on Windows from 3.9, and my first pip upgrade is causing issues with the installation. Problem seems to be that I run pip from a command prompt in the Scripts folder, and it seems pip is trying to completely remove the Scripts folder, which it can't because the command prompt is in that folder. It successfully removes the old installation of pip before dying, leaving you without a pip to run. (Yay for ensurepip) It also creates a ~ripts folder in the main Python folder, and a few ~ip folders in the site-packages folder before it dies, which then cause warnings to be issued by pip from then on (or at least until you go and delete those folders) Upgrading pip this way worked fine in 3.9.x, so I'm curious as to why the change for 3.10? And is there a chance to go back to it not trying to completely remove that Scripts folder? Or at least have it fail a little more gracefully without creating all those ~ folders? Maybe add a hint about ensurepip when it removes the old and can't install the new? For those who will say just change how you update it: I can do that going forward, yes. But nonetheless, running it like this probably shouldn't leave me without a pip, and with extra junk folders. Fresh install here of 3.10.5. For all users, not including the py launcher thing. Below is running in an Admin Command Prompt. C:\Program Files\Python310\Scripts>pip list PackageVersion -- --- pip22.0.4 setuptools 58.1.0 WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available. You should consider upgrading via the 'C:\Program Files\Python310\python.exe -m pip install --upgrade pip' command. C:\Program Files\Python310\Scripts>..\python.exe -m pip install --upgrade pip Requirement already satisfied: pip in c:\program files\python310\lib\site-packages (22.0.4) Collecting pip Using cached pip-22.1.2-py3-none-any.whl (2.1 MB) Installing collected packages: pip Attempting uninstall: pip Found existing installation: pip 22.0.4 Uninstalling pip-22.0.4: ERROR: Could not install packages due to an OSError: [WinError 32] The process cannot access the file because it is being used by another process: 'c:\\program files\\python310\\scripts\\' Consider using the `--user` option or check the permissions. At this point the Scripts folder is completely empty, there's a ~ripts folder in the main python310 folder, and a few ~ip... folders in the site-packages folder C:\Program Files\Python310\Scripts>..\python.exe -m ensurepip WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) Looking in links: c:\Users\usernamehere\AppData\Local\Temp\tmpt48ibdf3 Requirement already satisfied: setuptools in c:\program files\python310\lib\site-packages (58.1.0) Processing c:\users\usernamehere\appdata\local\temp\tmpt48ibdf3\pip-22.0.4-py3-none-any.whl WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) Installing collected packages: pip WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) Successfully installed pip-22.0.4 C:\Program Files\Python310\Scripts>cd.. C:\Program Files\Python310>python.exe -m pip install -U pip WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) Requirement already satisfied: pip in c:\program files\python310\lib\site-packages (22.0.4) Collecting pip Using cached pip-22.1.2-py3-none-any.whl (2.1 MB) WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) Installing collected packages: pip Attempting uninstall: pip WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) Found existing installation: pip 22.0.4 Uninstalling pip-22.0.4: Successfully uninstalled pip-22.0.4 WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) Successfully installed pip-22.1.2 WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) C:\Program Files\Python310>cd Scripts C:\Program Files\Python310\Scripts>pip list WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) PackageVersion -- --- pip22.1.2 setuptools 58.1.0 WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) WARNING: Ignoring invalid distribution -ip (c:\program files\pytho
RE: Parallel(?) programming with python
>> But, an easier and often >> better option for concurrent data access is use a (relational) >> database, then the appropriate transaction isolation levels >> when reading and/or writing. >> > > That would obviusly save some coding (but would introduce the need to > code the interaction with the database), but I'm not sure it would speed > up the thing. Would the RDBMS allow to read a table while something else > is writing to it? I doubt it and I'm not sure it doesn't flush the cache > before letting you read, which would include a normally slow disk access. SQLite for example allows only 1 write transaction at a time, but in WAL mode you can have as many read transactions as you want all going along at the same time as that 1 writer. It also allows you to specify how thorough it is in flushing data to disk, including not forcing a sync to disk at all and just leaving that to the OS to do on its own time. -- https://mail.python.org/mailman/listinfo/python-list
RE: Find 6-letter words that are hidden (embedded) within
> Find 6-letter words that are hidden (embedded) within each row of letters. > The letters are in the correct order. > > 1. JSOYOMFUBELR > 2. SCDUARWDRLYE > 3. DASNAGEFERTY > 4. CLULOOTSCEHN > 5. USENEARSEYNE > The letters are in the correct order. So this problem is not about > Anagraming. You can get every combination of 6 letters out of it with itertools.combinations like below. Just implement the isWord function to return whether a string actually counts as a legit word or not. 12 choose 6 is only 924 combinations to check, so shouldn't be too bad to check them all. def isWord(word): return True #Best left as an exercise to the reader startWord = "JSOYOMFUBELR" subLetterCount = 6 foundWords = set() for letters in itertools.combinations(startWord, subLetterCount): word = "".join(letters) if word not in foundWords and isWord(word): print(word) foundWords.add(word) -- https://mail.python.org/mailman/listinfo/python-list
RE: How to escape strings for re.finditer?
> I wrote my previous message before reading this. Thank you for the test you > ran -- it answers the question of performance. You show that re.finditer is > 30x faster, so that certainly recommends that over a simple loop, which > introduces looping overhead. >> def using_simple_loop(key, text): >> matches = [] >> for i in range(len(text)): >> if text[i:].startswith(key): >> matches.append((i, i + len(key))) >> return matches >> >> using_simple_loop: [0.1395295020792, 0.1306313000456, >> 0.1280345001249, 0.1318618002423, 0.1308461032626] >> using_re_finditer: [0.00386140005233, 0.00406190124297, >> 0.00347899970256, 0.00341310216218, 0.003732001273] With a slight tweak to the simple loop code using .find() it becomes a third faster than the RE version though. def using_simple_loop2(key, text): matches = [] keyLen = len(key) start = 0 while (foundSpot := text.find(key, start)) > -1: start = foundSpot + keyLen matches.append((foundSpot, start)) return matches using_simple_loop: [0.1732664997689426, 0.1601669997908175, 0.15792609984055161, 0.157397349591, 0.15759290009737015] using_re_finditer: [0.003412699792534113, 0.0032823001965880394, 0.0033694999292492867, 0.003354900050908327, 0.006998894810677] using_simple_loop2: [0.00256159994751215, 0.0025471001863479614, 0.0025424999184906483, 0.0025831996463239193, 0.002999018251896] -- https://mail.python.org/mailman/listinfo/python-list
RE: Debugging reason for python running unreasonably slow when adding numbers
> Or use the sum() builtin rather than reduce(), which was > *deliberately* removed from the builtins. The fact that you can get > sum() without importing, but have to go and reach for functools to get > reduce(), is a hint that you probably shouldn't use reduce when sum > will work. Out of curiosity I tried a couple variations and am a little confused by the results. Maybe I'm having a brain fart and am missing something obvious? Each of these was run with the same "data" and "perceptrons" values to keep that fair. Times are averages over 150 iterations like the original. The only thing changed in the trainPerceptron function was how to calculate sum_ Original: sum_ = 0 for key in input: v = weights[key] sum_ += v 418ms The reduce version: sum_ = reduce(lambda acc, key: acc + weights[key], input) 758ms Getting rid of the assignment to v in the original version: sum_ = 0 for key in input: sum_ += weights[key] 380ms But then using sum seems to be slower sum with generator expression: sum_ = sum(weights[key] for key in input) 638ms sum with list comprehension: sum_ = sum([weights[key] for key in input]) 496ms math.fsum with generator expression: sum_ = math.fsum(weights[key] for key in input) 618ms math.fsum with list comprehension: sum_ = math.fsum([weights[key] for key in input]) 480ms I'm not quite sure why the built-in sum functions are slower than the for loop, or why they're slower with the generator expression than with the list comprehension. -- https://mail.python.org/mailman/listinfo/python-list
RE: Debugging reason for python running unreasonably slow when adding numbers
> Then I'm very confused as to how things are being done, so I will shut > up. There's not enough information here to give performance advice > without actually being a subject-matter expert already. Short version: In this specific case "weights" is a 5,147 element list of floats, and "input" is a 10 element list of integers which has the indexes of the 10 elements in weights that he wants to add up. sum_ = 0 for key in input: sum_ += weights[key] vs sum_ = sum(weights[key] for key in input) vs... other ways -- https://mail.python.org/mailman/listinfo/python-list
RE: Problem with __sub__
I believe your problem is __rsub__, not __sub__. When you havethen that uses the "r" version of the operators. In your __rsub__ (used when you have - ) you instead return - which is backwards. Notice how the final return should also be -4,95 and not the +4,95 it's returning. > If on the left side is '0' the result of a subtraction is wrong. > > *b1 = Betragswert(500) > b2 = 0 + b1 > b3 = 0 - b1 > b4 = 5 + b1 > b5 = 5 - b1* > > print(b1, b2, b3, b4, b5) shows 5,00 5,00 5,00 5,05 4,95; the third > value (b3) should be -5,00 (not 5,00). > > Why is the substraction wrong? > def __rsub__(self, zweiter): > if not isinstance(zweiter, type(self)): > zweiter = Betragswert(zweiter) > return Betragswert(self._Betrag - zweiter._Betrag) -- https://mail.python.org/mailman/listinfo/python-list
RE: Initialising a Config class
Not sure if I'm fully understanding the question. But one option instead of
making everything class attributes is to just define __getattr__ for when it
doesn't find an attribute.
Won't work for every single valid section and option name (because of spaces,
name overlaps, etc) but should cover most things.
For example, you could use a dunder to separate section and option. Then
something like this?
import configparser
class Config:
def __init__(self, configFile):
self._config = configparser.ConfigParser()
self._config.read(configFile)
def __getattr__(self, option):
if "__" in option:
section, option = option.split("__", 1)
else:
section = self._config.default_section
return self._config[section][option]
c = Config("SomeConfigFile.txt")
print(c.uids__minimum_uid) #Will check for the option "minimum_uid" in the
"uids" section
print(c.minimum_uid) #Will check the default section
Not sure if that works as you said the Config class itself should not need to
be changed
> Hi,
>
> Having solved my problem regarding setting up 'logger' such that it is
> accessible throughout my program (thanks to the help on this list), I
> now have problem related to a slightly similar issue.
>
> My reading suggests that setting up a module with a Config class which
> can be imported by any part of the program might be a reasonable approach:
>
>
> import configparser
>
> class Config:
>
> def __init__(self, config_file):
>
> config = configparser.ConfigParser()
> config.read(config_file)
>
>
> However, in my config file I am using sections, so 'config' is a dict of
> dicts. Is there any cleverer generic way of initialising the class than
>
>
> self.config = config
>
>
> ?
>
> This seems a bit clunky, because I'll end up with something like
>
>
>import config
>...
>c = config.Config(config_file)
>uids = get_uids(int(c.config["uids"]["minimum_uid"]))
>
>
> rather than something like, maybe
>
>
>uids = get_uids(int(c.minimum_uid))
>
>
> or
>
>
>uids = get_uids(int(c.uids_minimum_uid))
>
>
> So the question is: How can I map a dict of dicts onto class attributes
> in a generic way such that only code which wants to use a new
> configuration parameter needs to be changed and not the Config class
> itself? Or should I be doing this differently?
>
> Note that the values from ConfigParser are all strings, so I am fine
> with the attributes being strings - I'll just convert them as needed at
> the point of use (but maybe there is also a better way of handling that
> within a class).
>
> Cheers,
>
> Loris
>
--
https://mail.python.org/mailman/listinfo/python-list
RE: How do you debug in Python? Coming from a Matlab and R user. I'm already aware of pdb.
In regards to the various comments about adding in print() calls what I've
found myself doing is to basically always use the logging module, and use
logging.debug() for those.
Somewhere at the top of the script I'll have a line like...
DEBUG = False
...and when initializing the handler to stdout I'll do something like this...
toScreen = logging.StreamHandler(sys.stdout)
toScreen.setLevel(logging.DEBUG if DEBUG else logging.INFO)
That way I can sprinkle in
logging.debug("Some message here")
in various spots. If things are going wrong I can change DEBUG to True to see
them on screen, and when I've fixed it I can just set DEBUG back to False. That
way I don't have to try finding all those print() statements that were only
there for debugging and comment them out or remove them.
As a bonus, if you also set a file logger for example with its level set to
logging.DEBUG, then you can have those go into the log file without them
cluttering the screen output.
As a side effect to using logging I found I also like having the timestamp
automatically prepended to my output by the logger. I'd find myself checking in
on something I left running in the background and thinking it's been on that
step for "a while", but I have no idea how long "a while" really is. So the
timestamps help quickly show "been stuck there for 3 hours" vs "just got to
that step 3 seconds before I switched to its window"
I don't claim these are the best practices :) But figured I'd offer another
take on it.
--
https://mail.python.org/mailman/listinfo/python-list
RE: on writing a while loop for rolling two dice
> def how_many_times(): > x, y = 0, 1 > c = 0 > while x != y: > c = c + 1 > x, y = roll() > return c, (x, y) Since I haven't seen it used in answers yet, here's another option using our new walrus operator def how_many_times(): roll_count = 1 while (rolls := roll())[0] != rolls[1]: roll_count += 1 return (roll_count, rolls) -- https://mail.python.org/mailman/listinfo/python-list
RE: frozenset can be altered by |=
>> (venv_3_10) marco@buzz:~$ python
>> Python 3.10.0 (heads/3.10-dirty:f6e8b80d20, Nov 18 2021, 19:16:18)
>> [GCC 10.1.1 20200718] on linux
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> a = frozenset((3, 4))
>> >>> a
>> frozenset({3, 4})
>> >>> a |= {5,}
>> >>> a
>> frozenset({3, 4, 5})
>
> That's the same as how "x = 4; x += 1" can "alter" four into five.
>
> >>> a = frozenset((3, 4))
> >>> id(a), a
> (140545764976096, frozenset({3, 4}))
> >>> a |= {5,}
> >>> id(a), a
> (140545763014944, frozenset({3, 4, 5}))
>
> It's a different frozenset.
>
> ChrisA
Another possible option is instead of
a |= {5,}
change it to
a.update({5,})
If a is a regular set it will update the original object, and if a is a
frozenset it will raise an AttributeError. Which may not be what you want, but
at least it won't quietly do something you weren't expecting.
It is a little confusing since the docs list this in a section that says they
don't apply to frozensets, and lists the two versions next to each other as the
same thing.
https://docs.python.org/3.9/library/stdtypes.html#set-types-set-frozenset
The following table lists operations available for set that do not apply to
immutable instances of frozenset:
update(*others)
set |= other | ...
Update the set, adding elements from all others.
--
https://mail.python.org/mailman/listinfo/python-list
RE: One-liner to merge lists?
> Is there a simpler way?
>>> d = {1: ['aaa', 'bbb', 'ccc'], 2: ['fff', 'ggg']}
>>> [a for b in d.values() for a in b]
['aaa', 'bbb', 'ccc', 'fff', 'ggg']
>>>
--
https://mail.python.org/mailman/listinfo/python-list
Re: best parallelisation strategy on python
On Wednesday, April 18, 2018 at 7:16:19 PM UTC-6, simona bellavista wrote: > I have a code fortran 90 that is parallelised with MPI. I would like to > traslate it in python, but I am not sure on the parallelisation strategy and > libraries. I work on clusters, with each node with 5GB memory and 12 > processors or 24 processors (depending on the cluster I am using). Ideally I > would like to split the computation on several nodes. > > Let me explain what this code does: It read ~100GB data, they are divided in > hdf5 files of ~25GB each. The code should read the data, go through it and > then select a fraction of the data, ~1GB and then some CPU intensive work on > it, and repeat this process many times, say 1000 times, then write the > results to a single final file. > > I was thinking that the CPU intensive part would be written as a shared > object in C. > > Do you have suggestions about which library to use? I would suggest the Python multiprocessing package. In Python you have to use processes to get full parallelism as there is a single lock on the Python interpreter. The multiprocessing package supports this computing model. -- https://mail.python.org/mailman/listinfo/python-list
RE: Checking whether type is None
https://docs.python.org/3.7/library/constants.html
"None
The sole value of the type NoneType..."
"x is None" and "type(x) is type(None)" are equivalent because of that.
I think though that the better way to do the first tests would be to use
isinstance
https://docs.python.org/3.7/library/functions.html#isinstance
isinstance({}, dict)
isinstance(3, int)
And I suppose if you really wanted:
isinstance(None, type(None))
-Original Message-
From: Python-list
[mailto:[email protected]] On Behalf Of
Tobiah
Sent: Tuesday, July 24, 2018 3:33 PM
To: [email protected]
Subject: Checking whether type is None
Consider:
>>> type({}) is dict
True
>>> type(3) is int
True
>>> type(None) is None
False
Obvious I guess, since the type object is not None.
So what would I compare type(None) to?
>>> type(None)
>>> type(None) is NoneType
Traceback (most recent call last):
File "", line 1, in
NameError: name 'NoneType' is not defined
I know I ask whether:
>>> thing is None
but I wanted a generic test.
I'm trying to get away from things like:
>>> type(thing) is type(None)
because of something I read somewhere preferring
my original test method.
Thanks
--
https://mail.python.org/mailman/listinfo/python-list
--
https://mail.python.org/mailman/listinfo/python-list
RE: Python bug in ArcGIS - Urban Network analysis tool
A note that Arc may have installed its own version of Python, which it is using from within its own tools. For example, I've got a full Python installation in C:\Python27\ArcGIS10.2 which Arc installed on top of a preexisting installation in C:\Python27. So you may need to explicitly run it with that version to get it to work and to import arcpy ok. ie instead of just running... pydevd.py or... python pydevd.py You might need to explicitly do... C:\Python27\ArcGIS10.2\python.exe pydevd.py ...or the equivalent of wherever it installed it on your system. -Original Message- From: Python-list [mailto:[email protected]] On Behalf Of ? Sent: Monday, July 30, 2018 2:03 AM To: [email protected] Subject: Re: Python bug in ArcGIS - Urban Network analysis tool понеделник, 30 юли 2018 г., 3:29:44 UTC+3, MRAB написа: > On 2018-07-29 22:00, Станимира Николова wrote: > > неделя, 29 юли 2018 г., 23:41:01 UTC+3, MRAB написа: > >> On 2018-07-29 18:56, [email protected] wrote: > >> > Hi there, > >> > > >> > I'm trying make reach analyze with Urban network analysis (UNA) tool in > >> > ArcGIS. It's based on points for buildings and pedestrian network of > >> > acceptability steets. The aim is to show how many people from each > >> > building can reach different building in radius of 150 meters (so i can > >> > find the density of the pedestrian routes). > >> > > >> > I run Urban network analysis but It shows similar mistake several times. > >> > The UNA tool is free plugin that i downloaded, it's not from the default > >> > intalled in ArcGIS packed. It ask for integer data. > >> > I checked the type of the attributes, it's all integer. PLus it's all in > >> > geo data based file. > >> > > >> > Unfortunately I don't understand Python, I'm one of those who use ArcGIS > >> > as sample customer. > >> > > >> > This is the mistake: > >> > > >> > Start Time: Fri Jul 27 14:48:32 2018 > >> > Running script Centrality... > >> > [started] Copying input buildings > >> > [finished] > >> > [1 started] Computing adjacency list > >> > [1 failed] > >> > Not successful > >> > Completed script Centrality... > >> > Succeeded at Fri Jul 27 14:48:36 2018 (Elapsed Time: 4,56 seconds) > >> > > >> > Any suggestions? How it's calling these adjaency list? What could be > >> > wrong? I even don't know how to get debugger, so it could give me more > >> > information. > >> > > >> > Thank you previously, > >> > Mira > >> > > >> All you've shown is that says that it was unsuccessful, and it doesn't > >> say why. > >> > >> We'd need a lot more info than that! > >> > >> The best way to debug it is to try it with the minimal amount of test > >> data that should give a result. > > > > I'm sorry. > > Could someone help me by telling me how to make these test data? (so i > > could send it to You) > > > > I realize the review of the bug is just the customer view. Any chance to > > help me out with the test of the process? > > > > Thank you previously. If there is another forum that i have to write to, i > > will switch. The fact is that in the Themes for UNA tool or GIS software, > > almost no one respond. > > > I'm unfamiliar with ArcGIS, so I doubt I could help. > > It might have generated a log file somewhere that gives more details. Ok, thank you for the patiance. I installed PyCharm for trying make some tests. I add in a project the .py file for the adjacency list. That's the main from the debuger: pydev debugger: process 8904 is connecting Connected to pydev debugger (build 182.3684.100) Traceback (most recent call last): File "C:\Program Files\JetBrains\PyCharm 2018.2\helpers\pydev\pydevd.py", line 1664, in main() File "C:\Program Files\JetBrains\PyCharm 2018.2\helpers\pydev\pydevd.py", line 1658, in main globals = debugger.run(setup['file'], None, None, is_module) File "C:\Program Files\JetBrains\PyCharm 2018.2\helpers\pydev\pydevd.py", line 1068, in run pydev_imports.execfile(file, globals, locals) # execute the script File "C:\Program Files\JetBrains\PyCharm 2018.2\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "D:/INSTALL/Urban Network Analysis Toolbox 1.01/src/Centrality/Adjacency_List_Computation.py", line 14, in from arcpy import AddField_management ModuleNotFoundError: No module named 'arcpy' May I found the following lines from the code? -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
RE: Python Console Menu
Take a look at the subprocess module for how to "spawn new processes, connect to their input/output/error pipes, and obtain their return codes." https://docs.python.org/2/library/subprocess.html -Original Message- From: Python-list [mailto:[email protected]] On Behalf Of Tcpip via Python-list Sent: Tuesday, July 31, 2018 10:56 AM To: [email protected] Subject: Python Console Menu Hi all, Im new with python, im working on a Python console Menu, I found some examples on Git, but what I need to understand is how I can call a subprocess. Here is an Example , if choice==1: print "Test SSH Connection (check ssh to all hosts)" ## You can add your code or functions here #print_menu() elif choice==2: print "Menu 2 has been selected" ## You can add your code or functions here elif choice==3: print "Menu 3 has been selected" ## You can add your code or functions here elif choice==4: print "Menu 4 has been selected" ## You can add your code or functions here elif choice==5: print "Menu 5 has been selected" ## You can add your code or function here Thanks. -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
RE: Dealing with errors in interactive subprocess running python interpreter that freeze the process
A couple notes: -I think the Python interpreter actually sends its output to stderr, so to capture it you'd probably want it to go to the same place as stdout, so use stderr = subprocess.STDOUT -You're only reading 1 line out output for each thing, so if 1 command creates multiple lines of output then you won't be showing them all. -You're never actually checking if the called process is still alive or not. It should normally be something like: ... interpreter = subprocess.Popen(...) ... interpreter.poll() while interpreter.returncode is not None: ... interpreter.poll() cleanup stuff -To the actual question it looks like it has the stdout stream in blocking mode somehow. So when you're reading from stdout and there's nothing there it's blocking and waiting for there to be something, which will never happen. Flipping through the documentation for subprocess, and io (interpreter.stdout is of ) io mentions blocking vs non blocking a lot, but it's not mentioned in subprocess. And I don't see in either how to tell if a stream is in blocking mode or not, or how or if it's possible to change that. So I don't know what to suggest for that, sorry. -Original Message- From: Python-list [mailto:[email protected]] On Behalf Of [email protected] Sent: Wednesday, August 01, 2018 4:11 PM To: [email protected] Subject: Dealing with errors in interactive subprocess running python interpreter that freeze the process I can run python3 interactively in a subprocess w/ Popen but if I sent it text, that throws an exception, the process freezes instead of just printing the exception like the normal interpreter.. why? how fix? Here is my code below. (I suspect when there is an exception, there is NO output to stdin so that the problem is the line below that tries to read from stdin never finishes. Maybe I need a different readline that can "survive" when there is no output and won't block?) import subprocess interpreter = subprocess.Popen(['python3', '-i'], stdin = subprocess.PIPE, stdout = subprocess.PIPE, stderr = subprocess.PIPE) while True: exp = input(">>> ").encode() + b"\n" interpreter.stdin.write(exp) interpreter.stdin.flush() print(interpreter.stdout.readline().strip()) interpreter.stdin.close() interpreter.terminate() -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
