Re: Queue cleanup
Dennis Lee Bieber writes: > Not to mention having to ensure that one finds ALL the references to > the object so that they can be updated to the new address! Somehow I > don't see a C compiler being smart enough to find intermediary pointer We're not talking about C compilers, which can cast any value at all into pointers. Languages designed for garbage collection are normally type-safe and gc is a well-understood problem, though (like compilers) the higher-performing ones are complicated. But, nothing in principle stops anyone from implementing Python with such methods. -- http://mail.python.org/mailman/listinfo/python-list
Re: Financial time series data
On Fri, 2010-09-03 at 19:58 +0200, Virgil Stokes wrote:
> import urllib2
> import re
>
> def get_SP500_symbolsX ():
> symbols = []
> lsttradestr = re.compile('Last Trade:')
> k = 0
> for page in range(10):
>url = 'http://finance.yahoo.com/q/cp?s=%5EGSPC&c='+str(page)
>print url
>f = urllib2.urlopen (url)
>html = f.readlines ()
>f.close ()
>for line in html:
> if line.lstrip ().startswith (' line_split = line.split (':')
> s = [item.strip ().upper () for item in line_split [5].replace
> ('"','').split (',')]
> for symb in s:
> url = "http://finance.yahoo.com/q?s="+symb
> f = urllib2.urlopen(url)
> html = f.readlines()
> f.close()
>
> for line in html:
>if lsttradestr.search(line):
> k += 1
> print 'k = %3d (%s)' %(k,symb)
> # Here is where I will extract the numerical values and place
> #
> # them in an approrpriate file
> symbols.extend (s [:-3])
>
> return symbols
> # Not quite 500 -- which is correct (for example p. 2 has only 49
> symbols!)
> # Actually the S&P 500 as shown does not contain 500 stocks (symbols)
>
>
> symbols = get_SP500_symbolsX()
> pass
>
> And thanks for your help Frederic --- Have a good day! :-)
>
> --V
Good going! You get the idea.
Here's my try for a cleaned-up version that makes the best use of the
facility and takes only fifteen seconds to complete (on my machine).
You may want to look at historical quotes too. Trent Nelson seems to
have a ready-made solution for this.
---
import urllib2
import re
def get_current_SP500_quotes_from_Yahoo ():
symbol_reader = re.compile ('([a-z-.]+,)+[a-z-.]+')
# Make sure you include all characters that may show up in symbols,
csv_data = ''
for page in range (10):
url = 'http://finance.yahoo.com/q/cp?s=%5EGSPC&c=' + str (page)
print url
f = urllib2.urlopen (url)
html = f.readlines ()
f.close ()
for line in html:
if line.lstrip ().startswith ('http://mail.python.org/mailman/listinfo/python-list
Re: Financial time series data
I write some object for Taiwan Stock ... http://github.com/toomore/goristock But still dev ... On Sep 3, 1:12 am, Virgil Stokes wrote: > Has anyone written code or worked with Python software for downloading > financial time series data (e.g. from Yahoo financial)? If yes, would you > please contact me. > > --Thanks, > V. Stokes -- http://mail.python.org/mailman/listinfo/python-list
Re: IDLE / Black frame on Mac
On Sep 4, 5:19 am, Ned Deily wrote: > In article > , > Kristoffer Follesdal wrote: > > > *Forgot to tell that I am using a Mac with Snow Leopard. > > Which version of Python 3.1.2? From the python.org installer? > MacPorts? Built from source - if so, which version of Tk? > > -- > Ned Deily, > [email protected] I used the installer from python.org. Tk version 8.4. -- http://mail.python.org/mailman/listinfo/python-list
pyla: python little algorithms
Dear all, pyla stands for Python Little Algorithm is a project in pure Python and includes simple, easy to use, yet powerful libraries for - 2D/3D plotting using Gnuplot - Matrix/Vector operations - ODE solvers - Optimization and nonlinear algebraic equation solvers - ... Homepage: pyla home is: http://pyla.codeplex.com pyla documentation is on Tiddlywiki (www.tiddlywiki.org) a lightweight and portable wiki. Current Status: Currently the gplot library with status alpha has been released, other libraries is planned to be released bi-monthly. Contribution: We welcome all kind of contribution to this project. pyla is free, and open source.. All the best Mohammad Rahmani Chem Eng Dep Amirkabir Uni of Tech -- http://mail.python.org/mailman/listinfo/python-list
Re: Queue cleanup
[gc] In article <[email protected]>, Paul Rubin wrote: > >A minimal naive implementation indeed doubles the memory requirements, >but from a Python perspective where every integer takes something like >24 bytes already, even that doesn't seem so terrible. Many people still use 32-bit Python -- an int is twelve bytes there. -- Aahz ([email protected]) <*> http://www.pythoncraft.com/ "...if I were on life-support, I'd rather have it run by a Gameboy than a Windows box." --Cliff Wells -- http://mail.python.org/mailman/listinfo/python-list
Re: String substitution VS proper mysql escaping
In article , =?UTF-8?B?zp3Or866zr/Pgg==?= wrote: > >After all () used to define tuples and [] usedd to define lists. Why >commas? No, "()" does *not* define tuples, except for the empty tuple. The comma defines tuples, with parentheses simply used for visual effect: >>> 1, 2, 3 (1, 2, 3) -- Aahz ([email protected]) <*> http://www.pythoncraft.com/ "...if I were on life-support, I'd rather have it run by a Gameboy than a Windows box." --Cliff Wells -- http://mail.python.org/mailman/listinfo/python-list
State Machines in Python
Hi girls & guys, Just joined the group. I'm new to Python but been picking it up pretty easy. I love it! I'm hoping to use it to make a controlling application for an experiment. Basically I want to use it to interface with some data acquisition (DAQ) hardware to accept incoming signals and respond sending signals to the outputs. I'm looking for an efficient State Machine algorithm as I need the timing to be as good as possible. As there is no switch statement in Python, I've been looking around for a good implementation. Most of the algorithms I've come across seem to be based on parsing applications. I'd like something more suited to my needs. I'd like to avoid excessive use of 'if-elif-else' statements as each would have to be checked to find the right conditions which would have an time overhead involved. I have seen an implementation of the switch using dictionaries but someone had commented that creating and throwing away dictionaries also comes at a cost. I was wondering if any of you could advise or point me in the right direction. Any help would be greatly appreciated. Thanks, Jack -- http://mail.python.org/mailman/listinfo/python-list
Re: State Machines in Python
On Sat, 4 Sep 2010 14:36:38 +0100 Jack Keegan wrote: > Just joined the group. I'm new to Python but been picking it up pretty easy. Welcome aboard. > As there is no switch statement in Python, I've been looking around for a > good implementation. Most of the algorithms I've come across seem to be There's no switch statement because there's no real need for one. Check out the following sample code and see if it gives you some ideas. #! /usr/bin/env python # -*- coding: utf-8 -*- # Sample state machine import sys data = dict(counter = 0, flag = False) def state1(d): d['counter'] += 1 print "In state 1, counter = %(counter)d" % d if d['flag']: sys.exit(0) return state2 def state2(d): d['counter'] += 1 print "In state 2, counter = %(counter)d" % d return state3 def state3(d): d['counter'] += 1 d['flag'] = True print "In state 3, counter = %(counter)d" % d return state1 state = state1 while True: state = state(data) -- D'Arcy J.M. Cain | Democracy is three wolves http://www.druid.net/darcy/| and a sheep voting on +1 416 425 1212 (DoD#0082)(eNTP) | what's for dinner. -- http://mail.python.org/mailman/listinfo/python-list
Re: State Machines in Python
Jack Keegan, 04.09.2010 15:36: Hi girls& guys, Just joined the group. I'm new to Python but been picking it up pretty easy. I love it! Welcome to the group. I'm hoping to use it to make a controlling application for an experiment. Basically I want to use it to interface with some data acquisition (DAQ) hardware to accept incoming signals and respond sending signals to the outputs. I'm looking for an efficient State Machine algorithm as I need the timing to be as good as possible. As there is no switch statement in Python, I've been looking around for a good implementation. Most of the algorithms I've come across seem to be based on parsing applications. I'd like something more suited to my needs. I'd like to avoid excessive use of 'if-elif-else' statements as each would have to be checked to find the right conditions which would have an time overhead involved. I have seen an implementation of the switch using dictionaries but someone had commented that creating and throwing away dictionaries also comes at a cost. I was wondering if any of you could advise or point me in the right direction. Dictionaries are a common way to do it, and there are different recipes. A second way is a class that dispatches to its methods. A third (IMHO rather beautiful) way is coroutines: http://gnosis.cx/publish/programming/charming_python_b5.html http://www.dabeaz.com/coroutines/Coroutines.pdf However, you'll have to do some benchmarking if you care about performance. Dictionaries are fast and likely the fastest way to do it, but coroutines are a lot more versatile. Stackless Python might also be worth a look in this context, it's fast *and* versatile. You should also be aware that there are various event driven frameworks (like Twisted, eventlet and others) that make asynchronous event handling fast and easy, and that use much higher-level abstractions than pure state machines. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: State Machines in Python
On 04-09-2010 15:36, Jack Keegan wrote: > Hi girls & guys, > > Just joined the group. I'm new to Python but been picking it up pretty easy. > I love it! I'm hoping > to use it to make a controlling application for an experiment. Basically I > want to use it to > interface with some data acquisition (DAQ) hardware to accept incoming > signals and respond sending > signals to the outputs. I'm looking for an efficient State Machine algorithm > as I need the timing > to be as good as possible. What is as good as possible, 1 usec, 1 msec ? What operating system are you using ? Are you planning feedback ? For a comparison, I did a few years ago sampling in Python, with NI cards (they ensure time accuracy which can never be achieved in software), 50 kHz (divided over 1 to 8 channels), 32 bit, storage and graphical display, and processor activity was about 10%. Maybe you should also look at what those radio guys from gnu radio achive. cheers, Stef > As there is no switch statement in Python, I've been looking around for a > good implementation. > Most of the algorithms I've come across seem to be based on parsing > applications. I'd like > something more suited to my needs. I'd like to avoid excessive use of > 'if-elif-else' statements as > each would have to be checked to find the right conditions which would have > an time overhead > involved. I have seen an implementation of the switch using dictionaries but > someone had commented > that creating and throwing away dictionaries also comes at a cost. > I was wondering if any of you could advise or point me in the right direction. > > Any help would be greatly appreciated. > > Thanks, > > Jack -- http://mail.python.org/mailman/listinfo/python-list
Re: Installation problem: Python 2.6.6 (32-Bit) on Windows 7 (32-Bit)
Am 01.09.2010 21:18, schrieb Cappy2112: > Has anyone else had problems running the msi for Python 2.6.6 on > Windows 7 Professional? I specifically tested whether "compile .py" works before the release, and it worked fine on my machine. I suspect you have a source file on your disk that it tries to compile and it won't, because of a syntax error. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: what is this kind of string: b'string' ?
Am 01.09.2010 23:32, schrieb Stef Mientki: > in winpdb I see strings like this: > a = b'string' a > 'string' type(a) > > > what's the "b" doing in front of the string ? It's redundant. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: what is this kind of string: b'string' ?
Martin v. Loewis, 04.09.2010 18:52: Am 01.09.2010 23:32, schrieb Stef Mientki: in winpdb I see strings like this: >>> a = b'string' >>> a 'string' >>> type(a) what's the "b" doing in front of the string ? It's redundant. Not completely. (I know that you know this, but to those who don't, your answer may be misleading.) If you use 2to3 to convert the above to Python 3 code, it will leave the 'b' in front of the string, so the resulting string literal will be a bytes string in Python 3. If you remove it, the string will become a unicode literal. Since the code is syntax compatible with Python 3, simply running it in a Python 3 interpreter will also show this behaviour. So it's redundant in Python 2, but it's no longer redundant when you plan to migrate the code to Python 3. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: what is this kind of string: b'string' ?
Am 04.09.2010 19:27, schrieb Stefan Behnel:
> Martin v. Loewis, 04.09.2010 18:52:
>> Am 01.09.2010 23:32, schrieb Stef Mientki:
>>> in winpdb I see strings like this:
>>>
>>> >>> a = b'string'
>>> >>> a
>>> 'string'
>>> >>> type(a)
>>>
>>>
>>> what's the "b" doing in front of the string ?
>>
>> It's redundant.
>
> Not completely. (I know that you know this, but to those who don't, your
> answer may be misleading.)
Actually, I didn't think of this (knowing something and being aware of
it are different things ...)
I personally don't use it in the 2to3 way, because it requires Python
2.6. For code that needs to go back further, I typically do
b('string')
with a custom b() function. That's less efficient, of course, since it
causes a function call on evaluation.
Thanks,
Martin
--
http://mail.python.org/mailman/listinfo/python-list
Re: State Machines in Python
In article , "D'Arcy J.M. Cain" wrote: > On Sat, 4 Sep 2010 14:36:38 +0100 > Jack Keegan wrote: > > Just joined the group. I'm new to Python but been picking it up pretty easy. > > Welcome aboard. > > > As there is no switch statement in Python, I've been looking around for a > > good implementation. Most of the algorithms I've come across seem to be > > There's no switch statement because there's no real need for one. > Check out the following sample code and see if it gives you some ideas. > > #! /usr/bin/env python > # -*- coding: utf-8 -*- > > # Sample state machine > > import sys > > data = dict(counter = 0, flag = False) > > def state1(d): > d['counter'] += 1 > print "In state 1, counter = %(counter)d" % d > if d['flag']: sys.exit(0) > return state2 > > def state2(d): > d['counter'] += 1 > print "In state 2, counter = %(counter)d" % d > return state3 > > def state3(d): > d['counter'] += 1 > d['flag'] = True > print "In state 3, counter = %(counter)d" % d > return state1 > > state = state1 > while True: > state = state(data) This is the pattern I've always used. Simple and effective for any state machine which is small enough to code by hand. I generally have my state methods return (next_state, output) tuples, but that's a detail. -- http://mail.python.org/mailman/listinfo/python-list
Re: State Machines in Python
On 04/09/2010 18:58, Roy Smith wrote: In article, "D'Arcy J.M. Cain" wrote: On Sat, 4 Sep 2010 14:36:38 +0100 Jack Keegan wrote: Just joined the group. I'm new to Python but been picking it up pretty easy. Welcome aboard. As there is no switch statement in Python, I've been looking around for a good implementation. Most of the algorithms I've come across seem to be There's no switch statement because there's no real need for one. Check out the following sample code and see if it gives you some ideas. #! /usr/bin/env python # -*- coding: utf-8 -*- # Sample state machine import sys data = dict(counter = 0, flag = False) def state1(d): d['counter'] += 1 print "In state 1, counter = %(counter)d" % d if d['flag']: sys.exit(0) return state2 def state2(d): d['counter'] += 1 print "In state 2, counter = %(counter)d" % d return state3 def state3(d): d['counter'] += 1 d['flag'] = True print "In state 3, counter = %(counter)d" % d return state1 state = state1 while True: state = state(data) This is the pattern I've always used. Simple and effective for any state machine which is small enough to code by hand. I generally have my state methods return (next_state, output) tuples, but that's a detail. I suppose that if they are that similar then you could generate the code from a list or table of the states. -- http://mail.python.org/mailman/listinfo/python-list
Error in Following python program
#/usr/bin/python
from numpy import matrix
n=input('Enter matrix range')
fr=open('mat.txt','r')
print ('Enter elements into the matrix\n')
a=matrix([[input()for j in range(n)] for i in range(n)])
for i in range(n):
for j in range(n):
print a[i][j]
print '\n'
When i run the above program the following error is Coming please
Error is
Enter matrix range3
Enter elements into the matrix
1
2
3
4
5
6
7
8
9
[[1 2 3]]
Traceback (most recent call last):
File "2.py", line 10, in
print a[i][j]
File "/usr/lib/python2.6/dist-packages/numpy/core/defmatrix.py",
line 265, in __getitem__
out = N.ndarray.__getitem__
please resolve my problem Thanks in advance
~
--
http://mail.python.org/mailman/listinfo/python-list
Re: State Machines in Python
On Sat, 04 Sep 2010 13:58:00 -0400 Roy Smith wrote: > > while True: > > state = state(data) > > This is the pattern I've always used. Simple and effective for any > state machine which is small enough to code by hand. I generally have > my state methods return (next_state, output) tuples, but that's a detail. What is "output" for? Is it a string or something else? What do you do with it? Notice that I create a dictionary which is passed around so that states can pass whatever information back that they deem useful and any state can pick up whatever info it needs. for example, in my sample code every state uses the counter but only two states use the flag element. -- D'Arcy J.M. Cain | Democracy is three wolves http://www.druid.net/darcy/| and a sheep voting on +1 416 425 1212 (DoD#0082)(eNTP) | what's for dinner. -- http://mail.python.org/mailman/listinfo/python-list
Re: State Machines in Python
On Sat, 04 Sep 2010 19:13:28 +0100 MRAB wrote: > I suppose that if they are that similar then you could generate the > code from a list or table of the states. They generally aren't as simple as the little example script that I cobbled together. -- D'Arcy J.M. Cain | Democracy is three wolves http://www.druid.net/darcy/| and a sheep voting on +1 416 425 1212 (DoD#0082)(eNTP) | what's for dinner. -- http://mail.python.org/mailman/listinfo/python-list
Re: Error in Following python program
On 04/09/2010 19:28, Pramod wrote:
#/usr/bin/python
from numpy import matrix
n=input('Enter matrix range')
fr=open('mat.txt','r')
print ('Enter elements into the matrix\n')
a=matrix([[input()for j in range(n)] for i in range(n)])
for i in range(n):
for j in range(n):
print a[i][j]
print '\n'
When i run the above program the following error is Coming please
Error is
Enter matrix range3
Enter elements into the matrix
1
2
3
4
5
6
7
8
9
[[1 2 3]]
Traceback (most recent call last):
File "2.py", line 10, in
print a[i][j]
File "/usr/lib/python2.6/dist-packages/numpy/core/defmatrix.py",
line 265, in __getitem__
out = N.ndarray.__getitem__
please resolve my problem Thanks in advance
~
The matrix is 2-dimensional, which in numpy is means you need to write:
a[i, i]
not:
a[i][j]
and no, they're not the same! :-)
--
http://mail.python.org/mailman/listinfo/python-list
Re: State Machines in Python
D'Arcy J.M. Cain, 04.09.2010 20:30: On Sat, 04 Sep 2010 13:58:00 -0400 Roy Smith wrote: while True: state = state(data) This is the pattern I've always used. Simple and effective for any state machine which is small enough to code by hand. I generally have my state methods return (next_state, output) tuples, but that's a detail. What is "output" for? Is it a string or something else? What do you do with it? Notice that I create a dictionary which is passed around so that states can pass whatever information back that they deem useful and any state can pick up whatever info it needs. for example, in my sample code every state uses the counter but only two states use the flag element. I guess the idea is that each of the states can't arbitrarily modify the global status (dict) but is restricted to designating a next state and returning something. So you don't take the risk of introducing side effects somewhere because all state implementations are pure functions (at least as far as the state machine itself is concerned). Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Error in Following python program
Pramod wrote:
> #/usr/bin/python
> from numpy import matrix
> n=input('Enter matrix range')
> fr=open('mat.txt','r')
> print ('Enter elements into the matrix\n')
> a=matrix([[input()for j in range(n)] for i in range(n)])
> for i in range(n):
> for j in range(n):
> print a[i][j]
> print '\n'
>
> When i run the above program the following error is Coming please
> Error is
>Enter matrix range3
> Enter elements into the matrix
>
> 1
> 2
> 3
> 4
> 5
> 6
> 7
> 8
> 9
> [[1 2 3]]
> Traceback (most recent call last):
> File "2.py", line 10, in
> print a[i][j]
> File "/usr/lib/python2.6/dist-packages/numpy/core/defmatrix.py",
> line 265, in __getitem__
> out = N.ndarray.__getitem__
>
> please resolve my problem Thanks in advance
You can either use an array instead of a matrix and continue to access the
elements like you did in your code
>>> a = numpy.array([[1,2],[3,4]])
>>> a[1][1]
4
or continue to use the matrix and access its elements with a tuple
>>> b = numpy.matrix([[1,2],[3,4]])
>>> b[1,1]
4
If you pass only one index you get another, smaller matrix:
>>> b[1]
matrix([[3, 4]])
Once you see this printed it should be clear that b[1][1] asks for the non-
existent second row of the above matrix. Hence the error:
>>> b[1][1]
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.6/dist-packages/numpy/core/defmatrix.py", line 265,
in __getitem__
out = N.ndarray.__getitem__(self, index)
IndexError: index out of bounds
By the way, these matrices are really strange beasts:
>>> b[0][0]
matrix([[1, 2]])
>>> b[0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0]
matrix([[1, 2]])
Peter
--
http://mail.python.org/mailman/listinfo/python-list
Re: State Machines in Python
In article , "D'Arcy J.M. Cain" wrote: > On Sat, 04 Sep 2010 13:58:00 -0400 > Roy Smith wrote: > > > while True: > > > state = state(data) > > > > This is the pattern I've always used. Simple and effective for any > > state machine which is small enough to code by hand. I generally have > > my state methods return (next_state, output) tuples, but that's a detail. > > What is "output" for? Is it a string or something else? I've often used this pattern for parsing a text file and extracting interesting data. At various points during the input stream, you will have read an entire piece of data, and return it as the output of the state machine. Not every state will result in output being produced. As a trivial example, let's say I had a file format which stored network addresses in a deliberately silly style: # This is a comment host = 10.2.3.4 port = 999 proto = TCP port = 1001 proto = TCP host = 192.168.1.1 status = ignore host = 1.2.3.4 port = 22 host = 127.0.0.1 proto = UDP port = 1001 host = 192.168.1.1 I want to parse out (host, port, proto) triples, i.e. the state machine should produce the following output: (10.2.3.4, 9099, TCP) (192.168.1.1, 1001, TCP) (192.168.1.1, 1001, UDP) As the end of each data block is recognized, a 3-tuple would be returned. Other state transitions would return None as the output. Then, the main loop can be something like: state = start for line in input: next_state, output = state(line) if output: print output state = next_state I'm not saying it has to be done that way, just that I've found it a handy pattern for the stuff I've done. -- http://mail.python.org/mailman/listinfo/python-list
Re: Queue cleanup
Lawrence D'Oliveiro writes: > That reinforces my point, about how easy it was to check the correctness of > the code. In this case one simple fix, like this ... > would render the code watertight. See how easy it is? Well, no, it's irrelevant how easy it is to fix the issue after it's pointed out. What matters is how easy it was to create it in the first place. You posted a 30-line code sample as obviously free of memory leaks, but even a good programmer like you didn't notice that it had the potential for a nasty memory overwrite error after an unbalanced decref. Keep in mind that a memory leak usually just means the program can eventually bog down and stops working, but an overwrite error is often a security hole that can lead to total compromise of your entire computer. Now extrapolate that error rate from 30 lines to a program the size of Firefox (something like 5 MLOC), and you should see how fraught with danger that style of programming is. Even the most skilled and careful programmers are going to slip up once in a while. Part of the problem is C itself. No language can eliminate many complicated bugs without creating severe usability problems, but good languages (unlike C) can eliminate most "silly" bugs. I had a "dark night of the soul" after reading some of the bug-finding papers at http://www.stanford.edu/~engler/ and have been terrified of C ever since. I'm just always skeptical when anyone says they're sure any piece of C code is obviously bug-free. It's quite easy to get overconfident about it (as I've done myself more than once). I spent about 5 minutes reviewing your patched code (and the underlying implementations of the C API functions it calls) and didn't see any other issues, and the code is probably ok now, but I'd have to spend a lot more time tracing through the API layer before I could be really sure. Anyway, you should check your patch into github if you haven't. -- http://mail.python.org/mailman/listinfo/python-list
Re: Queue cleanup
On 8/28/2010 5:42 AM, Aahz wrote: In article<[email protected]>, Steven D'Aprano wrote: On Fri, 27 Aug 2010 09:16:52 -0700, Aahz wrote: In article, MRAB wrote: An object will be available for garbage collection when nothing refers to it either directly or indirectly. If it's unreferenced then it will go away. This isn't actually garbage collection as most people think of it. Refcounting semantics mean that objects get reaped as soon as nothing points at them. OTOH, CPython does also have garbage collection to back up refcounting so that when you have unreferenced object cycles they don't stay around. I've repeatedly asked, both here and elsewhere, why reference counting isn't "real" garbage collection. Nobody has been able to give me a satisfactory answer. As far as I can tell, it's a bit of pretentiousness with no basis in objective fact. You'll notice that I was very careful to qualify my statement with "as most people think of it". Also, because CPython has two different memory management mechanisms, refcounting and cycle detection, and the module that controls cycle detection is called "gc", I think it's simpler to follow along with the Python docs -- and critically important to remind people that there are in fact two different systems. Personally, I'd like to have reference counting only, an enforced prohibition on loops (backpointers must be weak pointers), RAII, and reliably ordered finalization. A big advantage of reference counting is that finalization happens in the thread that releases the object, and in the correct order. GC and finalization/destructors do not play well together at all. Microsoft once tried to get the hard cases to work right. See "managed C++". Not a happy story. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
the script is failing without a useful report
hi all, need your help. i get a traceback that doesn't tell much about the actual error in my code: Traceback (most recent call last): File ".\eightqueens.py", line 32, in generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) here is the full source: def generate(n, x, col, up, down): for h in range(8): # place a queen if col[h] and up[n-h] and down[n+h]: x[n] = h col[h] = False up[n-h] = False down[n+h] = False n += 1 # 8 queen placed? if n == 8: print x else: generate(n, x, col, up, down) # displace a queen n -= 1 col[h] = True up[n-h] = True down[n+h] = True if __name__ == '__main__': n = 0 x = [None]*8 col = [True]*8 up = [True]*15 down = [True]*15 generate(n, x, col, up, down) thanks a lot in advance -- http://mail.python.org/mailman/listinfo/python-list
Re: the script is failing without a useful report
On 04/09/2010 22:22, nvictor wrote: hi all, need your help. i get a traceback that doesn't tell much about the actual error in my code: Traceback (most recent call last): File ".\eightqueens.py", line 32, in generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) File ".\eightqueens.py", line 17, in generate else: generate(n, x, col, up, down) here is the full source: def generate(n, x, col, up, down): for h in range(8): # place a queen if col[h] and up[n-h] and down[n+h]: x[n] = h col[h] = False up[n-h] = False down[n+h] = False n += 1 # 8 queen placed? if n == 8: print x else: generate(n, x, col, up, down) # displace a queen n -= 1 col[h] = True up[n-h] = True down[n+h] = True if __name__ == '__main__': n = 0 x = [None]*8 col = [True]*8 up = [True]*15 down = [True]*15 generate(n, x, col, up, down) The traceback ends with: RuntimeError: maximum recursion depth exceeded I think what's happening is that if the 'if' condition just after the "# place a queen" comment is false then 'n' won't be incremented, and therefore 'generate' will call itself with the exactly same values which were passed in, resulting in infinite recursion (or at least until the limit is reached). -- http://mail.python.org/mailman/listinfo/python-list
Re: the script is failing without a useful report
thank you so much. -- http://mail.python.org/mailman/listinfo/python-list
Re: Speed-up for loops
Maybe for the simple sum you can just use the sum builtin: python -m timeit -s 'sum((10,)*1)' 1000 loops, best of 3: 0.0985 usec per loop About the loop in general it's a good practice to use list comprehension and generator expressions 2010/9/2 Michael Kreim > Hi, > > I was comparing the speed of a simple loop program between Matlab and > Python. > > My Codes: > $ cat addition.py > imax = 10 > a = 0 > for i in xrange(imax): >a = a + 10 > print a > > $ cat addition.m > imax = 1e9; > a = 0; > for i=0:imax-1 >a = a + 10; > end > disp(a); > exit; > > The results look like this: > $ /usr/bin/time --verbose python addition.py > 100 >Command being timed: "python addition.py" >User time (seconds): 107.30 >System time (seconds): 0.08 >Percent of CPU this job got: 97% >Elapsed (wall clock) time (h:mm:ss or m:ss): 1:50.09 >[...] > > $ /usr/bin/time --verbose matlab -nodesktop -nosplash -r "addition" > [...] >1.e+10 >Command being timed: "matlab -nodesktop -nosplash -r addition" >User time (seconds): 7.65 >System time (seconds): 0.18 >Percent of CPU this job got: 94% >Elapsed (wall clock) time (h:mm:ss or m:ss): 0:08.25 >[...] > > Unfortunately my Python Code was much slower and I do not understand why. > > Are there any ways to speed up the for/xrange loop? > Or do I have to live with the fact that Matlab beats Python in this > example? > > Thanks a lot for your answers. > > With best regards, > > Michael > > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Question about Reading Files
Hello. I am still really new to python and I have a project where I am trying to use the data files from another program and write a new program with new user interface and all. My first step was to open one of the files in 'rb' mode and print the contents, but I am unfamiliar with the format. Here is what was printed to the terminal: b'URES\x04\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0f \x00\x00\x00\x03\t\x00c\x01\x00\x00\x0c#\x00\x00\x02\x1b\x00\x00\x00Y \x00\x00\x00\x08\x98"\x00\x00t\x00\x00\x00\x01\'\x01\x00\x00z$ \x00\x00\x04,\xa7\x00\x00\xa1%\x00\x00\x05\x0b\x00\x00\x00o$\x00\x00\n \x11\x00\x00\x00\xcd\xcc\x00\x00\x0b\xf8\x00\x00\x00\xde\xcc \x00\x00\x0c\x19\x00\x00' I am using Python 3.1 on a Fedora 13 box if that makes any difference. Any advise on how to decode the data would be greatly appreciated. -- http://mail.python.org/mailman/listinfo/python-list
Re: Question about Reading Files
I forgot to mention that the output was the first 100 bytes of the output -- http://mail.python.org/mailman/listinfo/python-list
Re: Question about Reading Files
On 2010-09-04, genxtech wrote: > Hello. I am still really new to python and I have a project where I > am trying to use the data files from another program and write a new > program with new user interface and all. My first step was to open > one of the files in 'rb' mode and print the contents, but I am > unfamiliar with the format. Here is what was printed to the terminal: > > I am using Python 3.1 on a Fedora 13 box if that makes any difference. > Any advise on how to decode the data would be greatly appreciated. It's difficult to elaborate with only that information. What you have done now is opened a file in read binary mode (r = read, b = binary) and then tried to print it. Python has escaped the data as hex (\x01) and is basically a hex dump of the data file. For decoding the data, you either need to somehow figure out the format of the data and then decode it accordingly. If you're on unix box the 'file' command might be of help. If you're not on a unix box, you could check out how the file command tries to find the type of the file. The man page for magic [1] could be of help. Also see list of magic numbers [2] [1] http://linux.die.net/man/5/magic [2] http://www.astro.keele.ac.uk/oldusers/rno/Computing/File_magic.html -- http://mail.python.org/mailman/listinfo/python-list
Re: Question about Reading Files
On 05/09/2010 00:04, genxtech wrote: Hello. I am still really new to python and I have a project where I am trying to use the data files from another program and write a new program with new user interface and all. My first step was to open one of the files in 'rb' mode and print the contents, but I am unfamiliar with the format. Here is what was printed to the terminal: b'URES\x04\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0f \x00\x00\x00\x03\t\x00c\x01\x00\x00\x0c#\x00\x00\x02\x1b\x00\x00\x00Y \x00\x00\x00\x08\x98"\x00\x00t\x00\x00\x00\x01\'\x01\x00\x00z$ \x00\x00\x04,\xa7\x00\x00\xa1%\x00\x00\x05\x0b\x00\x00\x00o$\x00\x00\n \x11\x00\x00\x00\xcd\xcc\x00\x00\x0b\xf8\x00\x00\x00\xde\xcc \x00\x00\x0c\x19\x00\x00' I am using Python 3.1 on a Fedora 13 box if that makes any difference. Any advise on how to decode the data would be greatly appreciated. I googled and found this: http://stackoverflow.com/questions/2754751/what-file-format-contents-starts-with-ures which suggests "Universal Resource Editor (URE) that comes with the OS/2 Toolkit". -- http://mail.python.org/mailman/listinfo/python-list
Re: Question about Reading Files
On Sep 4, 7:23 pm, Mats Rauhala wrote: > On 2010-09-04, genxtech wrote: > > > Hello. I am still really new to python and I have a project where I > > am trying to use the data files from another program and write a new > > program with new user interface and all. My first step was to open > > one of the files in 'rb' mode and print the contents, but I am > > unfamiliar with the format. Here is what was printed to the terminal: > > > I am using Python 3.1 on a Fedora 13 box if that makes any difference. > > Any advise on how to decode the data would be greatly appreciated. > > It's difficult to elaborate with only that information. What you have > done now is opened a file in read binary mode (r = read, b = binary) and > then tried to print it. Python has escaped the data as hex (\x01) and is > basically a hex dump of the data file. > > For decoding the data, you either need to somehow figure out the format > of the data and then decode it accordingly. If you're on unix box the > 'file' command might be of help. > > If you're not on a unix box, you could check out how the file command > tries to find the type of the file. The man page for magic [1] could be > of help. Also see list of magic numbers [2] > > [1]http://linux.die.net/man/5/magic > [2]http://www.astro.keele.ac.uk/oldusers/rno/Computing/File_magic.html I am using Fedora 13. When I run the file command the response is that it is a 'data' file. If there are any tips on how to programatically figure out the format, I would greatly appreciate it. -- http://mail.python.org/mailman/listinfo/python-list
Re: Queue cleanup
In message , MRAB wrote: > Lawrence D'Oliveiro writes: >> >> Wonder why Sun’s licence explicitly forbade its use in danger-critical >> areas like nuclear power plants and the like, then? > > I thought it was just that if it wasn't explicitly forbidden then > someone might try to use it and then sue if something went wrong, even > though common sense would have said that it was a bad idea in the first > place! :-) But you’ll notice that Free Software comes with no such restrictions. In fact, it is contrary to commonly-accepted Free Software guidelines to impose any sort of restrictions on areas of use. -- http://mail.python.org/mailman/listinfo/python-list
Re: Queue cleanup
In message <[email protected]>, John Nagle wrote: > Personally, I'd like to have reference counting only, an enforced > prohibition on loops (backpointers must be weak pointers), RAII, > and reliably ordered finalization. Is there a cheap way of checking at runtime for circular structures? > A big advantage of reference counting is that finalization happens > in the thread that releases the object, and in the correct order. > GC and finalization/destructors do not play well together at all. > Microsoft once tried to get the hard cases to work right. See > "managed C++". Not a happy story. Thank you for that. Another arrow for my anti-GC quiver. :) -- http://mail.python.org/mailman/listinfo/python-list
Re: Queue cleanup
In message <[email protected]>, Paul Rubin wrote: > Lawrence D'Oliveiro writes: > >> In message <[email protected]>, Paul Rubin wrote: >> >>> GC's for large systems generally don't free (or even examine) individual >>> garbage objects. They copy the live objects to a new contiguous heap >>> without ever touching the garbage, and then they release the old heap. >> >> And suddenly you’ve doubled the memory requirements. And on top of that, >> since you’re moving the valid objects into different memory, you’re >> forcing cache misses on all of them as well. > > A minimal naive implementation indeed doubles the memory requirements, > but from a Python perspective where every integer takes something like > 24 bytes already, even that doesn't seem so terrible. Doubling 24 is less terrible than doubling 4 or 8?? You’re kidding, right? > More sophisticated implementations use multiple small heaps or other > tricks. More and more complications to patch up the idea. At some point, you have to admit there is something fundamentally flawed about the whole concept. > The new heap is filled sequentially so accesses to it will have good > locality. Unfortunately, that‘s not how locality of reference works. It doesn’t matter whether the objects being accessed are close together in memory or far apart (not with modern fully-associative caches, anyway), what does matter is the frequency distribution of references, namely that the vast majority of references are to a tiny minority of objects. Your generational garbage collector completely breaks this assumption, by regularly forcing an access to every single object in the heap. Cache- thrashing, anyone? > It's also the case that programs with very large memory consumption tend > to use most of the memory for large arrays that don't contain pointers > (think of a database server with a huge cache). That means the gc > doesn't really have to think about all that much of the memory. But your generational garbage collector still has to copy those very large objects to the new heap, with all the cache-hostile consequences therefrom. By the way, isn’t this the opposite of the array-of-pointers example you were using earlier to try to cast reference-counting in a bad light? It seems to me a reference count would work very well for such a large, simple object. >> This is the continuing problem with garbage collection: all the attempts >> to make it cheaper just end up moving the costs somewhere else. > > Same thing with manual allocation. That moves the costs off the > computer and onto the programmer. Not good, most of the time. Unfortunately, your argument falls down. It is a truism that hardware costs continue to come down, while programmers remain expensive. As I said before, computing performance has improved by something like five orders of magnitude over the last half-century. This has rendered all kinds of techniques, like high-level languages, dynamic memory allocation, stacks, hardware floating-point, memory protection and so on, which were once considered controversial because of their expense, cheap enough to become commonplace. But not garbage collection. This is because of the asymmetric way in which hardware has become faster: the biggest improvements have been in the parts that were already the fastest to begin with (the CPU), while RAM speeds have improved much less, and backing-store speeds least of all. Hence the need for intermediate layers of cache to bridge the gap. But the effectiveness of that caching depends crucially on certain assumptions about the runtime behaviour of the programs: and garbage collection breaks those assumptions. > Really, I'm no gc expert, but the stuff you're saying about gc is quite > ill-informed. You might want to check out some current literature. You may want to enlighten yourself by meditating on this seeming paradox of modern computing hardware: memory is cheap, but accessing memory is expensive. -- http://mail.python.org/mailman/listinfo/python-list
Re: Queue cleanup
Lawrence D'Oliveiro writes: >> A minimal naive implementation indeed doubles the memory requirements, >> but from a Python perspective where every integer takes something like >> 24 bytes already, even that doesn't seem so terrible. > > Doubling 24 is less terrible than doubling 4 or 8?? You’re kidding, right? No, it would be doubling 4 or 8 bytes. The extra overhead like the reference count would not be there to bloat up the integer like in Python. >> More sophisticated implementations use multiple small heaps or other tricks. > More and more complications to patch up the idea. At some point, you have to > admit there is something fundamentally flawed about the whole concept. Oh sheesh, that's silly. Every area of programming where performance matters is full of optimization tricks. Look at any serious database implementation for example. Or any compiler. Look at Python's implementation of dictionaries. Yeah, the optimizations add complexity to improve performance, sometimes even in heuristic ways that can fail. That doesn't mean the concepts are fundamentally flawed. GC is no different. >> The new heap is filled sequentially so accesses to it will have good >> locality. > what does matter is the frequency distribution of references, Sorry, just I meant during the gc operation itself. The gc's access pattern in the new heap is completely sequential as the gc just copies stuff to it linearly from from the old heap, bumping a pointer upwards. The access pattern when the user application is running is of course not predictable. > Your generational garbage collector completely breaks this assumption, by > regularly forcing an access to every single object in the heap. Cache- > thrashing, anyone? In the minor collections, the whole minor heap fits in cache, so there's no thrashing. The major collections can use a different strategy, or you can simply rely on their relative infrequency. Why do you speculate like this? If you run a GHC program with profiling active, it tells you exactly how much time is spent in minor gc and how much time is in major gc, and it's all generally pretty tolerable unless your program has bugs. (Unfortunately Haskell programs are notoriously susceptable to a certain type of bug that causes them to still give the right answers, but use much more memory than they should. The usual sign of that happening is high gc load). >> It's also the case that programs with very large memory consumption tend >> to use most of the memory for large arrays that don't contain pointers > But your generational garbage collector still has to copy those very large > objects to the new heap, with all the cache-hostile consequences therefrom. Not necessarily, depends on how you write the program and how the gc works. > By the way, isn’t this the opposite of the array-of-pointers example you > were using earlier to try to cast reference-counting in a bad light? I wasn't trying to cast reference counting in a bad light, I was pointing out that reference counting can experience pauses just like traditional gc approaches. Most programs including "soft" real time programs can tolerate an occasional pause. If your program is not one of those, and you need guaranteed upper bounds on pauses so you can't use traditional gc, switching from gc to reference counting won't save you. > It seems to me a reference count would work very well for such a > large, simple object. Mark/sweep would do it too. Some gc's use a hybrid approach, with mark/sweep for older or larger objects. > But not garbage collection. This is because of the asymmetric way in > which hardware has become faster:... the effectiveness of that > caching depends crucially on certain assumptions about the runtime > behaviour of the programs: and garbage collection breaks those > assumptions. ... > You may want to enlighten yourself by meditating on this seeming paradox of > modern computing hardware: memory is cheap, but accessing memory is > expensive. I'm interested in receiving enlightment if you've got some pointers into the research literature that back up your views. Right now it sounds like you're going by some intuitions you have that aren't backed up by evidence. Anyone who has performance-tuned a program knows that intuition can give reasonable starting points for experiments and measurements, but once the results start coming in, a lot of the intuitions end up invalidated. By now, there is enough experimental and theoretical literature about gc that opinions like yours, that don't seem to be grounded in any knowledge of that literature, are not very persuasive no matter how superficially attractive the raw intuition might be. Especially in your case, where you seem to have decided ahead of time what conclusion you want to reach and are looking for ways to justify it. -- http://mail.python.org/mailman/listinfo/python-list
Network and socket programming in python
i want to learn network and socket programming but i would like to do this in python.Reason behind this is that python is very simple and the only language i know . anybody can suggest me which book should i pick. the book should have following specification-- 1)not tedious to follow 2)lots of example 3)starts with some networking stuff and then get into codes thanks in advance with regards. -- http://mail.python.org/mailman/listinfo/python-list
Re: Queue cleanup
On 9/4/2010 6:44 PM, Lawrence D'Oliveiro wrote: In message<[email protected]>, John Nagle wrote: Personally, I'd like to have reference counting only, an enforced prohibition on loops (backpointers must be weak pointers), RAII, and reliably ordered finalization. Is there a cheap way of checking at runtime for circular structures? It's an interesting technical problem to design a system where circular references are detected immediately, at the moment of creation. However, Python already detects loops during garbage collection. If you set gc.set_debug(gc.DEBUG_SAVEALL) all the loops show up in "gc.garbage". A big advantage of reference counting is that finalization happens in the thread that releases the object, and in the correct order. GC and finalization/destructors do not play well together at all. Microsoft once tried to get the hard cases to work right. See "managed C++". Not a happy story. Thank you for that. Another arrow for my anti-GC quiver. :) Unoptimized reference counting, which is what CPython does, isn't all that great either. The four big bottlenecks in Python are boxed numbers, attribute lookups, reference count updates, and the GIL. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Subclassing by monkey-patching
I'm attempting to implement a recursive directory monitor based on the GIO file monitor in PyGTK. My approach is basically to take the gio.FileMonitor returned by the method gio.File.monitor_directory(), connect to the "changed" signal, and add or remove monitors on create/ delete events for subdirectories. I'd really like to do this by subclassing gio.FileMonitor. But the problem I'm having is that it's never explicitly initialised in user code (but somewhere in gio.File.monitor_directory() ). So there's no point in just declaring a subclass — where would I actually create an instance? Is there a way I can write the subclass but then somehow... extend an existing instance all at once rather than monkeypatch methods on one by one? So I could take an existing instance of a FileMonitor and make it an instance of my subclass? This would even allow me to override the gio.File.monitor_directory() method to take the monitor returned by the original method and decide whether to make it recursive based on a parameter passed to monitor_directory(). Cheers, Jason PS. Asked a similar question on the pygtk list a few days ago: http://www.daa.com.au/pipermail/pygtk/2010-September/018965.html -- http://mail.python.org/mailman/listinfo/python-list
Re: Speed-up for loops
On Thu, Sep 2, 2010 at 7:02 PM, Michael Kreim wrote: > Hi, > > I was comparing the speed of a simple loop program between Matlab and > Python. > > My Codes: > $ cat addition.py > imax = 10 > a = 0 > for i in xrange(imax): > a = a + 10 > print a > > $ cat addition.m > imax = 1e9; > a = 0; > for i=0:imax-1 > a = a + 10; > end > disp(a); > exit; > > The results look like this: > $ /usr/bin/time --verbose python addition.py > 100 > Command being timed: "python addition.py" > User time (seconds): 107.30 > System time (seconds): 0.08 > Percent of CPU this job got: 97% > Elapsed (wall clock) time (h:mm:ss or m:ss): 1:50.09 > [...] > > $ /usr/bin/time --verbose matlab -nodesktop -nosplash -r "addition" > [...] > 1.e+10 > Command being timed: "matlab -nodesktop -nosplash -r addition" > User time (seconds): 7.65 > System time (seconds): 0.18 > Percent of CPU this job got: 94% > Elapsed (wall clock) time (h:mm:ss or m:ss): 0:08.25 > [...] > > Unfortunately my Python Code was much slower and I do not understand why. Getting the above kind of code fast requires the interpreter to be clever enough so that it will use native machine operations on a int type instead of converting back and forth between internal representations. Matlab since version 6 I believe, has a JIT to do just that. There is no mature JIT-like implementation of python which will give you the same speed up for this exact case today. > Or do I have to live with the fact that Matlab beats Python in this example? Yes. Without a JIT, python cannot hope to get the same kind of speeds for this kind of examples. That being said, neither matlab nor matlab are especially good at doing what you do in your example - for this exact operation, doing it in C or other compiled languages will be at least one order of magnitude faster. Generally, you use matlab's vectorized operations, and in that case, numpy gives you similar performances (sometimes faster, sometimes slower, but in the same ballpark in general). cheers, David -- http://mail.python.org/mailman/listinfo/python-list
Re: Queue cleanup
John Nagle writes: > Unoptimized reference counting, which is what CPython does, isn't > all that great either. The four big bottlenecks in Python are boxed > numbers, attribute lookups, reference count updates, and the GIL. The performance hit of having to lock the refcounts before update has been the historical reason for keeping the GIL. The LOCK prefix takes something like 100 cycles on an x86. Is optimizing the refcount updates going to anywhere near make up for that? Python's "with" statement as an approach to RAII has seemed ok to me. I can't think of a time when I've really had to use a finalizer for something with dynamic extent. They've always seemed like a code smell to me. -- http://mail.python.org/mailman/listinfo/python-list
