Re: [Tutor] Python and algorithms
On Thu, 18 Feb 2010 09:11:22 -0500 Kent Johnson wrote: > It's true that solving a problem often involves creating an algorithm > in a broad sense. The formal study of algorithms studies specific > techniques and algorithms that have proven to be useful to solve many > hard problems. In my experience most programming problems do not > require use of these formal algorithms, at least not explicitly. Hello, I would say that what is commonly called "algorithm" in computer science is a given class of possible algorithm that can (more easily) be formally expressed. Especially in mathematical terms. The reason why these are much more studied. But algorithmics can also be more generally understood as the "art & technique" of software design. Then, every programming task involves algorithmics. This may also be called "modelizing", a term than imo sensibly suggests how similar it is to the job of scientists. Modelizing is hard and hard to study because close to infinitely various and complex. Improving one's skills in this field is a whole life's yoga ;-) "I want to get a clearer mind"; "I want to become more lucid". An extremely big, difficult and rich book on the topic of thinking complexity is "la méthode" by http://en.wikipedia.org/wiki/Edgar_Morin (I don't have references to the english version). Denis la vita e estrany http://spir.wikidot.com/ ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] Reading large bz2 Files
Hello, i am trying to read a large bz2 file with this code: source_file = bz2.BZ2File(file, "r") for line in source_file: print line.strip() But after 4311 lines, it stoppes without a errormessage. The bz2 file is much bigger though. How can i read the whole file line by line? Thank you. Regards, Norman ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Wrestling with the Py2exe Install, Win7[XP!], Py2.5
Wayne, Somewhere in the tutorial should be a comment about py2exe creating two additional directories: build and dist. Forget about the build directory. If you look in the dist directory you will find the exe file and all supporting files. If you look back to an earlier email you will see a more detailed explanation I sent you. Robert Berman > -Original Message- > From: Wayne Watson [mailto:sierra_mtnv...@sbcglobal.net] > Sent: Thursday, February 18, 2010 11:10 PM > To: Robert Berman > Cc: tutor@python.org > Subject: Re: [Tutor] Wrestling with the Py2exe Install, Win7[XP!], > Py2.5 > > There's a bit of an anomaly. I've compiled 3 small programs now, and > in > cmd prompt a Dir does not find the file. It finds the py file, but > not > the completed file. Nevertheless, if I type in the prefix, the > desired > program executes. > > On 2/18/2010 4:48 PM, Wayne Watson wrote: > > Got it. Fooled myself. I'm converting to Win7 and have my XP > keyboard > > and monitor side by side with the same for XP. I did the world > program > > in XP and py2exe module in W7!! > > > > world compiled and ran successfully. Now for a bigger program with > > matplotlib and tkinter. Maybe I'll just settle for a small > matplotlib > > program for the moment. VBG > > > > Thanks very much. > > > > -- > "There is nothing so annoying as to have two people > talking when you're busy interrupting." -- Mark Twain ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] The Disappearing Program (py2exe)
I've successfully compiled several small python programs on Win XP into executables using py2exe. A program goes from a name like snowball.py to snowball. A dir in the command prompt window finds snowball.py but not snowball. If I type in snowball, it executes. What's up with that? -- "There is nothing so annoying as to have two people talking when you're busy interrupting." -- Mark Twain ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Wrestling with the Py2exe Install, Win7[XP!], Py2.5
Sneaky! :-) Yes, I now recall you mentioning it earlier. I'm looking at dist right now. I see a program I built in a folder above dist, pylab_scatter.exe. Interestingly, if I fire it up from the Win folder, a dos-window appears and it dies. A few lines appear too quickly to read. If I execute it from the command prompt, it works fine. Still the mystery to me is why I don't need to add exe in the cmd prompt to execute it. Further, how did it know to look in the dist folder? I checked with IDLE's path browser, and don't see it there. I'm not yet on board with the browser, but it looks like a starting point for finding py files I've run under IDLE. I looked for your post that had details, and I don't see it. I know you did post it. I had some trouble a few days ago trying to reply to one of your posts. It's not in my trash. Strange. The two directories are discussed just above section 4, right at the end of the a long output list. I missed that, since I thought the paragraph described the list, which I wasn't really interested in at the time. I would think that all the extras in dist are useful to other compiles of programs in my py folder? I have a comment about the tutorial. The command line shown a few lines into section 3. does not need python in the line in my case. setup.py py2exe works. I've glanced at section 5 and understand the basics. I'll be back to it later as needed. I didn't notice your reply in the queue this morning, and posted a msg about the disappearing file thinking there was a disconnect on the posts I had trouble with as above. I'll fix that post shortly. On 2/19/2010 5:34 AM, Robert Berman wrote: Wayne, Somewhere in the tutorial should be a comment about py2exe creating two additional directories: build and dist. Forget about the build directory. If you look in the dist directory you will find the exe file and all supporting files. If you look back to an earlier email you will see a more detailed explanation I sent you. Robert Berman -Original Message- From: Wayne Watson [mailto:sierra_mtnv...@sbcglobal.net] Sent: Thursday, February 18, 2010 11:10 PM To: Robert Berman Cc: tutor@python.org Subject: Re: [Tutor] Wrestling with the Py2exe Install, Win7[XP!], Py2.5 There's a bit of an anomaly. I've compiled 3 small programs now, and in cmd prompt a Dir does not find the file. It finds the py file, but not the completed file. Nevertheless, if I type in the prefix, the desired program executes. On 2/18/2010 4:48 PM, Wayne Watson wrote: Got it. Fooled myself. I'm converting to Win7 and have my XP keyboard and monitor side by side with the same for XP. I did the world program in XP and py2exe module in W7!! world compiled and ran successfully. Now for a bigger program with matplotlib and tkinter. Maybe I'll just settle for a small matplotlib program for the moment. VBG Thanks very much. -- "There is nothing so annoying as to have two people talking when you're busy interrupting." -- Mark Twain -- "There is nothing so annoying as to have two people talking when you're busy interrupting." -- Mark Twain ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] The Disappearing Program (py2exe)
The answer now appears in "Wrestling with ...". It's in a dist folder that py2exe produces. See Robert Berman post today, 2/19 early in morning. 5:34 am here. On 2/19/2010 7:00 AM, Wayne Watson wrote: I've successfully compiled several small python programs on Win XP into executables using py2exe. A program goes from a name like snowball.py to snowball. A dir in the command prompt window finds snowball.py but not snowball. If I type in snowball, it executes. What's up with that? -- "There is nothing so annoying as to have two people talking when you're busy interrupting." -- Mark Twain ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Reading large bz2 Files
On Fri, 19 Feb 2010 11:42:07 pm Norman Rieß wrote:
> Hello,
>
> i am trying to read a large bz2 file with this code:
>
> source_file = bz2.BZ2File(file, "r")
> for line in source_file:
> print line.strip()
>
> But after 4311 lines, it stoppes without a errormessage. The bz2 file
> is much bigger though.
>
> How can i read the whole file line by line?
"for line in file" works for me:
>>> import bz2
>>>
>>> writer = bz2.BZ2File('file.bz2', 'w')
>>> for i in xrange(2):
... # write some variable text to a line
... writer.write('abc'*(i % 5) + '\n')
...
>>> writer.close()
>>> reader = bz2.BZ2File('file.bz2', 'r')
>>> i = 0
>>> for line in reader:
... i += 1
...
>>> reader.close()
>>> i
2
My guess is one of two things:
(1) You are mistaken that the file is bigger than 4311 lines.
(2) You are using Windows, and somehow there is a Ctrl-Z (0x26)
character in the file, which Windows interprets as End Of File when
reading files in text mode. Try changing the mode to "rb" and see if
the behaviour goes away.
--
Steven D'Aprano
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Reading large bz2 Files
Norman Rieß, 19.02.2010 13:42: > i am trying to read a large bz2 file with this code: > > source_file = bz2.BZ2File(file, "r") > for line in source_file: > print line.strip() > > But after 4311 lines, it stoppes without a errormessage. What does "stops" mean here? Does it crash? Does it exit from the loop? Is the above code exactly what you used for testing? Are you passing a filename? What platform is this on? > The bz2 file is much bigger though. How many lines does it have? How did you count them? Did you make sure that you are reading from the right file? > How can i read the whole file line by line? Just as you do above, and it works for me. So the problem is likely elsewhere. Stefan ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Python and algorithms
spir wrote: On Thu, 18 Feb 2010 09:11:22 -0500 Kent Johnson wrote: It's true that solving a problem often involves creating an algorithm in a broad sense. The formal study of algorithms studies specific techniques and algorithms that have proven to be useful to solve many hard problems. In my experience most programming problems do not require use of these formal algorithms, at least not explicitly. Hello, I would say that what is commonly called "algorithm" in computer science is a given class of possible algorithm that can (more easily) be formally expressed. Especially in mathematical terms. The reason why these are much more studied. But algorithmics can also be more generally understood as the "art & technique" of software design. Then, every programming task involves algorithmics. This may also be called "modelizing", a term than imo sensibly suggests how similar it is to the job of scientists. Modelizing is hard and hard to study because close to infinitely various and complex. Improving one's skills in this field is a whole life's yoga ;-) "I want to get a clearer mind"; "I want to become more lucid". An extremely big, difficult and rich book on the topic of thinking complexity is "la méthode" by http://en.wikipedia.org/wiki/Edgar_Morin (I don't have references to the english version). Denis la vita e estrany http://spir.wikidot.com/ ___ Tutor maillist - To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor Thanks Denis, I'm realizing there is a problem with the definition of algorithm. Thanks for the link. T ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] The Disappearing Program (py2exe)
Wayne, I am assuming you are using Win 7 and I'll answer with that unless you tell me you are using XP in which case I will walk over to my wife's desk and test what I am telling you on her XP driven machine. Assuming Windows 7. Looking at your directory you should be using Windows Explorer. Click on Organize then click on folder and search options. Then click on view. Make sure the check box that says 'hide extensions for known file types' is not clicked. Once that is done, all your file extensions (all of them) will be seen and shown. Robert > -Original Message- > From: Wayne Watson [mailto:sierra_mtnv...@sbcglobal.net] > Sent: Friday, February 19, 2010 12:11 PM > To: Robert Berman > Subject: Re: [Tutor] The Disappearing Program (py2exe) > > I'm sure you are right. I'm not sure what you mean about view > params. > Properties? > > On 2/19/2010 7:08 AM, Robert Berman wrote: > > I think you will find that snowball is actually snowball.exe. You > might check > > the view parameters on how your extensions are being displayed. > > > > > >> -Original Message- > >> From: tutor-bounces+bermanrl=cfl.rr@python.org [mailto:tutor- > >> bounces+bermanrl=cfl.rr@python.org] On Behalf Of Wayne Watson > >> Sent: Friday, February 19, 2010 10:00 AM > >> To: tutor@python.org > >> Subject: [Tutor] The Disappearing Program (py2exe) > >> > >> I've successfully compiled several small python programs on Win > XP > >> into > >> executables using py2exe. A program goes from a name like > >> snowball.py to > >> snowball. A dir in the command prompt window finds snowball.py > but > >> not > >> snowball. If I type in snowball, it executes. What's up with > that? > >> > >> -- > >> "There is nothing so annoying as to have two people > >>talking when you're busy interrupting." -- Mark > Twain > >> > >> ___ > >> Tutor maillist - Tutor@python.org > >> To unsubscribe or change subscription options: > >> http://mail.python.org/mailman/listinfo/tutor > >> > > > > > > -- > "There is nothing so annoying as to have two people > talking when you're busy interrupting." -- Mark Twain ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Reading large bz2 Files
Am 19.02.2010 17:04, schrieb Steven D'Aprano: > My guess is one of two things: > (1) You are mistaken that the file is bigger than 4311 lines. > > (2) You are using Windows, and somehow there is a Ctrl-Z (0x26) > character in the file, which Windows interprets as End Of File when > reading files in text mode. Try changing the mode to "rb" and see if > the behaviour goes away. > Am 19.02.2010 17:15, schrieb Stefan Behnel: > What does "stops" mean here? Does it crash? Does it exit from the loop? Is > the above code exactly what you used for testing? Are you passing a > filename? What platform is this on? > > > How many lines does it have? How did you count them? Did you make sure that > you are reading from the right file? > > Hello, i took the liberty and copied your mails together, so i do not have to repeat things. How big is the file and how did i count that: sm...@loki ~/osm $ bzcat planet-100210.osm.bz2 | wc -l 1717362770 (this took a looong time ;-)) sm...@loki ~/osm $ du -h planet-100210.osm.bz2 8,0Gplanet-100210.osm.bz2 So as you can see, the file really is bigger. I am not using Windows and the next character would be a period. sm...@loki ~/osm/osmcut $ ./osmcut.py ../planet-100210.osm.bz2 [...] I did set the mode to "rb" with the same result. I also edited the code to see if the loop was exited or the program crashed. As you can see, there is no error, the loop just exits. This is the _exact_ code i use: source_file = bz2.BZ2File(osm_file, "r") for line in source_file: print line.strip() print "Exiting" print "I used file: " + osm_file As you can see above, the loop exits, the prints are executed and the right file is used. The content of the file is really distinctive, so there is no doubt, that it is the right file. Here is my platform information: Python 2.6.4 Linux 2.6.32.8 #1 SMP Fri Feb 12 13:29:10 CET 2010 x86_64 Intel(R) Core(TM)2 Duo CPU U9400 @ 1.40GHz GenuineIntel GNU/Linux Note: This symptome shows on another platform (SuSE 11.1) with different software versions as well. Is there a possibility, that the bz2 module reads only into a limited buffer and no further? If so, the same behaviour of the two independent systems would be explained and that it works in Stevens smaller example. How could i avoid that? Oh and the content of the file is free, so i do not get into legal issues exposing it. Thanks. Regards, Norman ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] Is it possible for a Python Program to send commands to the Python Interpreter?
I am trying to teach a computer program - to program. It builds grammars and procedural memories on dictionary networks. How do I get the program to be able to input to the interpreter/command line and read the results? I have done this a wee bit with making a dictionary of functions, but in a sense I am creating a script language on top of a script language so it seems really dumb to me especially since I have to recreate the grammar. I hope this doesn't get me flamed too much for asking... James ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Is it possible for a Python Program to send commands to the Python Interpreter?
On Fri, Feb 19, 2010 at 1:56 PM, Carnell, James E wrote: > > I am trying to teach a computer program - to program. It builds grammars and > procedural memories on dictionary networks. How do I get the program to be > able to input to the interpreter/command line and read the results? I have > done this a wee bit with making a dictionary of functions, but in a sense I > am creating a script language on top of a script language so it seems really > dumb to me especially since I have to recreate the grammar. It sounds like you are looking for eval() (Standard warning - use eval() only on trusted data) Kent ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Is it possible for a Python Program to send commands to thePython Interpreter?
"Carnell, James E" wrote I am trying to teach a computer program - to program. It builds grammars and procedural memories on dictionary networks. How do I get the program to be able to input to the interpreter/command line and read the results? You probably don;t want to do that. I suspect you just want to execute a string that your program has created? If so you can use the exec or eval functions in Python. If you do want to create an interactive prompt for your users you might like to look at the cmd module which includes the tools for building a command prompt type environment similar to the pdb debugger. I have done this a wee bit with making a dictionary of functions, but in a sense I am creating a script language on top of a script language so it seems really dumb to me especially since I have to recreate the grammar. Not really sure I understand? Are you creating a new grammar or are you using Python syntax etc but just adding new features/commands? If the outputr is valid python syntax then you can use exec/eval but if you are creating your own dialect then you will need to buld a parser - using any of the excellent python parsing tools available? HTH, -- Alan Gauld Author of the Learn to Program web site http://www.alan-g.me.uk/ ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Wrestling with the Py2exe Install, Win7[XP!], Py2.5
"Wayne Watson" wrote pylab_scatter.exe. Interestingly, if I fire it up from the Win folder, a dos-window appears and it dies. A few lines appear too quickly to read. If I execute it from the command prompt, it works fine. Still the mystery to me is why I don't need to add exe in the cmd prompt to execute it. DOS(*) automatically looks for executable file extensions (exe,com, bat etc) so you don't need to type them. You don't type cmd.exe to start a DOS shell do you? You only type cmd... I hope! (*)Actually the Windows command processor CMD.EXE Further, how did it know to look in the dist folder? Thats more interesting! It presumably added dist to the PATH environment variable. Not the Python path the DOS one - after all the EXE is not a python program any more its an exe file running under DOS. I have a comment about the tutorial. The command line shown a few lines into section 3. does not need python in the line in my case. setup.py py2exe works. That only works because you have the file asociation set to recognise .py files as associated with the interpreter. Using python explicitly removes that potential trip wire and so for a tutorial writer makes it a much safer option. HTH, -- Alan Gauld Author of the Learn to Program web site http://www.alan-g.me.uk/ ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Reading large bz2 Files
On 02/19/10 23:42, Norman Rieß wrote: > Hello, > > i am trying to read a large bz2 file with this code: > > source_file = bz2.BZ2File(file, "r") > for line in source_file: > print line.strip() > > But after 4311 lines, it stoppes without a errormessage. The bz2 file is > much bigger though. > How can i read the whole file line by line? Is the bz2 file an archive[1]? [1] archive: contains more than one file ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Reading large bz2 Files
Am 19.02.2010 21:42, schrieb Lie Ryan: > On 02/19/10 23:42, Norman Rieß wrote: > >> Hello, >> >> i am trying to read a large bz2 file with this code: >> >> source_file = bz2.BZ2File(file, "r") >> for line in source_file: >> print line.strip() >> >> But after 4311 lines, it stoppes without a errormessage. The bz2 file is >> much bigger though. >> How can i read the whole file line by line? >> > Is the bz2 file an archive[1]? > > [1] archive: contains more than one file > No it is a single file. But how could i check for sure? Its extracts to a single file... ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Reading large bz2 Files
On Fri, Feb 19, 2010 at 7:42 AM, Norman Rieß wrote: > Hello, > > i am trying to read a large bz2 file with this code: > > source_file = bz2.BZ2File(file, "r") > for line in source_file: > print line.strip() > > But after 4311 lines, it stoppes without a errormessage. The bz2 file is > much bigger though. > How can i read the whole file line by line? I wonder if it is dying after reading 2^31 or 2^32 bytes? It sounds a bit like this (fixed) bug: http://bugs.python.org/issue1215928 Kent ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Reading large bz2 Files
On 02/20/10 07:42, Lie Ryan wrote: > On 02/19/10 23:42, Norman Rieß wrote: >> Hello, >> >> i am trying to read a large bz2 file with this code: >> >> source_file = bz2.BZ2File(file, "r") >> for line in source_file: >> print line.strip() >> >> But after 4311 lines, it stoppes without a errormessage. The bz2 file is >> much bigger though. >> How can i read the whole file line by line? > > Is the bz2 file an archive[1]? > > [1] archive: contains more than one file Or more clearly, is the bz2 contains multiple file compressed using -c flag? The -c flag will do a simple concatenation of multiple compressed streams to stdout; it is only decompressible with bzip2 0.9.0 or later[1]. You cannot use bz2.BZ2File to open this, instead use the stream decompressor bz2.BZ2Decompressor. A better approach, is to use a real archiving format (e.g. tar). [1] http://www.bzip.org/1.0.3/html/description.html ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Reading large bz2 Files
On 02/20/10 07:49, Norman Rieß wrote: > Am 19.02.2010 21:42, schrieb Lie Ryan: >> On 02/19/10 23:42, Norman Rieß wrote: >> >>> Hello, >>> >>> i am trying to read a large bz2 file with this code: >>> >>> source_file = bz2.BZ2File(file, "r") >>> for line in source_file: >>> print line.strip() >>> >>> But after 4311 lines, it stoppes without a errormessage. The bz2 file is >>> much bigger though. >>> How can i read the whole file line by line? >>> >> Is the bz2 file an archive[1]? >> >> [1] archive: contains more than one file >> > > No it is a single file. But how could i check for sure? Its extracts to > a single file... use "bzip2 -dc" or "bunzip2" instead of "bzcat" since bzcat concatenates its output file to a single file. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Wrestling with the Py2exe Install, Win7[XP!], Py2.5
I suppose I'm in an interesting situation with regard to Win cmd prompt. I did this work on XP. There the facility is a bit more constraining than Win7 on my new PC. On XP, I do not have name completion w/o setting something. I only recently started with cmd prompt again. In Win7, it's automatic. There are other differences. An oddity, to me at least, name completion in W7 does not halt at the first difference. It goes all the way to completion at the first file that it can find, I think. I have to back up and try again. I think today will end my use of Python on XP. I have all files on Win7 now. I'll likely test py3exe there today to see how it behaves. What you say about the path change makes sense, but it's unfortunate the producers of py2exe haven't given some insight into this and the misc files produced in the dist folder. Of course, I have not Googled much at all on any of this. I'm glad I finally worked my way to this facility. It should help a good deal on the distribution of my demos to non-python friends, and fellow project workers at far flung places from here. On 2/19/2010 11:44 AM, Alan Gauld wrote: "Wayne Watson" wrote pylab_scatter.exe. Interestingly, if I fire it up from the Win folder, a dos-window appears and it dies. A few lines appear too quickly to read. If I execute it from the command prompt, it works fine. Still the mystery to me is why I don't need to add exe in the cmd prompt to execute it. DOS(*) automatically looks for executable file extensions (exe,com, bat etc) so you don't need to type them. You don't type cmd.exe to start a DOS shell do you? You only type cmd... I hope! (*)Actually the Windows command processor CMD.EXE Further, how did it know to look in the dist folder? Thats more interesting! It presumably added dist to the PATH environment variable. Not the Python path the DOS one - after all the EXE is not a python program any more its an exe file running under DOS. I have a comment about the tutorial. The command line shown a few lines into section 3. does not need python in the line in my case. setup.py py2exe works. That only works because you have the file asociation set to recognise .py files as associated with the interpreter. Using python explicitly removes that potential trip wire and so for a tutorial writer makes it a much safer option. HTH, -- "There is nothing so annoying as to have two people talking when you're busy interrupting." -- Mark Twain ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Is it possible for a Python Program to send commands to the Python Interpreter?
Thanks Kent and Alan!! Problem solved, eval() will work great! ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Reading large bz2 Files
Am 19.02.2010 22:03, schrieb Kent Johnson: > On Fri, Feb 19, 2010 at 7:42 AM, Norman Rieß wrote: > >> Hello, >> >> i am trying to read a large bz2 file with this code: >> >> source_file = bz2.BZ2File(file, "r") >> for line in source_file: >>print line.strip() >> >> But after 4311 lines, it stoppes without a errormessage. The bz2 file is >> much bigger though. >> How can i read the whole file line by line? >> > I wonder if it is dying after reading 2^31 or 2^32 bytes? It sounds a > bit like this (fixed) bug: > http://bugs.python.org/issue1215928 > > Kent > > ./osmcut.py ../planet-100210.osm.bz2 > test.txt sm...@loki ~/osm/osmcut $ ls -lh test.txt -rw-r--r-- 1 871K 19. Feb 22:41 test.txt Seems like far from it. Norman ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Wrestling with the Py2exe Install, Win7[XP!], Py2.5
Things were not quite what the seem. I just tried to run a program that was not converted, and left off py. It worked. So maybe the only way to execute the compiled code is to to to dist? -- "There is nothing so annoying as to have two people talking when you're busy interrupting." -- Mark Twain ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] The Disappearing Program (py2exe)
XP. Win 7, I hope, by the end of the day. Stuff is working under Win 7 from my transfer of it from a week or so ago. The only thing I left on XP was the py2exe stuff. I'm writing this from my Win 7 machine. There's a big twist in this. I've verified that when I typed the name w/o py, it really executed it anyway. I did that with a py file that was never setup by py2exe. I then went to dist and fired up the compiled exe file and got a complaint matplotlib. It couldn't find its data files. Certainly the step in section 4, test your executable, has been of no use. Maybe I need to go to step 5? Perhaps I need the msvcr71.dll file. Forget that.It's in dist. Time to read more of 5. On 2/19/2010 9:46 AM, Robert Berman wrote: Wayne, I am assuming you are using Win 7 and I'll answer with that unless you tell me you are using XP in which case I will walk over to my wife's desk and test what I am telling you on her XP driven machine. Assuming Windows 7. Looking at your directory you should be using Windows Explorer. Click on Organize then click on folder and search options. Then click on view. Make sure the check box that says 'hide extensions for known file types' is not clicked. Once that is done, all your file extensions (all of them) will be seen and shown. Robert -Original Message- From: Wayne Watson [mailto:sierra_mtnv...@sbcglobal.net] Sent: Friday, February 19, 2010 12:11 PM To: Robert Berman Subject: Re: [Tutor] The Disappearing Program (py2exe) I'm sure you are right. I'm not sure what you mean about view params. Properties? On 2/19/2010 7:08 AM, Robert Berman wrote: I think you will find that snowball is actually snowball.exe. You might check the view parameters on how your extensions are being displayed. -Original Message- From: tutor-bounces+bermanrl=cfl.rr@python.org [mailto:tutor- bounces+bermanrl=cfl.rr@python.org] On Behalf Of Wayne Watson Sent: Friday, February 19, 2010 10:00 AM To: tutor@python.org Subject: [Tutor] The Disappearing Program (py2exe) I've successfully compiled several small python programs on Win XP into executables using py2exe. A program goes from a name like snowball.py to snowball. A dir in the command prompt window finds snowball.py but not snowball. If I type in snowball, it executes. What's up with that? -- "There is nothing so annoying as to have two people talking when you're busy interrupting." -- Mark Twain ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor -- "There is nothing so annoying as to have two people talking when you're busy interrupting." -- Mark Twain -- "There is nothing so annoying as to have two people talking when you're busy interrupting." -- Mark Twain ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Reading large bz2 Files
Am 19.02.2010 22:24, schrieb Lie Ryan: > On 02/20/10 07:49, Norman Rieß wrote: > >> Am 19.02.2010 21:42, schrieb Lie Ryan: >> >>> On 02/19/10 23:42, Norman Rieß wrote: >>> >>> Hello, i am trying to read a large bz2 file with this code: source_file = bz2.BZ2File(file, "r") for line in source_file: print line.strip() But after 4311 lines, it stoppes without a errormessage. The bz2 file is much bigger though. How can i read the whole file line by line? >>> Is the bz2 file an archive[1]? >>> >>> [1] archive: contains more than one file >>> >>> >> No it is a single file. But how could i check for sure? Its extracts to >> a single file... >> > use "bzip2 -dc" or "bunzip2" instead of "bzcat" since bzcat concatenates > its output file to a single file. > > > Yes, it is a single file. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Is it possible for a Python Program to send commands to the Python Interpreter?
On Sat, 20 Feb 2010 06:33:04 am Kent Johnson wrote:
> It sounds like you are looking for eval()
>
> (Standard warning - use eval() only on trusted data)
This is the tutor list, aimed at beginners to Python, many of whom are
also beginners to programming as well. Even experienced programmers
often get security very, very, very badly wrong. Do you think that a
glib eight-word one-line "standard warning" really is sufficient?
James, if you are still reading, I should expand on Kent's warning.
The eval function, like exec, can be very dangerous in the wrong hands.
There is a whole class of very, very common security bugs caused by
functions like eval:
http://en.wikipedia.org/wiki/Code_injection
The way the bug works is that the programmer writes a function that
takes some data, and directly or indirectly applies eval to it:
>>> def mylist(string):
... # Convert a string to a list.
... string = string.strip()
... if string.startswith('[') and string.endswith(']'):
... return eval(string)
... else:
... raise ValueError('not a list')
...
>>> mylist(" [1, 2, 3] ")
[1, 2, 3]
This seems pretty innocuous, but it contains a deadly land-mine.
This function then gets used in an application that uses strings
produced by untrusted users. Say, it ends up in a web application, and
the user types text into a field and the application ends up calling
mylist on the contents of that field.
Then, some malicious user types this into the input form:
"[] or __import__('os').system('echo YOUR SYSTEM IS MINE') or [1,2,3]"
(only imagine something much worse than an echo command) and your web
application does this:
>>> s = "[] or __import__('os').system('echo YOUR SYSTEM IS MINE') or
[1,2,3]"
>>> mylist(s)
YOUR SYSTEM IS MINE
[1, 2, 3]
You have just had your web server compromised by a code injection bug.
(A web application is only one example of how you can get untrusted data
into your app. It is the biggest risk though.)
Now, you might think that you can work around this by clever
programming. Well, maybe, but sanitising strings so they are safe is a
VERY difficult job. And naturally if you aren't aware they need to be
sanitised, you won't do it.
My advice to anyone thinking they need to use eval or exec is:
(1) Don't do it.
(2) If you think you need to use them, you probably don't.
(3) If you really, really need to use them, then use the most
restrictive environment possible. Instead of eval(s), use:
eval(s, {'__builtins__': None}, {})
which gives you some protection against naive attackers. The really
clever ones will still break it.
(4) Sanitise your data. Don't use Javascript to sanitise it at the
browser, because the Bad Guys know how to bypass your Javascript
checks. If you're expecting (say) a list of integers, then there is no
reason for the list to contain *any* alphabetic characters or
underscores, and if there are any, reject the string and report an
error:
def sanitise(string):
safe = "1234567890[], \n\t"
for c in string:
if c not in safe:
raise ValueError('unsafe string')
If your needs are more complicated, then sanitising the string becomes
exponentially more difficult. It will probably be less work to write
your own safe parser than to sanitise the input.
Have I scared you about using eval? If so, good. Don't let eval or exec
anywhere near data provided by untrusted users, and don't confuse
authentication with trust.
--
Steven D'Aprano
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Is it possible for a Python Program to send commands to the Python Interpreter?
Looks like the eval() will work great. Thanks so much for telling me. > >We normally warn against using eval/exec because they form a security risk. But in your case you are in control of the code that they execute so its ok. That is the kind of usage where they are the best solution. Be very wary of cases where you read stored code from a file and execute it however. Its opssible a malicious user could add/change the code to do something bad. But provided you create the executing code yourself you should be fine. Alan G ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Is it possible for a Python Program to send commands to the Python Interpreter?
On Fri, Feb 19, 2010 at 7:07 PM, Steven D'Aprano wrote: > On Sat, 20 Feb 2010 06:33:04 am Kent Johnson wrote: > >> It sounds like you are looking for eval() >> >> (Standard warning - use eval() only on trusted data) > > > This is the tutor list, aimed at beginners to Python, many of whom are > also beginners to programming as well. Even experienced programmers > often get security very, very, very badly wrong. Do you think that a > glib eight-word one-line "standard warning" really is sufficient? No, but it is all I had time for, I figured James would ask for clarification or someone else would chime in :-) Kent ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] The Disappearing Program (py2exe)
OK, I'm completely on Win7. hello.exe works as expected by the tutorial. That file is in dist. I've now tried this program, pylab_scatter.py: #!/usr/bin/env python from pylab import * N = 30 x = 0.9*rand(N) y = 0.9*rand(N) area = pi*(10 * rand(N))**2 # 0 to 10 point radiuses scatter(x,y,s=area, marker='^', c='r') show() It runs properly in IDLE. == It appears to compile properly. Now from the cmd prompt window: C:\Users\Wayne\Sandia_Meteors\Sentinel_Development\Learn_Python\Py2exe_Test\dist>pylab_scatter.exe Traceback (most recent call last): File "pylab_scatter.py", line 2, in File "pylab.pyc", line 1, in File "matplotlib\__init__.pyc", line 677, in File "matplotlib\__init__.pyc", line 598, in rc_params File "matplotlib\__init__.pyc", line 552, in matplotlib_fname File "matplotlib\__init__.pyc", line 242, in wrapper File "matplotlib\__init__.pyc", line 482, in _get_data_path_cached File "matplotlib\__init__.pyc", line 478, in _get_data_path RuntimeError: Could not find the matplotlib data files <---What is this? C:\Users\Wayne\Sandia_Meteors\Sentinel_Development\Learn_Python\Py2exe_Test\dist> == I Googled this py2exe message RuntimeError: Could not find the matplotlib data files. As of yet, it does not look like a solution ia available for matplotlib. -- "There is nothing so annoying as to have two people talking when you're busy interrupting." -- Mark Twain ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] ask
How to describe a math formula: sphere=(4/3)*PI*R**3? -- Shurui Liu (Aaron Liu) Computer Science & Engineering Technology University of Toledo ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] ask
On Sat, Feb 20, 2010 at 8:07 AM, Shurui Liu (Aaron Liu) wrote: > How to describe a math formula: sphere=(4/3)*PI*R**3? A function seems like the logical thing to do: import math def spherical_volume(radius): return (4.0/3)*math.pi*radius**3 -- André Engels, andreeng...@gmail.com ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
