Re: [Tutor] memory error files over 100MB

2009-03-17 Thread Kent Johnson
On Tue, Mar 17, 2009 at 6:34 AM, A.T.Hofkamp wrote: >> http://personalpages.tds.net/~kent37/kk/00012.html > > Nice web-page! Thanks! > You can do the above statements also iteratively of course > > for i in ... >  s = read() >  # write s > > but since the loop does nothing with either s or read

Re: [Tutor] memory error files over 100MB

2009-03-17 Thread A.T.Hofkamp
Kent Johnson wrote: On Mon, Mar 16, 2009 at 12:30 PM, A.T.Hofkamp wrote: I don't know what code is executed in an assignment exactly, but **possibly**, first the 'read()' is executed (thus loading a very big string into memory), before assigning the value to the variable (which releases the pr

Re: [Tutor] memory error files over 100MB

2009-03-16 Thread Kent Johnson
On Mon, Mar 16, 2009 at 12:30 PM, A.T.Hofkamp wrote: > I don't know what code is executed in an assignment exactly, but > **possibly**, first the 'read()' is executed (thus loading a very big string > into memory), before assigning the value to the variable (which releases the > previous value of

Re: [Tutor] memory error files over 100MB

2009-03-16 Thread A.T.Hofkamp
Cheetah1000 wrote: I can't speak for Python 2.6, but using Jython 2.1 (Python 2.1 for Java), the code only looks at the file you are trying to extract()/read(). Near the end of the zip archive is a directory of all the files in the archive, with the start position and length of each file. Jytho

Re: [Tutor] memory error files over 100MB

2009-03-16 Thread Cheetah1000
Alan Gauld wrote: > > > "Sander Sweers" wrote > >> ... I would expect zf.read(zfilename) to only read the >> requested file in the zipfile. > > That's a dangerous assumption. You might be right but I'd want to > do some tests first to see. But if even one zipped file was big the > same wou

Re: [Tutor] memory error files over 100MB

2009-03-10 Thread Lie Ryan
Sander Sweers wrote: 2009/3/10 Alan Gauld : newFile.write(zf.read(zfilename)) Remember you are reading the file into memory and then writing it out again in a single operation, that will use twice the space of the uncompressed files - plus some extra for overhead. Question, Do you m

Re: [Tutor] memory error files over 100MB

2009-03-10 Thread Alan Gauld
"Sander Sweers" wrote out again in a single operation, that will use twice the space of the uncompressed files - plus some extra for overhead. Question, Do you mean the file in the zipfile (zfilename) or the whole zipfile (zf)? I would expect zf.read(zfilename) to only read the requested

Re: [Tutor] memory error files over 100MB

2009-03-10 Thread Sander Sweers
2009/3/10 Alan Gauld : >>           newFile.write(zf.read(zfilename)) > > Remember you are reading the file into memory and then writing it > out again in a single operation, that will use twice the space of the > uncompressed files - plus some extra for overhead. Question, Do you mean the file in

Re: [Tutor] memory error files over 100MB

2009-03-10 Thread Alan Gauld
"Harris, Sarah L" wrote However I still have a memory error when I try to run it on three or more files that are over 100 MB? And this surprises you? :-) How much memory do you have free on your computer when you run this? newFile.write(zf.read(zfilename)) Remember you are re

Re: [Tutor] memory error files over 100MB

2009-03-10 Thread Moos Heintzen
On Tue, Mar 10, 2009 at 8:45 AM, Harris, Sarah L wrote: > That looks better, thank you. > However I still have a memory error when I try to run it on three or more > files that are over 100 MB? How big are files in the zip file? It seems that in this line newFile.write(zf.read(zfilename)) the

[Tutor] memory error files over 100MB

2009-03-10 Thread Harris, Sarah L
That looks better, thank you. However I still have a memory error when I try to run it on three or more files that are over 100 MB? import zipfile, glob, os from os.path import isfile zipnames=filter(isfile, glob.glob('*.zip')) for zipname in zipnames: zf=zipfile.ZipFile(zipname, 'r')