Re: [Numpy-discussion] Efficient reading of binary data

2008-04-04 Thread Christopher Barker
Nicolas Bigaouette wrote: > So the next step would be to only read the needed data from the binary > file... You've gotten some suggestions, but another option is to use file.seek(0 to get where your data is, and numpy.fromfile() from there. -CHB -- Christopher Barker, Ph.D. Oceanographer E

Re: [Numpy-discussion] Efficient reading of binary data

2008-04-04 Thread Sebastian Haase
On Fri, Apr 4, 2008 at 11:33 AM, Jarrod Millman <[EMAIL PROTECTED]> wrote: > On Fri, Apr 4, 2008 at 1:50 AM, Sebastian Haase <[EMAIL PROTECTED]> wrote: > > Hi, > > Accidentally I'm exactly trying to do the same thing right now . > > > > What is the best way of memmapping into a file that

Re: [Numpy-discussion] Efficient reading of binary data

2008-04-04 Thread Jarrod Millman
On Fri, Apr 4, 2008 at 1:50 AM, Sebastian Haase <[EMAIL PROTECTED]> wrote: > Hi, > Accidentally I'm exactly trying to do the same thing right now . > > What is the best way of memmapping into a file that is already open !? > > I have to read some text (header info) off the beginning of the

Re: [Numpy-discussion] Efficient reading of binary data

2008-04-04 Thread Sebastian Haase
On Fri, Apr 4, 2008 at 2:14 AM, Nicolas Bigaouette <[EMAIL PROTECTED]> wrote: > 2008/4/3, Robert Kern <[EMAIL PROTECTED]>: > > > On Thu, Apr 3, 2008 at 6:53 PM, Nicolas Bigaouette > > <[EMAIL PROTECTED]> wrote: > > > Thanx for the fast response Robert ;) > > > > > > I changed my code to use the sli

Re: [Numpy-discussion] Efficient reading of binary data

2008-04-03 Thread Nicolas Bigaouette
2008/4/3, Robert Kern <[EMAIL PROTECTED]>: > > On Thu, Apr 3, 2008 at 6:53 PM, Nicolas Bigaouette > <[EMAIL PROTECTED]> wrote: > > Thanx for the fast response Robert ;) > > > > I changed my code to use the slice: > > E = data[6::9]It is indeed faster and less eat less memory. Great. > > > > Thanx

Re: [Numpy-discussion] Efficient reading of binary data

2008-04-03 Thread Robert Kern
On Thu, Apr 3, 2008 at 6:53 PM, Nicolas Bigaouette <[EMAIL PROTECTED]> wrote: > Thanx for the fast response Robert ;) > > I changed my code to use the slice: > E = data[6::9]It is indeed faster and less eat less memory. Great. > > Thanx for the endiannes! I knew there was something like this ;) I

Re: [Numpy-discussion] Efficient reading of binary data

2008-04-03 Thread Nicolas Bigaouette
Thanx for the fast response Robert ;) I changed my code to use the slice: E = data[6::9] It is indeed faster and less eat less memory. Great. Thanx for the endiannes! I knew there was something like this ;) I suspect that, in '>f8', "f" means float and "8" means 8 bytes? >From some benchmarks, I

Re: [Numpy-discussion] Efficient reading of binary data

2008-04-03 Thread Robert Kern
On Thu, Apr 3, 2008 at 3:30 PM, Nicolas Bigaouette <[EMAIL PROTECTED]> wrote: > Hi, > > I have a C program which outputs large (~GB) files. It is a simple binary > dump of an array of structure containing 9 doubles. You can see this as a > double 1D array of size 9*Stot (Stot being the allocated si

[Numpy-discussion] Efficient reading of binary data

2008-04-03 Thread Nicolas Bigaouette
Hi, I have a C program which outputs large (~GB) files. It is a simple binary dump of an array of structure containing 9 doubles. You can see this as a double 1D array of size 9*Stot (Stot being the allocated size of the array of structure). The 1D array represents a 3D array (Sx * Sy * Sz = Stot)