Re: [Numpy-discussion] striding through arbitrarily large files

2014-02-05 Thread RayS
At 12:11 PM 2/5/2014, Richard Hattersley wrote: On 4 February 2014 15:01, RayS <r...@blue-cove.com> wrote: I was struggling with methods of reading large disk files into numpy efficiently (not FITS or .npy, just raw files of IEEE floats from numpy.tostring()). When l

Re: [Numpy-discussion] striding through arbitrarily large files

2014-02-05 Thread Richard Hattersley
On 4 February 2014 15:01, RayS wrote: > I was struggling with methods of reading large disk files into numpy > efficiently (not FITS or .npy, just raw files of IEEE floats from > numpy.tostring()). When loading arbitrarily large files it would be nice to > not bother reading more than the plot

Re: [Numpy-discussion] striding through arbitrarily large files

2014-02-04 Thread Sturla Molden
RayS wrote: > Thanks Daniele, I'll be trying mmap with Python64. With 32 bit the > mmap method throws MemoryError with 2.5GB files... > The idea is that we allow the users to inspect the huge files > graphically, then they can "zoom" into regions of interest and then > load a ~100 MB en block

Re: [Numpy-discussion] striding through arbitrarily large files

2014-02-04 Thread RayS
At 07:35 AM 2/4/2014, Julian Taylor wrote: On Tue, Feb 4, 2014 at 4:27 PM, RayS <r...@blue-cove.com> wrote: At 07:09 AM 2/4/2014, you wrote: >On 04/02/2014 16:01, RayS wrote: > > I was struggling with methods of reading large disk files into numpy > > efficiently (not

Re: [Numpy-discussion] striding through arbitrarily large files

2014-02-04 Thread Julian Taylor
On Tue, Feb 4, 2014 at 4:27 PM, RayS wrote: > At 07:09 AM 2/4/2014, you wrote: > >On 04/02/2014 16:01, RayS wrote: > > > I was struggling with methods of reading large disk files into numpy > > > efficiently (not FITS or .npy, just raw files of IEEE floats from > > > numpy.tostring()). When load

Re: [Numpy-discussion] striding through arbitrarily large files

2014-02-04 Thread RayS
At 07:09 AM 2/4/2014, you wrote: >On 04/02/2014 16:01, RayS wrote: > > I was struggling with methods of reading large disk files into numpy > > efficiently (not FITS or .npy, just raw files of IEEE floats from > > numpy.tostring()). When loading arbitrarily large files it would be nice > > to not

[Numpy-discussion] striding through arbitrarily large files

2014-02-04 Thread RayS
I was struggling with methods of reading large disk files into numpy efficiently (not FITS or .npy, just raw files of IEEE floats from numpy.tostring()). When loading arbitrarily large files it would be nice to not bother reading more than the plot can display before zooming in. There apparent

Re: [Numpy-discussion] striding through arbitrarily large files

2014-02-04 Thread Daniele Nicolodi
On 04/02/2014 16:01, RayS wrote: > I was struggling with methods of reading large disk files into numpy > efficiently (not FITS or .npy, just raw files of IEEE floats from > numpy.tostring()). When loading arbitrarily large files it would be nice > to not bother reading more than the plot can disp