Re: [Numpy-discussion] R: fast numpy.fromfile skipping data chunks

2013-03-13 Thread Charles R Harris
On Wed, Mar 13, 2013 at 9:54 AM, Andrea Cimatoribus < andrea.cimatori...@nioz.nl> wrote: > Thanks a lot for the feedback, I'll try to modify my function to overcome > this issue. > Since I'm in the process of buying new hardware too, a slight OT (but > definitely related). > Does an ssd provide su

Re: [Numpy-discussion] R: fast numpy.fromfile skipping data chunks

2013-03-13 Thread Daπid
On 13 March 2013 16:54, Andrea Cimatoribus wrote: > Since I'm in the process of buying new hardware too, a slight OT (but > definitely related). > Does an ssd provide substantial improvement in these cases? > It should help. Nevertheless, when talking about performance, it is difficult to predic

[Numpy-discussion] R: fast numpy.fromfile skipping data chunks

2013-03-13 Thread Andrea Cimatoribus
Thanks a lot for the feedback, I'll try to modify my function to overcome this issue. Since I'm in the process of buying new hardware too, a slight OT (but definitely related). Does an ssd provide substantial improvement in these cases? Da: numpy-discussio

Re: [Numpy-discussion] R: fast numpy.fromfile skipping data chunks

2013-03-13 Thread Nathaniel Smith
On Wed, Mar 13, 2013 at 2:18 PM, Andrea Cimatoribus wrote: > This solution does not work for me since I have an offset before the data > that is not a multiple of the datatype (it's a header containing various > stuff). np.memmap takes an offset= argument. -n __

[Numpy-discussion] R: fast numpy.fromfile skipping data chunks

2013-03-13 Thread Andrea Cimatoribus
I see that pytables deals with hdf5 data. It would be very nice if the data were in such a standard format, but that is not the case, and that can't be changed. Da: numpy-discussion-boun...@scipy.org [numpy-discussion-boun...@scipy.org] per conto di Fréd

[Numpy-discussion] R: fast numpy.fromfile skipping data chunks

2013-03-13 Thread Andrea Cimatoribus
This solution does not work for me since I have an offset before the data that is not a multiple of the datatype (it's a header containing various stuff). I'll at pytables. # Exploit the operating system's virtual memory manager to get a "virtual copy" of the entire file in memory # (This does no