> Since the files are huge, and would make me run out of memory, I need to
read data skipping some records
Is it possible to describe what you're doing with the data once you have
subsampled it? And if there were a way to work with the full resolution
data, would that be desirable?
I ask because
On 3/13/13 3:53 PM, Francesc Alted wrote:
> On 3/13/13 2:45 PM, Andrea Cimatoribus wrote:
>> Hi everybody, I hope this has not been discussed before, I couldn't
>> find a solution elsewhere.
>> I need to read some binary data, and I am using numpy.fromfile to do
>> this. Since the files are huge,
On 3/13/13 2:45 PM, Andrea Cimatoribus wrote:
> Hi everybody, I hope this has not been discussed before, I couldn't find a
> solution elsewhere.
> I need to read some binary data, and I am using numpy.fromfile to do this.
> Since the files are huge, and would make me run out of memory, I need to
Hi,
I would suggest that you look at pytables[1]. It use a different file
format, but it seam to do exactly what you want and give an object
that have a very similar interface to numpy.ndarray (but fewer
function). You would just ask for the slice/indices that you want and
it return you a numpy.nd
On Wed, Mar 13, 2013 at 1:45 PM, Andrea Cimatoribus
wrote:
> Hi everybody, I hope this has not been discussed before, I couldn't find a
> solution elsewhere.
> I need to read some binary data, and I am using numpy.fromfile to do this.
> Since the files are huge, and would make me run out of memo
Hi everybody, I hope this has not been discussed before, I couldn't find a
solution elsewhere.
I need to read some binary data, and I am using numpy.fromfile to do this.
Since the files are huge, and would make me run out of memory, I need to read
data skipping some records (I am reading data re
