On Thu, May 19, 2011 at 1:53 AM, Pauli Virtanen wrote:
> On Wed, 18 May 2011 16:36:31 -0700, G Jones wrote:
> [clip]
> > As a followup, I managed to install tcmalloc as described in the article
> > I mentioned. Running the example I sent now shows a constant memory foot
> > print as expected. I a
On Wed, 18 May 2011 16:36:31 -0700, G Jones wrote:
[clip]
> As a followup, I managed to install tcmalloc as described in the article
> I mentioned. Running the example I sent now shows a constant memory foot
> print as expected. I am surprised such a solution was necessary.
> Certainly others must
Hello,
I have seen the effect you describe, I had originally assumed this was the
case, but in fact there seems to be more to the problem. If it were only the
effect you mention, there should not be any memory error because the OS will
drop the pages when the memory is actually needed for something
On Wed, 18 May 2011 15:09:31 -0700, G Jones wrote:
[clip]
> import numpy as np
>
> x = np.memmap('mybigfile.bin',mode='r',dtype='uint8') print x.shape #
> prints (42940071360,) in my case ndat = x.shape[0]
> for k in range(1000):
> y = x[k*ndat/1000:(k+1)*ndat/1000].astype('float32') #The ast
Hello,
I need to process several large (~40 GB) files. np.memmap seems ideal for
this, but I have run into a problem that looks like a memory leak or memory
fragmentation. The following code illustrates the problem
import numpy as np
x = np.memmap('mybigfile.bin',mode='r',dtype='uint8')
print x.s
