Re: [Numpy-discussion] Numpy doesn't use RAM

An alternative solution may be https://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html If you are sure your subsequent computation against the array data has enough locality to avoid thrashing, I think numpy

Re: [Numpy-discussion] Numpy doesn't use RAM

Another thing to point out about having an array of that percentage of the available memory is that it severely restricts what you can do with it. Since you are above 50% of the available memory, you won't be able to create another array that would be the result of computing something with that arr

Re: [Numpy-discussion] Numpy doesn't use RAM

In addition to what Sebastian said about memory fragmentation and OS limits about memory allocations, I do think it will be hard to work with an array that close to the memory limit in NumPy regardless. Almost any operation will need to make a temporary array and exceed your memory limit. You mig

Re: [Numpy-discussion] Numpy doesn't use RAM

On Tue, 2020-03-24 at 13:59 -0400, Keyvis Damptey wrote: > Hi Numpy dev community, > > I'm keyvis, a statistical data scientist. > > I'm currently using numpy in python 3.8.2 64-bit for a clustering > problem, > on a machine with 1.9 TB RAM. When I try using np.zeros to create a > 600,000 > by 60

[Numpy-discussion] Numpy doesn't use RAM

Hi Numpy dev community, I'm keyvis, a statistical data scientist. I'm currently using numpy in python 3.8.2 64-bit for a clustering problem, on a machine with 1.9 TB RAM. When I try using np.zeros to create a 600,000 by 600,000 matrix of dtype=np.float32 it says "Unable to allocate 1.31 TiB for a