Re: [Numpy-discussion] Change in memmap behaviour

2012-07-04 Thread Nathaniel Smith
On Tue, Jul 3, 2012 at 4:08 PM, Nathaniel Smith wrote: > On Tue, Jul 3, 2012 at 10:35 AM, Thouis (Ray) Jones wrote: >> On Mon, Jul 2, 2012 at 11:52 PM, Sveinung Gundersen >> wrote: >>> >>> On 2. juli 2012, at 22.40, Nathaniel Smith wrote: >>> On Mon, Jul 2, 2012 at 6:54 PM, Sveinung Gunder

Re: [Numpy-discussion] Change in memmap behaviour

2012-07-03 Thread Nathaniel Smith
On Tue, Jul 3, 2012 at 10:35 AM, Thouis (Ray) Jones wrote: > On Mon, Jul 2, 2012 at 11:52 PM, Sveinung Gundersen > wrote: >> >> On 2. juli 2012, at 22.40, Nathaniel Smith wrote: >> >>> On Mon, Jul 2, 2012 at 6:54 PM, Sveinung Gundersen >>> wrote: [snip] Your actual mem

Re: [Numpy-discussion] Change in memmap behaviour

2012-07-03 Thread Thouis (Ray) Jones
On Mon, Jul 2, 2012 at 11:52 PM, Sveinung Gundersen wrote: > > On 2. juli 2012, at 22.40, Nathaniel Smith wrote: > >> On Mon, Jul 2, 2012 at 6:54 PM, Sveinung Gundersen >> wrote: >>> [snip] >>> >>> >>> >>> Your actual memory usage may not have increased as much as you think, >>> since memmap obj

Re: [Numpy-discussion] Change in memmap behaviour

2012-07-02 Thread Sveinung Gundersen
On 2. juli 2012, at 22.40, Nathaniel Smith wrote: > On Mon, Jul 2, 2012 at 6:54 PM, Sveinung Gundersen wrote: >> [snip] >> >> >> >> Your actual memory usage may not have increased as much as you think, >> since memmap objects don't necessarily take much memory -- it sounds >> like you're leak

Re: [Numpy-discussion] Change in memmap behaviour

2012-07-02 Thread Nathaniel Smith
On Mon, Jul 2, 2012 at 6:54 PM, Sveinung Gundersen wrote: > [snip] > > > > Your actual memory usage may not have increased as much as you think, > since memmap objects don't necessarily take much memory -- it sounds > like you're leaking virtual memory, but your resident set size > shouldn't go up

Re: [Numpy-discussion] Change in memmap behaviour

2012-07-02 Thread Sveinung Gundersen
[snip] > > Your actual memory usage may not have increased as much as you think, > since memmap objects don't necessarily take much memory -- it sounds > like you're leaking virtual memory, but your resident set size > shouldn't go up as much. As I understand it, memmap objects retain the conten

Re: [Numpy-discussion] Change in memmap behaviour

2012-07-02 Thread Nathaniel Smith
On Mon, Jul 2, 2012 at 3:53 PM, Sveinung Gundersen wrote: > Hi, > > We are developing a large project for genome analysis > (http://hyperbrowser.uio.no), where we use memmap vectors as the basic data > structure for storage. The stored data are accessed in slices, and used as > basis for calculati

[Numpy-discussion] Change in memmap behaviour

2012-07-02 Thread Sveinung Gundersen
Hi, We are developing a large project for genome analysis (http://hyperbrowser.uio.no), where we use memmap vectors as the basic data structure for storage. The stored data are accessed in slices, and used as basis for calculations. As the stored data may be large (up to 24 GB), the memory foo