A Thursday 11 March 2010 14:35:49 Gael Varoquaux escrigué:
> > So, in my experience, numpy.memmap is really using that large chunk of
> > memory (unless my testbed is badly programmed, in which case I'd be
> > grateful if you can point out what's wrong).
>
> OK, so what you are saying is that my a
On Thu, Mar 11, 2010 at 02:26:49PM +0100, Francesc Alted wrote:
> > I believe that your above assertion is 'half' right. First I think that
> > it is not SWAP that the memapped file uses, but the original disk space,
> > thus you avoid running out of SWAP. Second, if you open several times the
> >
A Thursday 11 March 2010 10:36:42 Gael Varoquaux escrigué:
> On Thu, Mar 11, 2010 at 10:04:36AM +0100, Francesc Alted wrote:
> > As far as I know, memmap files (or better, the underlying OS) *use* all
> > available RAM for loading data until RAM is exhausted and then start to
> > use SWAP, so the "
Nadav
-Original Message-
From: numpy-discussion-boun...@scipy.org on behalf of Gael Varoquaux
Sent: Thu 11-Mar-10 11:36
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] multiprocessing shared arrays and numpy
On Thu, Mar 11, 2010 at 10:04:36AM +0100, Francesc A
On Thu, Mar 11, 2010 at 10:04:36AM +0100, Francesc Alted wrote:
> As far as I know, memmap files (or better, the underlying OS) *use* all
> available RAM for loading data until RAM is exhausted and then start to use
> SWAP, so the "memory pressure" is still there. But I may be wrong...
I believ
A Sunday 07 March 2010 20:03:21 Gael Varoquaux escrigué:
> On Sun, Mar 07, 2010 at 07:00:03PM +, René Dudfield wrote:
> > 1. Mmap'd files are useful since you can reuse disk cache as program
> > memory. So large files don't waste ram on the disk cache.
>
> I second that. mmaping has worked ve
On Sun, Mar 07, 2010 at 07:00:03PM +, René Dudfield wrote:
> 1. Mmap'd files are useful since you can reuse disk cache as program
> memory. So large files don't waste ram on the disk cache.
I second that. mmaping has worked very well for me for large datasets,
especialy in the context of re
On Fri, Mar 5, 2010 at 7:29 PM, Brian Granger wrote:
> Francesc,
>
>> Yeah, 10% of improvement by using multi-cores is an expected figure for
>> memory
>> bound problems. This is something people must know: if their computations
>> are
>> memory bound (and this is much more common that one may in
the memory mapped
file from the child process.
Nadav
-Original Message-
From: numpy-discussion-boun...@scipy.org on behalf of Brian Granger
Sent: Fri 05-Mar-10 21:29
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] multiprocessing shared arrays and numpy
Francesc,
Yeah
> > -Original Message-
> > From: numpy-discussion-boun...@scipy.org on behalf of Francesc Alted
> > Sent: Thu 04-Mar-10 15:12
> > To: Discussion of Numerical Python
> > Subject: Re: [Numpy-discussion] multiprocessing shared arrays and numpy
> >
> > What kind o
A Friday 05 March 2010 14:46:00 Gael Varoquaux escrigué:
> On Fri, Mar 05, 2010 at 08:14:51AM -0500, Francesc Alted wrote:
> > > FWIW, I observe very good speedups on my problems (pretty much linear
> > > in the number of CPUs), and I have data parallel problems on fairly
> > > large data (~100Mo a
On Fri, Mar 05, 2010 at 08:14:51AM -0500, Francesc Alted wrote:
> > FWIW, I observe very good speedups on my problems (pretty much linear in
> > the number of CPUs), and I have data parallel problems on fairly large
> > data (~100Mo a piece, doesn't fit in cache), with no synchronisation at
> > all
Gael,
On Fri, Mar 05, 2010 at 10:51:12AM +0100, Gael Varoquaux wrote:
> On Fri, Mar 05, 2010 at 09:53:02AM +0100, Francesc Alted wrote:
> > Yeah, 10% of improvement by using multi-cores is an expected figure for
> > memory bound problems. This is something people must know: if their
> > computati
On Fri, Mar 05, 2010 at 09:53:02AM +0100, Francesc Alted wrote:
> Yeah, 10% of improvement by using multi-cores is an expected figure for
> memory bound problems. This is something people must know: if their
> computations are memory bound (and this is much more common that one
> may initially thi
t;
> Nadav.
>
>
> -Original Message-
> From: numpy-discussion-boun...@scipy.org on behalf of Francesc Alted
> Sent: Thu 04-Mar-10 15:12
> To: Discussion of Numerical Python
> Subject: Re: [Numpy-discussion] multiprocessing shared arrays and numpy
>
> What ki
: [Numpy-discussion] multiprocessing shared arrays and numpy
What kind of calculations are you doing with this module? Can you please send
some examples and the speed-ups you are getting?
Thanks,
Francesc
A Thursday 04 March 2010 14:06:34 Nadav Horesh escrigué:
> Extended module that I u
oresh
> Sent: Thu 04-Mar-10 11:55
> To: Discussion of Numerical Python
> Subject: RE: [Numpy-discussion] multiprocessing shared arrays and numpy
>
> Maybe the attached file can help. Adpted and tested on amd64 linux
>
> Nadav
>
>
> -Original Message-
> From: n
-boun...@scipy.org on behalf of Nadav Horesh
Sent: Thu 04-Mar-10 11:55
To: Discussion of Numerical Python
Subject: RE: [Numpy-discussion] multiprocessing shared arrays and numpy
Maybe the attached file can help. Adpted and tested on amd64 linux
Nadav
-Original Message-
From: numpy-discussion
Maybe the attached file can help. Adpted and tested on amd64 linux
Nadav
-Original Message-
From: numpy-discussion-boun...@scipy.org on behalf of Nadav Horesh
Sent: Thu 04-Mar-10 10:54
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] multiprocessing shared arrays and
There is a work by Sturla Molden: look for multiprocessing-tutorial.pdf
and sharedmem-feb13-2009.zip. The tutorial includes what is dropped in
the cookbook page. I am into the same issue and going to test it today.
Nadav
On Wed, 2010-03-03 at 15:31 +0100, Jesper Larsen wrote:
> Hi people,
>
>
A Wednesday 03 March 2010 15:31:29 Jesper Larsen escrigué:
> Hi people,
>
> I was wondering about the status of using the standard library
> multiprocessing module with numpy. I found a cookbook example last
> updated one year ago which states that:
>
> "This page was obsolete as multiprocessing'
Hi people,
I was wondering about the status of using the standard library
multiprocessing module with numpy. I found a cookbook example last
updated one year ago which states that:
"This page was obsolete as multiprocessing's internals have changed.
More information will come shortly; a link to t
22 matches
Mail list logo
