Re: [Numpy-discussion] parallel compilation of numpy

On Thu, Feb 19, 2009 at 12:47 AM, David Cournapeau wrote: > On Thu, Feb 19, 2009 at 4:26 PM, Andrew Straw wrote: > > > Maybe if you need a level of backward compatibility, (and really, to > > gain a decent audience for this idea, I think you do need some level of > > backward compatibility) the n

Re: [Numpy-discussion] parallel compilation of numpy

On 2/19/2009 7:04 AM, David Cournapeau wrote: > Does pyprocessing work well on windows as well ? I have 0 experience > with it. Yes it works well on Windows, albeit process creation is a bit slower than on Unix (there is no os.fork in Windows, so more Python objects has to be pickled). From Pyt

Re: [Numpy-discussion] parallel compilation of numpy

On Thu, Feb 19, 2009 at 4:26 PM, Andrew Straw wrote: > Maybe if you need a level of backward compatibility, (and really, to > gain a decent audience for this idea, I think you do need some level of > backward compatibility) the new tool could emit setup.py files for > consumption by distutils as

Re: [Numpy-discussion] parallel compilation of numpy

David Cournapeau wrote: > On Thu, Feb 19, 2009 at 3:24 PM, Andrew Straw wrote: > >> It's an interesting idea to build Python package distributions without >> distutils. For pure Python installables, if all you seek better is >> distutils, the bar seems fairly low. >> > > :) Being better th

Re: [Numpy-discussion] parallel compilation of numpy

David Cournapeau wrote: > Michael Abshoff wrote: Hi David, >> Sure, it also works for incremental builds and I do that many, many >> times a day, i.e. for each patch I merge into the Sage library. What >> gets recompiled is decided by our own dependency tracking code which we >> want to push i

Re: [Numpy-discussion] parallel compilation of numpy

On Thu, Feb 19, 2009 at 3:24 PM, Andrew Straw wrote: > David Cournapeau wrote: >>> * Integration with setuptools and eggs, which enables things like >>> namespace packages. >>> >> >> This is not. eggs are not specified, and totally implementation defined. >> I tried some time ago to add an egg bu

Re: [Numpy-discussion] parallel compilation of numpy

David Cournapeau wrote: >> * Integration with setuptools and eggs, which enables things like >> namespace packages. >> > > This is not. eggs are not specified, and totally implementation defined. > I tried some time ago to add an egg builder to scons, but I gave up. > And I don't think you can

Re: [Numpy-discussion] parallel compilation of numpy

Michael Abshoff wrote: > > Sure, it also works for incremental builds and I do that many, many > times a day, i.e. for each patch I merge into the Sage library. What > gets recompiled is decided by our own dependency tracking code which we > want to push into Cython itself. Figuring out dependen

Re: [Numpy-discussion] parallel compilation of numpy

Brian Granger wrote: > The reason that Ondrej and I are asking all of these questions is that > we are currently exploring build systems for a couple of projects we > are working on (together). In general, my sense is that distutils > won't be sufficient for our build needs. Thus, we are looking

Re: [Numpy-discussion] parallel compilation of numpy

David Cournapeau wrote: > Michael Abshoff wrote: >> David Cournapeau wrote: Hi David, >> With Sage we do the cythonization in parallel and for now build >> extension serially, but we have code to do that in parallel, too. Given >> that we are building 180 extensions or so the speedup is linear.

Re: [Numpy-discussion] parallel compilation of numpy

Michael Abshoff wrote: > David Cournapeau wrote: > >> Christian Heimes wrote: >> >>> David Cournapeau wrote: >>> > > Hi, > > >>> You may call me naive and ignorant. Is it really that hard to archive >>> some kind of poor man's concurrency? You don't have to parallelize >>> everyth

Re: [Numpy-discussion] parallel compilation of numpy

David, Thank you very much for writing this summary up. It collects a lot of useful information about the subtle and difficult issues related to building multi-language python packages. The reason that Ondrej and I are asking all of these questions is that we are currently exploring build system

Re: [Numpy-discussion] parallel compilation of numpy

David Cournapeau wrote: > Christian Heimes wrote: >> David Cournapeau wrote: Hi, >> You may call me naive and ignorant. Is it really that hard to archive >> some kind of poor man's concurrency? You don't have to parallelize >> everything to get a speed up on multi core machines. Usually the compi

Re: [Numpy-discussion] parallel compilation of numpy

On Thu, Feb 19, 2009 at 9:14 AM, Ondrej Certik wrote: > Hi, > > I have a shiny new computer with 8 cores and numpy still takes forever > to compile --- is there a way to compile it in parallel (make -j9)? > > Do distutils allow that? If not, let's move to some build system that > allows that? Sin

Re: [Numpy-discussion] parallel compilation of numpy

Christian Heimes wrote: > David Cournapeau wrote: > >> No, and it never will. Parallel builds requires to build with >> dependency handling. Even make does not handle it well: it works most >> of the time by accident, but there are numerous problems (try for >> example building lapack with make

Re: [Numpy-discussion] parallel compilation of numpy

On Wed, Feb 18, 2009 at 8:19 PM, David Cournapeau wrote: > On Thu, Feb 19, 2009 at 11:07 AM, Ryan May wrote: > > > > > Not to nitpick, but this is the second time I've seen this lately: > > > > physician == medical doctor != physicist :) > > You're right of course - the French word for physicist

Re: [Numpy-discussion] parallel compilation of numpy

David Cournapeau wrote: > No, and it never will. Parallel builds requires to build with > dependency handling. Even make does not handle it well: it works most > of the time by accident, but there are numerous problems (try for > example building lapack with make -j8 on your 8 cores machine - it >

Re: [Numpy-discussion] parallel compilation of numpy

On Thu, Feb 19, 2009 at 11:07 AM, Ryan May wrote: > > Not to nitpick, but this is the second time I've seen this lately: > > physician == medical doctor != physicist :) You're right of course - the French word for physicist being "physicien", it may be one more mistake perpetuated by the French

Re: [Numpy-discussion] parallel compilation of numpy

On Wed, Feb 18, 2009 at 8:00 PM, David Cournapeau wrote: > On Thu, Feb 19, 2009 at 10:50 AM, Sturla Molden wrote: > > > >> I have a shiny new computer with 8 cores and numpy still takes forever > >> to compile > > > > Yes, forever/8 = forever. > > Not if you are a physician: my impression in unde

Re: [Numpy-discussion] parallel compilation of numpy

On Thu, Feb 19, 2009 at 10:50 AM, Sturla Molden wrote: > >> I have a shiny new computer with 8 cores and numpy still takes forever >> to compile > > Yes, forever/8 = forever. Not if you are a physician: my impression in undergrad was that infinity / 8 could be anything from 0 to infinity in physi

Re: [Numpy-discussion] parallel compilation of numpy

On Thu, Feb 19, 2009 at 9:14 AM, Ondrej Certik wrote: > Hi, > > I have a shiny new computer with 8 cores and numpy still takes forever > to compile Forever ? It takes one minute to build :) scipy takes for ever, but it is because of C++ more than anything else. > --- is there a way to compile it

Re: [Numpy-discussion] parallel compilation of numpy

On Thu, Feb 19, 2009 at 02:50:01AM +0100, Sturla Molden wrote: > > I have a shiny new computer with 8 cores and numpy still takes forever > > to compile > Yes, forever/8 = forever. Good point. nan_to_num() could be helpful here. -- Matthew Miller mat...@mattdm.org

Re: [Numpy-discussion] parallel compilation of numpy

> I have a shiny new computer with 8 cores and numpy still takes forever > to compile Yes, forever/8 = forever. Sturla Molden ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] parallel compilation of numpy

On Wed, Feb 18, 2009 at 18:14, Ondrej Certik wrote: > Hi, > > I have a shiny new computer with 8 cores and numpy still takes forever > to compile --- is there a way to compile it in parallel (make -j9)? > > Do distutils allow that? No. numscons will, though. > If not, let's move to some build sy

[Numpy-discussion] parallel compilation of numpy

Hi, I have a shiny new computer with 8 cores and numpy still takes forever to compile --- is there a way to compile it in parallel (make -j9)? Do distutils allow that? If not, let's move to some build system that allows that? Just wanted to check if there is some reason for that, apart from patch