Robert Kern wrote:
>
> It's still pretty bad, though. I do recommend running Disk Repair like Bill
> did.
I did that, and it found and did nothing -- I suspect it ran when I
re-booted -- it did take a while to reboot.
However, this is pretty consistently what I'm getting now:
$ time python -
I just added ticket 874
http://scipy.org/scipy/numpy/ticket/874
which on my machine takes the import time from 0.15 seconds down to
0.093 seconds.
A bit over a month ago it was about 0.33 seconds. :)
The biggest trick I didn't apply was to defer importing the re
module, and only compile
On Mon, Aug 4, 2008 at 18:01, Christopher Barker <[EMAIL PROTECTED]> wrote:
> OK,
>
> So I'm an idiot. After reading this, I thought "I haven't rebooted for a
> while". It turns out it's been 35 days. I think I've been having slow
> startup for a longer than that, but none the less, I re-booted (wh
OK,
So I'm an idiot. After reading this, I thought "I haven't rebooted for a
while". It turns out it's been 35 days. I think I've been having slow
startup for a longer than that, but none the less, I re-booted (which
took a long time), and presto:
$ time python -c "import numpy"
real0m0.6
On Mon, Aug 4, 2008 at 14:24, Christopher Barker <[EMAIL PROTECTED]> wrote:
> Robert Kern wrote:
>> It isn't. The problem is on Chris's file system.
>
> Thanks for all your help, Robert. Interestingly, I haven't noticed any
> problems anywhere else, but who knows?
>
> I guess this is what Linux Tor
Robert Kern wrote:
> It isn't. The problem is on Chris's file system.
Thanks for all your help, Robert. Interestingly, I haven't noticed any
problems anywhere else, but who knows?
I guess this is what Linux Torvalds meant when he said that OS-X's file
system was "brain dead"
> Whatever is wron
I've got a proof of concept that take the time on my machine to
"import numpy" from 0.21 seconds down to 0.08 seconds. Doing that
required some somewhat awkward things, like deferring all 'import re'
statements. I don't think that's stable in the long run because
people will blithely impo
Robert Kern wrote:
>
> It isn't. The problem is on Chris's file system. Whatever is wrong
> with his file system (Bill Spotz's identical problem suggests too many
> temporary but unused inodes) increases the traversal of the file
> system.
Ah, I did not think it could indeed affect the whole fs. T
On Sat, Aug 2, 2008 at 00:06, David Cournapeau
<[EMAIL PROTECTED]> wrote:
> Christopher Barker wrote:
>>
>> OK, I just installed wxPython, and whoa!
>>
>> time python -c "import numpy"
>>
>> real0m2.793s
>> user0m0.294s
>> sys 0m2.494s
>>
>> so it's taking almost two seconds more to imp
Christopher Barker wrote:
>
> OK, I just installed wxPython, and whoa!
>
> time python -c "import numpy"
>
> real0m2.793s
> user0m0.294s
> sys 0m2.494s
>
> so it's taking almost two seconds more to import numpy, now that
> wxPython is installed. I haven't even imported it yet. importin
On Sat, Aug 2, 2008 at 5:33 AM, Ondrej Certik <[EMAIL PROTECTED]> wrote:
> But I am still unhappy about it, I'd like if the package could import
> much faster, because it adds up, when you need to import 7 packages
> like that, it's suddenly 1s and that's just too much.
Too much for what ? We nee
Robert Kern wrote:
> File/Save As..., pick a file name. When asked about whether to embed
> source files or strip them out, choose Strip. Then email the resulting
> .mshark file to me.
I've done that, and sent it to you directly -- it's too big to put in
the mailing list.
> It looks like your Py
Christopher Barker wrote:
> $ time python -c "import numpy"
>
> real0m0.973s
> user0m0.290s
> sys 0m0.682s
>
> much more in line with what others are getting.
>
> I'll start installing stuff that I actually am using now, and maybe I'll
> see when (if) it breaks down.
OK, I just ins
On Fri, Aug 1, 2008 at 17:50, Christopher Barker <[EMAIL PROTECTED]> wrote:
> Robert Kern wrote:
>>
>> Can you send foo.stats, too?
>
> sure can.
>
> Also, I've got Shark up and running, and have run it on a script that
> does nothing but import numpy, but really have no idea what I'm looking
> at,
Robert Kern wrote:
> So cut it out.
>
> Chris, please profile your import so we actually have some real
> information to work with instead of prejudices.
OK, while we're working on that, I've tried just re-naming my entire
python install, then starting from scratch. So far, I've python
re-insta
On Fri, Aug 1, 2008 at 17:50, Christopher Barker <[EMAIL PROTECTED]> wrote:
> Robert Kern wrote:
>>
>> Can you send foo.stats, too?
>
> sure can.
>
> Also, I've got Shark up and running, and have run it on a script that
> does nothing but import numpy, but really have no idea what I'm looking
> at,
On Fri, Aug 1, 2008 at 17:22, Christopher Barker <[EMAIL PROTECTED]> wrote:
> David Cournapeau wrote:
>>
>> IOW, I don't think the problem is the numbers themselves. It has to be
>> something else. A simple profiling like
>>
>> python -m cProfile -o foo.stats foo.py
>>
>> and then:
>>
>> python -c
David Cournapeau wrote:
IOW, I don't think the problem is the numbers themselves. It has to be
something else. A simple profiling like
python -m cProfile -o foo.stats foo.py
and then:
python -c "import pstats; p = pstats.Stats("foo.stats");
p.sort_stats('cumulative').print_stats(50)"
OK, see
On Thu, Jul 31, 2008 at 10:02 PM, Robert Kern <[EMAIL PROTECTED]> wrote:
> On Thu, Jul 31, 2008 at 05:43, Andrew Dalke <[EMAIL PROTECTED]> wrote:
>> On Jul 31, 2008, at 12:03 PM, Robert Kern wrote:
>
>>> But you still can't remove them since they are being used inside
>>> numerictypes. That's why I
On Fri, Aug 1, 2008 at 11:53, Gael Varoquaux
<[EMAIL PROTECTED]> wrote:
> On Fri, Aug 01, 2008 at 09:18:48AM -0700, Christopher Barker wrote:
>> > What does python -c "import sys; print sys.path" say ?
>
>> A lot! 41 entries, and lot's of eggs -- are eggs an issue? I'm also
>> wondering how the ord
On Sat, Aug 2, 2008 at 1:18 AM, Christopher Barker
<[EMAIL PROTECTED]> wrote:
>
> A lot! 41 entries, and lot's of eggs -- are eggs an issue? I'm also
> wondering how the order is determined -- if it looked in site-packages
> first, it would find numpy a whole lot faster.
I don't think the number i
On Fri, Aug 01, 2008 at 09:18:48AM -0700, Christopher Barker wrote:
> > What does python -c "import sys; print sys.path" say ?
> A lot! 41 entries, and lot's of eggs -- are eggs an issue? I'm also
> wondering how the order is determined -- if it looked in site-packages
> first, it would find num
David Cournapeau wrote:
>> time python -c "import numpy"
>>
>> real0m8.383s
>> user0m0.320s
>> sys 0m7.805s
> What does python -c "import sys; print sys.path" say ?
A lot! 41 entries, and lot's of eggs -- are eggs an issue? I'm also
wondering how the order is determined -- if it look
On Thu, Jul 31, 2008 at 11:45, Christopher Barker <[EMAIL PROTECTED]> wrote:
> On my OS-X box (10.4.11, python2.5, numpy '1.1.1rc2'), it takes about 7
> seconds to import numpy!
Can you try running a Python process that just imports numpy under Shark.app?
http://developer.apple.com/tools/shark
On Thu, Jul 31, 2008 at 16:09, Andrew Dalke <[EMAIL PROTECTED]> wrote:
> On Jul 31, 2008, at 10:02 PM, Robert Kern wrote:
>> 1) Everything exposed by "from numpy import *" still needs to work.
>
> Does that include numpy.Tester? I don't mean numpy.test() nor
> numpy.bench().
>
> Does that include
On Jul 31, 2008, at 10:02 PM, Robert Kern wrote:
> 1) Everything exposed by "from numpy import *" still needs to work.
Does that include numpy.Tester? I don't mean numpy.test() nor
numpy.bench().
Does that include numpy.PackageLoader? I don't mean numpy.pkgload.
> 2) The improvement in impo
On Fri, Aug 1, 2008 at 5:02 AM, Robert Kern <[EMAIL PROTECTED]> wrote:
>
> 5) Proxy objects ... I would really like to avoid proxy objects. They
> have caused fragility in the past.
One recurrent problem around import times optimization is that it is
some work to improve it, but it takes one line
On Thu, Jul 31, 2008 at 05:43, Andrew Dalke <[EMAIL PROTECTED]> wrote:
> On Jul 31, 2008, at 12:03 PM, Robert Kern wrote:
>> But you still can't remove them since they are being used inside
>> numerictypes. That's why I labeled them "internal utility functions"
>> instead of leaving them with mini
On Thu, Jul 31, 2008 at 07:46:20AM -0500, Nathan Bell wrote:
> On Thu, Jul 31, 2008 at 7:31 AM, Hanni Ali <[EMAIL PROTECTED]> wrote:
> >
> > I would just to highlight an alternate use of numpy to interactive use. We
> > have a cluster of machines which process tasks on an individual basis where
> >
On Thu, Jul 31, 2008 at 10:12:22AM -0700, Christopher Barker wrote:
> I've been wondering about that.
> time python -c "import numpy"
> real0m8.383s
> user0m0.320s
> sys 0m7.805s
I don't know what is wrong, but this is plain wrong, unless you are on a
distant file system, or somethin
>
> hot -- it takes about 10 cold.
>
> I've been wondering about that.
>
> time python -c "import numpy"
>
> real0m8.383s
> user0m0.320s
> sys 0m7.805s
>
> and similar results if run multiple times in a row.
What does python -c "import sys; print sys.path" say ?
> Any idea what could
On Thu, Jul 31, 2008 at 1:12 PM, Christopher Barker
<[EMAIL PROTECTED]>wrote:
> David Cournapeau wrote:
> > Christopher Barker wrote:
> >> On my OS-X box (10.4.11, python2.5, numpy '1.1.1rc2'), it takes about 7
> >> seconds to import numpy!
> >
> > Hot or cold ? If hot, there is something horribly
On Thu, 31 Jul 2008 10:12:22 -0700
Christopher Barker <[EMAIL PROTECTED]> wrote:
> David Cournapeau wrote:
>> Christopher Barker wrote:
>>> On my OS-X box (10.4.11, python2.5, numpy '1.1.1rc2'),
>>>it takes about 7
>>> seconds to import numpy!
>>
>> Hot or cold ? If hot, there is something hor
David Cournapeau wrote:
> Christopher Barker wrote:
>> On my OS-X box (10.4.11, python2.5, numpy '1.1.1rc2'), it takes about 7
>> seconds to import numpy!
>
> Hot or cold ? If hot, there is something horribly wrong with your setup.
hot -- it takes about 10 cold.
I've been wondering about that.
Stéfan van der Walt wrote:
> No one is *waiting* for NumPy to start.
I am, and probably 10 times, a day, yes.
And it's a major issue for CGI, though maybe no one's using that anymore
anyway.
> Just by answering this
> e-mail I could have (and maybe should have) started NumPy three
> hundred
Christopher Barker wrote:
> On my OS-X box (10.4.11, python2.5, numpy '1.1.1rc2'), it takes about 7
> seconds to import numpy!
>
>
Hot or cold ? If hot, there is something horribly wrong with your setup.
On my macbook, it takes ~ 180 ms to to python -c "import numpy", and ~
100 ms on linux (sa
Andrew Dalke wrote:
> If I had my way, remove things like (in numpy/__init__.py)
>
> import linalg
> import fft
> import random
> import ctypeslib
> import ma
as a side benefit, this might help folks using py2exe, py2app and
friends -- as it stands all those sub-modules
On Thu, Jul 31, 2008 at 10:34:04AM -0400, Kevin Jacobs <[EMAIL PROTECTED]>
wrote:
>The morale of this discussion, for me, is that just because _you_ don't
>care about a particular aspect or feature, doesn't mean that others don't
>or shouldn't. Your workarounds may not be viable for m
On Thu, Jul 31, 2008 at 11:16:12PM +0900, David Cournapeau wrote:
> Gael Varoquaux wrote:
> > Obviously, the build part has to be well-tuned for the machinery to work,
> > but there is a lot of value here.
> Ah yes, setuptools does have this. But this is specific to setuptools,
> bare distutils d
On Thu, Jul 31, 2008 at 10:14 AM, Gael Varoquaux <
[EMAIL PROTECTED]> wrote:
> On Thu, Jul 31, 2008 at 12:43:17PM +0200, Andrew Dalke wrote:
> > Startup performance has not been a numpy concern. It a concern for
> > me, and it has been (for other packages) a concern for some of my
> > clients.
>
Gael Varoquaux wrote:
> Obviously, the build part has to be well-tuned for the machinery to work,
> but there is a lot of value here.
>
Ah yes, setuptools does have this. But this is specific to setuptools,
bare distutils does not have this test command, right ?
cheers,
David
On Thu, Jul 31, 2008 at 11:05:33PM +0900, David Cournapeau wrote:
> Gael Varoquaux wrote:
> > That's why distutils have a test target. You can do "python setup.py
> > test", and if you have setup you setup.py properly it should work
> > (obviously it is easy to make this statement, and harder to g
Gael Varoquaux wrote:
>
> That's why distutils have a test target. You can do "python setup.py
> test", and if you have setup you setup.py properly it should work
> (obviously it is easy to make this statement, and harder to get the thing
> working).
>
I have already seen some discussion about
On Thu, Jul 31, 2008 at 2:12 AM, Andrew Dalke <[EMAIL PROTECTED]> wrote:
> Hmm. And it looks like testing/nosetester.py (which implements the
> 'test' function above) is meant to make it easier to run nose, except
> my feeling is the extra level of wrapping makes things more
> complicated. The nos
On Thu, Jul 31, 2008 at 12:43:17PM +0200, Andrew Dalke wrote:
> Startup performance has not been a numpy concern. It a concern for
> me, and it has been (for other packages) a concern for some of my
> clients.
I am curious, if startup performance is a problem, I guess it is because
you are ru
On Thu, Jul 31, 2008 at 03:41:15PM +0900, David Cournapeau wrote:
> Yes. Nothing that an easy make file cannot solve, nonetheless (I am sure
> I am not the only one with a makefile/script which automates the above,
> to test a new svn updated numpy in one command).
That's why distutils have a test
On Thu, Jul 31, 2008 at 5:36 AM, Andrew Dalke <[EMAIL PROTECTED]> wrote:
>
> The user base for numpy might be .. 10,000 people? 100,000 people?
> Let's go with the latter, and assume that with command-line scripts,
> CGI scripts, and the other programs that people write in order to
> help do resea
Nathan Bell wrote:
>
> There are other components of NumPy/SciPy that are more worthy of
> optimization. Given that programmer time is a scarce resource, it's
> more sensible to direct our efforts towards making the other 98.5% of
> the computation faster.
>
To be fair, when I took a look at t
On Thu, Jul 31, 2008 at 7:31 AM, Hanni Ali <[EMAIL PROTECTED]> wrote:
>
> I would just to highlight an alternate use of numpy to interactive use. We
> have a cluster of machines which process tasks on an individual basis where
> a master tasks may spawn 600 slave tasks to be processed. These tasks
Hi All,
I've been reading this discussion with interest.
I would just to highlight an alternate use of numpy to interactive use. We
have a cluster of machines which process tasks on an individual basis where
a master tasks may spawn 600 slave tasks to be processed. These tasks are
spread across t
2008/7/31 Andrew Dalke <[EMAIL PROTECTED]>:
> The user base for numpy might be .. 10,000 people? 100,000 people?
> Let's go with the latter, and assume that with command-line scripts,
> CGI scripts, and the other programs that people write in order to
> help do research means that numpy is started
On Jul 31, 2008, at 12:03 PM, Robert Kern wrote:
> That said, the reason those particular docstrings are verbose is
> because I wanted people to know why those functions exist there (e.g.
> "This is an internal utility function").
Err, umm, you mean that first line of the second paragraph
in t
On Jul 31, 2008, at 11:42 AM, Stéfan van der Walt wrote:
> Maybe when we're convinced that there is a lot to be gained from
> making such a change. From my perspective, it doesn't look good:
>
> I) Major code breakage
> II) Confused users
> III) More difficult function discovery for beginners
I'm
On Thu, Jul 31, 2008 at 04:42, Stéfan van der Walt <[EMAIL PROTECTED]> wrote:
> 2008/7/31 Andrew Dalke <[EMAIL PROTECTED]>:
>> I wrote that I don't know the reasons for why the design was as it
>> is. Are those functions ("english_upper", "english_lower",
>> "english_capitalize") expected as part
2008/7/31 Andrew Dalke <[EMAIL PROTECTED]>:
>> Numpy has a very flat namespace, for better or worse, which implies
>> many imports.
>
> I don't get the feeling that numpy is flat. Python's stdlib is flat.
> Numpy has many 2- and 3-level modules.
With 500+ functions in the root namespace, I'd call
On Thu, Jul 31, 2008 at 01:12, Andrew Dalke <[EMAIL PROTECTED]> wrote:
> On Jul 31, 2008, at 3:53 AM, David Cournapeau wrote:
>> You are supposed to run the tests on an installed numpy, not in the
>> sources:
>>
>> import numpy
>> numpy.test(verbose = 10)
>
> Doesn't that make things more cumbersom
On Thu, 2008-07-31 at 08:12 +0200, Andrew Dalke wrote:
> On Jul 31, 2008, at 3:53 AM, David Cournapeau wrote:
> > You are supposed to run the tests on an installed numpy, not in the
> > sources:
> >
> > import numpy
> > numpy.test(verbose = 10)
>
> Doesn't that make things more cumbersome to test?
On Jul 31, 2008, at 3:53 AM, David Cournapeau wrote:
> You are supposed to run the tests on an installed numpy, not in the
> sources:
>
> import numpy
> numpy.test(verbose = 10)
Doesn't that make things more cumbersome to test? That is, if I were
to make a change I would need to:
- python se
On Jul 31, 2008, at 4:21 AM, Alan McIntyre wrote:
> They actually do two different things; numpy.test() runs test for all
> of numpy, and numpy.testing.test() runs tests for numpy.testing only.
> There are similar functions in numpy.lib, numpy.core, etc.
Really? This is the code from numpy/__init
On Wed, Jul 30, 2008 at 8:19 PM, Andrew Dalke <[EMAIL PROTECTED]> wrote:
> numpy defines
>
> numpy.test
> numpy.bench
>
> and
>
> numpy.testing.test
>
> The two 'test's use the same implementation. This is a likely
> unneeded duplication and one should be removed. The choice depends on
> if peo
On Thu, 2008-07-31 at 02:07 +0200, Andrew Dalke wrote:
> On Jul 30, 2008, at 10:59 PM, Stéfan van der Walt wrote:
> > I.e. most people don't start up NumPy all the time -- they import
> > NumPy, and then do some calculations, which typically take longer than
> > the import time.
>
> Is that intera
On Jul 30, 2008, at 10:51 PM, Alan McIntyre wrote:
> I suppose it's necessary for providing the test() and bench()
> functions in subpackages, but I that isn't a good reason to impose
> upon all users the time required to set up numpy.testing.
I just posted this in my reply to Stéfan, but I'll say
On Jul 30, 2008, at 10:59 PM, Stéfan van der Walt wrote:
> I.e. most people don't start up NumPy all the time -- they import
> NumPy, and then do some calculations, which typically take longer than
> the import time.
Is that interactively, or is that through programs?
> For a benefit of 0.03s, I
2008/7/30 Andrew Dalke <[EMAIL PROTECTED]>:
> Based on those results I've been digging into the code trying to
> figure out why numpy imports so many files, and at the same time I've
> been trying to guess at the use case Robert Kern regards as typical
> when he wrote:
>
> Your use case isn't s
On Wed, Jul 30, 2008 at 4:12 PM, Andrew Dalke <[EMAIL PROTECTED]> wrote:
> 4) numpy.testing takes 0.041 seconds to import. The text I quoted
> above says that it's a numpy requirement that 'testing' always be
> imported, even though I'm hard pressed to figure out why that's
> important.
I suppos
On Jul 4, 2008, at 2:22 PM, Andrew Dalke wrote:
> [josiah:numpy/build/lib.macosx-10.3-fat-2.5] dalke% time python -c
> 'pass'
> 0.015u 0.042s 0:00.06 83.3% 0+0k 0+0io 0pf+0w
> [josiah:numpy/build/lib.macosx-10.3-fat-2.5] dalke% time python -c
> 'import numpy'
> 0.084u 0.231s 0:00.33 93.9%
On Jul 3, 2008, at 9:06 AM, Robert Kern wrote:
> Can you try the SVN trunk?
Sure. Though did you know it's not easy to find how to get numpy
from SVN? I had to go to the second page of Google, which linked to
someone's talk.
I expected to find a link to it at http://numpy.scipy.org/ .
Just
On Mon, Jun 30, 2008 at 18:32, Andrew Dalke <[EMAIL PROTECTED]> wrote:
> Why does numpy/__init__.py need to import all of these other modules
> and submodules? Any chance of cutting down on the number, in order
> to improve startup costs?
Can you try the SVN trunk? In another thread (it must be "
2008/7/1 Hanni Ali <[EMAIL PROTECTED]>:
> Would it not be possible to import just the necessary module of
> numpy to
> meet the necessary functionality of your application.
Matthieu Brucher responded:
> IIRC, il you do import numpy.core as numpy, it starts by importing
> numpy, so it will be eve
You are correct, it appears to take slightly longer to import numpy.core and
longer again to import numpy.core as numpy
I should obviously check first in future.
Hanni
2008/7/1 Matthieu Brucher <[EMAIL PROTECTED]>:
> Hi,
>
> IIRC, il you do import numpy.core as numpy, it starts by importing
> n
Hi,
IIRC, il you do import numpy.core as numpy, it starts by importing
numpy, so it will be even slower.
Matthieu
2008/7/1 Hanni Ali <[EMAIL PROTECTED]>:
> Would it not be possible to import just the necessary module of numpy to
> meet the necessary functionality of your application.
>
> i.e.
>
Would it not be possible to import just the necessary module of numpy to
meet the necessary functionality of your application.
i.e.
import numpy.core
or whatever you're using
you could even do:
import numpy.core as numpy
I think, to simplify your code, I'm no expert though.
Hanni
2008/7/1
On Jul 1, 2008, at 2:22 AM, Robert Kern wrote:
> Your use case isn't so typical and so suffers on the import time
> end of the
> balance.
I'm working on my presentation for EuroSciPy. "Isn't so typical"
seems to be a good summary of my first slide. :)
>> Any chance of cutting down on the nu
On Mon, Jun 30, 2008 at 18:32, Andrew Dalke <[EMAIL PROTECTED]> wrote:
> (Trying again now that I'm subscribed. BTW, there's no link to the
> subscription page from numpy.scipy.org .)
>
>
> The initial 'import numpy' loads a huge number of modules, even when
> I don't need them.
>
> Python 2.5 (r2
(Trying again now that I'm subscribed. BTW, there's no link to the
subscription page from numpy.scipy.org .)
The initial 'import numpy' loads a huge number of modules, even when
I don't need them.
Python 2.5 (r25:51918, Sep 19 2006, 08:49:13)
[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on
75 matches
Mail list logo
