Re: [Numpy-discussion] Histograms of extremely large data sets

2006-12-14 Thread Cameron Walsh
Using Eric's latest speed-testing, here's David's results: [EMAIL PROTECTED]:~/code_snippets/histogram$ python histogram_speed.py type: uint8 millions of elements: 100.0 sec (C indexing based): 8.44 1 sec (numpy iteration based): 8.91 1 sec (rick's pure python): 6.4 1 sec (

Re: [Numpy-discussion] Can we change how fortran compiler version strings are handled?!

2006-12-14 Thread Robert Kern
Brian Granger wrote: > Can we please change how Numpy handles the version string of fortran > compilers? Yes, please. I'll be happy to apply any patch you might provide for this. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible

[Numpy-discussion] Can we change how fortran compiler version strings are handled?!

2006-12-14 Thread Brian Granger
Hi, I have been doing quite a bit of numpy evangelism here at my work and slowly people are starting to use it. One of the main things people are interested in is f2py. But, I am finding that there is one persistent problem that keeps coming up when people try to install numpy on various systems

Re: [Numpy-discussion] Histograms of extremely large data sets

2006-12-14 Thread David Huard
Hi, I spent some time a while ago on an histogram function for numpy. It uses digitize and bincount instead of sorting the data. If I remember right, it was significantly faster than numpy's histogram, but I don't know how it will behave with very large data sets. I attached the file if you want

Re: [Numpy-discussion] Pyrex and numpy

2006-12-14 Thread Francesc Altet
El dj 14 de 12 del 2006 a les 11:21 -0700, en/na Tim Hochberg va escriure: > > I was just going to try pyrex out with numpy to see how it compares with > weave (which is cool but quirky). My first attempt ended in failure: I > tried to compile the demo in in numpy/doc/pyrex and got this error: >

Re: [Numpy-discussion] Histograms of extremely large data sets

2006-12-14 Thread eric jones
I just noticed a bug in this code. "PyArray_ITER_NEXT(iter);" should be moved out of the if statement. eric eric jones wrote: > > > Rick White wrote: >> Just so we don't get too smug about the speed, if I do this in IDL >> on the same machine it is 10 times faster (0.28 seconds instead of >>

Re: [Numpy-discussion] Histograms of extremely large data sets

2006-12-14 Thread eric jones
Rick White wrote: Just so we don't get too smug about the speed, if I do this in IDL on the same machine it is 10 times faster (0.28 seconds instead of 4 seconds). I'm sure the IDL version uses the much faster approach of just sweeping through the array once, incrementing counts in the

[Numpy-discussion] Pyrex and numpy

2006-12-14 Thread Tim Hochberg
I was just going to try pyrex out with numpy to see how it compares with weave (which is cool but quirky). My first attempt ended in failure: I tried to compile the demo in in numpy/doc/pyrex and got this error: c_numpy.pxd:99:22: Array element cannot be a Python object Does anyone who us

Re: [Numpy-discussion] Histograms of extremely large data sets

2006-12-14 Thread Brian Granger
This same idea could be used to parallelize the histogram computation. Then you could really get into large (many Gb/TB/PB) data sets. I might try to find time to do this with ipython1, but someone else could do this as well. Brian On 12/13/06, Rick White <[EMAIL PROTECTED]> wrote: > On Dec 12,

Re: [Numpy-discussion] empty data matrix (are they really empty ?)

2006-12-14 Thread Christopher Barker
Sven Schreiber wrote: >> In the old file I created a matrix on the fly. I know that Numpy and >> python cannot do that so I found a workaround numpy can create matrices on the fly, in fact, you are doing that with this code! The only thing it doesn't do is have a lateral that joins matrices

Re: [Numpy-discussion] empty data matrix (are they really empty ?)

2006-12-14 Thread Sven Schreiber
[you probably should have started a new thread instead of replying to another one...] Giorgio Luciano schrieb: > In the old file I created a matrix on the fly. I know that Numpy and > python cannot do that so I found a workaround I'm not sure what you mean what numpy cannot do, but... > here

[Numpy-discussion] empty data matrix (are they really empty ?)

2006-12-14 Thread Giorgio Luciano
I was converting a matlab file to my new favority scientific language Numpy :) In the old file I created a matrix on the fly. I know that Numpy and python cannot do that so I found a workaround here's the code lev2=empty((1,h)) ir=1 for j in arange(1,nstep+2): #a=gr[[arange(ir-1,ir+nstep)],:

Re: [Numpy-discussion] Histograms of extremely large data sets

2006-12-14 Thread Rick White
On Dec 14, 2006, at 2:56 AM, Cameron Walsh wrote: > At some point I might try and test > different cache sizes for different data-set sizes and see what the > effect is. For now, 65536 seems a good number and I would be happy to > see this replace the current numpy.histogram. I experimented a li