Re: [Numpy-discussion] Medians that ignore values

2008-09-22 Thread Peter Saffrey
David Cournapeau ar.media.kyoto-u.ac.jp> writes: > Unfortunately, we can't, because we would loose generality: we need to > compute median on any axis, not only the last one. The proper solution > would be to have a sort/max/min/etc... which knows about nan in numpy, > which is what Chuck and I a

Re: [Numpy-discussion] Medians that ignore values

2008-09-22 Thread David Cournapeau
Peter Saffrey wrote: > > I've found that if I just cut nans from the list and use regular numpy median, > it is quicker - 10 times slower than list median, rather than 35 times slower. > Could you just wire nanmedian to do it this way? Unfortunately, we can't, because we would loose generality: w

Re: [Numpy-discussion] Medians that ignore values

2008-09-22 Thread Peter Saffrey
David Cournapeau ar.media.kyoto-u.ac.jp> writes: > Still, it is indeed really slow for your case; when I fixed nanmean and > co, I did not know much about numpy, I just wanted them to give the > right answer :) I think this can be made faster, specially for your case > (where the axis along which

Re: [Numpy-discussion] Medians that ignore values

2008-09-21 Thread David Cournapeau
David Cournapeau wrote: > > The isnan thing is surprising, because the whole point to have a isnan > is that you can do it without branching. I checked, and numpy does use > the macro of isnan, not the function (glibc has both). Ok, see my patch #913 for this. The slowdown is actually specific to

Re: [Numpy-discussion] Medians that ignore values

2008-09-21 Thread Charles R Harris
On Sun, Sep 21, 2008 at 12:56 AM, David Cournapeau < [EMAIL PROTECTED]> wrote: > David Cournapeau wrote: > > Anne Archibald wrote: > >> If users are concerned about performance, it's worth noting that on > >> some machines nans force a fallback to software floating-point > >> handling, with a corr

Re: [Numpy-discussion] Medians that ignore values

2008-09-21 Thread David Cournapeau
David Cournapeau wrote: > Anne Archibald wrote: >> If users are concerned about performance, it's worth noting that on >> some machines nans force a fallback to software floating-point >> handling, with a corresponding very large performance hit. This >> includes some but not all x86 (and I think x

Re: [Numpy-discussion] Medians that ignore values

2008-09-21 Thread David Cournapeau
Anne Archibald wrote: > > If users are concerned about performance, it's worth noting that on > some machines nans force a fallback to software floating-point > handling, with a corresponding very large performance hit. This > includes some but not all x86 (and I think x86-64) CPUs. How this > comp

Re: [Numpy-discussion] Medians that ignore values

2008-09-20 Thread Jake Harris
On Sat, Sep 20, 2008 at 11:02 AM, Jake Harris <[EMAIL PROTECTED]>wrote: > > Because you're always working with probabilities, there is almost always no > ambiguity...whenever NaN is encounter, 0 is what is desired. > ...of course, division presents a good counterexample. > Bad idea? > So prob

Re: [Numpy-discussion] Medians that ignore values

2008-09-20 Thread Jake Harris
(sorry for starting a new thread...I wasn't subscribed yet) Stéfan van der Walt wrote the following on 09/19/2008 02:10 AM: > > So am I. In all my use cases, NaNs indicate trouble. > I can provide a use case where NaNs do not indicate trouble. In fact, they need to be treated as 0. For example

Re: [Numpy-discussion] Medians that ignore values

2008-09-20 Thread David Cournapeau
Charles R Harris wrote: > > > > I would be happy to implement nan sorts if someone can provide me with > a portable and easy way to detect nans for single, double, and long > double floats. And not have it fail if the architecture doesn't > support nans. I think getting all the needed nan detection

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Anne Archibald
2008/9/19 Eric Firing <[EMAIL PROTECTED]>: > Pierre GM wrote: > >>> It seems to me that there are pragmatic reasons >>> why people work with NaNs for missing values, >>> that perhaps shd not be dismissed so quickly. >>> But maybe I am overlooking a simple solution. >> >> nansomething solutions tend

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Robert Kern
On Sat, Sep 20, 2008 at 01:15, Charles R Harris <[EMAIL PROTECTED]> wrote: > I would be happy to implement nan sorts if someone can provide me with a > portable and easy way to detect nans for single, double, and long double > floats. And not have it fail if the architecture doesn't support nans.

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Charles R Harris
On Fri, Sep 19, 2008 at 11:41 PM, David Cournapeau < [EMAIL PROTECTED]> wrote: > Anne Archibald wrote: > > > > I, on the other hand, was making specifically that suggestion: users > > should not use nans to indicate missing values. Users should use > > masked arrays to indicate missing values. > >

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread David Cournapeau
Anne Archibald wrote: > > I, on the other hand, was making specifically that suggestion: users > should not use nans to indicate missing values. Users should use > masked arrays to indicate missing values. I agree it is the nicest solution in theory, but I think it is impractical (as mentioned by

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Anne Archibald
2008/9/19 David Cournapeau <[EMAIL PROTECTED]>: > I guess my formulation was poor: I never use NaN as missing values > because I never use missing values, which is why I wanted the opinion of > people who use NaN in a different manner (because I don't have a good > idea on how those people would l

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread David Cournapeau
Robert Kern wrote: > On Fri, Sep 19, 2008 at 22:25, David Cournapeau > <[EMAIL PROTECTED]> wrote: > > > How, exactly? ndarray.min() is the where the implementation is. > Ah, I keep forgetting those are implemented in the array object, sorry for that. Now I understand Stefan point. Do I under

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread David Cournapeau
Alan G Isaac wrote: > On 9/19/2008 4:35 AM David Cournapeau apparently wrote: >> I never use NaN as missing value > > What do you use? > > Recently I needed to fill a 2d array with values > from computations that could "go wrong". > I created an array of NaN and then replaced > the elements where t

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Robert Kern
On Fri, Sep 19, 2008 at 22:25, David Cournapeau <[EMAIL PROTECTED]> wrote: > Stéfan van der Walt wrote: >> >> Why shouldn't we have "nanmin"-like behaviour for the C min itself? >> > > Ah, I was not arguing we should not do it in C, but rather we did not > have to do in C. The current behavior for

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread David Cournapeau
Stéfan van der Walt wrote: > > Why shouldn't we have "nanmin"-like behaviour for the C min itself? > Ah, I was not arguing we should not do it in C, but rather we did not have to do in C. The current behavior for nan with functions relying on ordering is broken; if someone prefer fixing it in C

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Pierre GM
On Friday 19 September 2008 17:25:53 Alan G Isaac wrote: > On 9/19/2008 4:54 PM Pierre GM apparently wrote: > > Another way is > > ma.array(np.empty(yourshape,yourdtype), mask=True) > > which should work with earlier versions. > > Seems like ``mask`` would be a natural > keyword for ``ma.empty``?

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Alan G Isaac
On 9/19/2008 4:54 PM Pierre GM apparently wrote: > Another way is > ma.array(np.empty(yourshape,yourdtype), mask=True) > which should work with earlier versions. Seems like ``mask`` would be a natural keyword for ``ma.empty``? Thanks, Alan Isaac ___ N

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Pierre GM
On Friday 19 September 2008 16:35:23 Alan G Isaac wrote: > On 9/19/2008 4:54 AM Pierre GM apparently wrote: > > I know. I was more dreading the time when MaskedArrays would have to be > > ported to C. In a way, that would probably simplify a few issues. OTOH, I > > don't really see it happening any

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Pierre GM
On Friday 19 September 2008 16:28:34 Alan G Isaac wrote: > On 9/19/2008 11:46 AM Pierre GM apparently wrote: > a.mask=True > This is great, but is apparently > new behavior as of NumPy 1.2? I'm not sure, sorry. Another way is ma.array(np.empty(yourshape,yourdtype), mask=True) which should w

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Alan G Isaac
On 9/19/2008 4:54 AM Pierre GM apparently wrote: > I know. I was more dreading the time when MaskedArrays would have to be > ported > to C. In a way, that would probably simplify a few issues. OTOH, I don't > really see it happening any time soon. Is this possibly a GSoC sized project? Alan Isa

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Alan G Isaac
On 9/19/2008 11:46 AM Pierre GM apparently wrote: a.mask=True This is great, but is apparently new behavior as of NumPy 1.2? Alan Isaac ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-d

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Alan G Isaac
On 9/19/2008 1:58 PM Robert Kern apparently wrote: > there are no objects inside non-object arrays. There is > nothing with identity inside the arrays to compare against. Got it. Thanks. Alan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Pierre GM
On Friday 19 September 2008 14:01:13 Eric Firing wrote: > Pierre GM wrote: > 2) convenient interfacing with extension code in C or C++. > > The latter is a factor in the present use of nan in matplotlib; using > nan for missing values in an array passed into extension code saves > having to pass a

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Eric Firing
Pierre GM wrote: >> It seems to me that there are pragmatic reasons >> why people work with NaNs for missing values, >> that perhaps shd not be dismissed so quickly. >> But maybe I am overlooking a simple solution. > > nansomething solutions tend to be considerably faster, that might be one > re

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Robert Kern
On Fri, Sep 19, 2008 at 11:34, Alan G Isaac <[EMAIL PROTECTED]> wrote: > On 9/19/2008 12:02 PM Peter Saffrey apparently wrote: >> >>> a = array([1,2,nan]) >> >>> nan in a >> False > > Huh. I'm inclined to call this a bug, > since normal Python behavior is that > ``in`` should check for identity::

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Alan G Isaac
On 9/19/2008 12:02 PM Peter Saffrey apparently wrote: > >>> a = array([1,2,nan]) > >>> nan in a > False Huh. I'm inclined to call this a bug, since normal Python behavior is that ``in`` should check for identity:: >>> xl = [1.,np.nan] >>> np.nan in xl True Alan

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Charles R Harris
On Fri, Sep 19, 2008 at 1:11 AM, David Cournapeau < [EMAIL PROTECTED]> wrote: > Anne Archibald wrote: > > > > Well, for example, you might ask that all the non-nan elements be in > > order, even if you don't specify where the nan goes. > > > Ah, there are two problems, then: >- sort >- how

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Alan G Isaac
On 9/19/2008 11:46 AM Pierre GM apparently wrote: > No, but you may do the opposite: just start with an array completely masked, > and unmasked it as you need: Very useful example. I did not understand this possibility. Alan ___ Numpy-discussion maili

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Alan G Isaac
On 9/19/2008 11:46 AM Pierre GM apparently wrote: > You can't compare NaNs to anything. How do you know this np.miss is a masked > value, when np.sqrt(-1.) is NaN ? I thought you could use ``is``. E.g., >>> np.nan == np.nan False >>> np.nan is np.nan True Alan

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Pierre GM
On Friday 19 September 2008 12:02:08 Peter Saffrey wrote: > Alan G Isaac american.edu> writes: > > Recently I needed to fill a 2d array with values > > from computations that could "go wrong". > Should I take the earlier advice and switch to masked arrays? > > Peter Yes. As you've noticed, you c

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Peter Saffrey
Alan G Isaac american.edu> writes: > Recently I needed to fill a 2d array with values > from computations that could "go wrong". > I created an array of NaN and then replaced > the elements where the computation produced > a useful value. I then applied ``nanmax``, > to get the maximum of the us

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Pierre GM
On Friday 19 September 2008 11:36:17 Alan G Isaac wrote: > On 9/19/2008 11:09 AM Stefan Van der Walt apparently wrote: > > Masked arrays. Using NaN's for missing values is dangerous. You may > > do some operation, which generates invalid results, and then you have > > a mixed bag of missing and i

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Alan G Isaac
On 9/19/2008 11:09 AM Stefan Van der Walt apparently wrote: > Masked arrays. Using NaN's for missing values is dangerous. You may > do some operation, which generates invalid results, and then you have > a mixed bag of missing and invalid values. That rather evades my full question, I think?

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Stefan Van der Walt
On 19 Sep 2008, at 16:07 , Alan G Isaac wrote: > On 9/19/2008 4:35 AM David Cournapeau apparently wrote: >> I never use NaN as missing value > > What do you use? Masked arrays. Using NaN's for missing values is dangerous. You may do some operation, which generates invalid results, and then you

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Alan G Isaac
On 9/19/2008 4:35 AM David Cournapeau apparently wrote: > I never use NaN as missing value What do you use? Recently I needed to fill a 2d array with values from computations that could "go wrong". I created an array of NaN and then replaced the elements where the computation produced a useful va

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Stéfan van der Walt
2008/9/19 David Cournapeau <[EMAIL PROTECTED]>: > But cannot this be fixed at the python level of the max function ? I Why shouldn't we have "nanmin"-like behaviour for the C min itself? I'd rather have a specialised function to deal with the rare kinds of datasets where NaNs are guaranteed never

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread David Cournapeau
Peter Saffrey wrote: > > I've posted my test code below, which gives me the results: > > $ ./arrayspeed3.py > list build time: 0.01 > list median time: 0.01 > array nanmedian time: 0.36 > > I must have done something wrong to hobble nanmedian in this way... I'm quite > new to numpy, so feel free to

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread David Cournapeau
Peter Saffrey wrote: > Pierre GM gmail.com> writes: > >> I think there were some changes on the C side of numpy between 1.0 and 1.1, >> you may have to recompile scipy and matplotlib from sources. What versions >> are you using for those 2 packages ? >> > > $ dpkg -l | grep scipy > ii python-sc

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Peter Saffrey
Pierre GM gmail.com> writes: > I think there were some changes on the C side of numpy between 1.0 and 1.1, > you may have to recompile scipy and matplotlib from sources. What versions > are you using for those 2 packages ? > $ dpkg -l | grep scipy ii python-scipy

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Peter Saffrey
David Cournapeau ar.media.kyoto-u.ac.jp> writes: > It may be that nanmedian is slow. But I would sincerly be surprised if > it were slower than python list, except for some pathological cases, or > maybe a bug in nanmedian. What do your data look like ? (size, number of > nan, etc...) > I've po

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread David Cournapeau
Peter Saffrey wrote: > > I rejoiced when I saw this answer, because it looks like a function I can just > drop in and it works. Unfortunately, nanmedian seems to be quite a bit slower > than just using lists (ignoring nan values from my experiments) and a > home-brew > implementation of median. I

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Pierre GM
On Friday 19 September 2008 05:51:55 Peter Saffrey wrote: > I would like to try the masked array approach, but the Ubuntu packages for > scipy and matplotlib depend on numpy. Does anybody know whether I can > naively do "sudo python setup.py install" on a more modern numpy without > disturbing sci

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Peter Saffrey
David Cournapeau ar.media.kyoto-u.ac.jp> writes: > You can use nanmean (from scipy.stats): > I rejoiced when I saw this answer, because it looks like a function I can just drop in and it works. Unfortunately, nanmedian seems to be quite a bit slower than just using lists (ignoring nan values fr

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread David Cournapeau
Stéfan van der Walt wrote: > > So am I. In all my use cases, NaNs indicate trouble. Yes, so I would like to have the opinion of people with other usage than ours. > > Because we have x.max() silently ignoring NaNs, which causes a lot of > head-scratching, swearing and failed experiments. But ca

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Stéfan van der Walt
2008/9/19 David Cournapeau <[EMAIL PROTECTED]>: > Stéfan van der Walt wrote: >> >> I agree completely. > > Me too, but I am extremely biased toward nan is always bogus by my own > usage of numpy/scipy (I never use NaN as missing value, and nan is > always caused by divide by 0 and co). So am I. I

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Pierre GM
On Friday 19 September 2008 04:31:38 David Cournapeau wrote: > Pierre GM wrote: > > That said, numpy.nanmin, numpy.nansum... don't come with the heavy > > machinery of numpy.ma, and are therefore faster. > > I'm really going to have to learn C. > > FWIW, nanmean/nanmean/etc... are written in python

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread David Cournapeau
Stéfan van der Walt wrote: > > I agree completely. Me too, but I am extremely biased toward nan is always bogus by my own usage of numpy/scipy (I never use NaN as missing value, and nan is always caused by divide by 0 and co). I like that sort raise an exception by default with NaN: it breaks the

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread David Cournapeau
Pierre GM wrote: > That said, numpy.nanmin, numpy.nansum... don't come with the heavy machinery > of numpy.ma, and are therefore faster. > I'm really going to have to learn C. > FWIW, nanmean/nanmean/etc... are written in python, cheers, David ___

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Pierre GM
On Friday 19 September 2008 04:10:24 Anne Archibald wrote: > (is there a convenience > function that makes a masked array with a mask everywhere the data is > nan?). numpy.ma.fix_invalid, that masks your Nans and Infs and sets the underlying data to some filling value. That way, you don't carry

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Stéfan van der Walt
2008/9/19 Anne Archibald <[EMAIL PROTECTED]>: > I think the numpy attitude to nans should be that they are unexpected > bogus values that signify that something went wrong with the > calculation somewhere. They can be left in place for most operations, > but any operation that depends on the value

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Anne Archibald
2008/9/19 Pierre GM <[EMAIL PROTECTED]>: > On Friday 19 September 2008 03:11:05 David Cournapeau wrote: > >> Hm, I am always puzzled when I think about nan handling :) It always >> seem there is not good answer. > > Which is why we have masked arrays, of course ;) I think the numpy attitude to nan

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread Pierre GM
On Friday 19 September 2008 03:11:05 David Cournapeau wrote: > Hm, I am always puzzled when I think about nan handling :) It always > seem there is not good answer. Which is why we have masked arrays, of course ;) ___ Numpy-discussion mailing list Numpy

Re: [Numpy-discussion] Medians that ignore values

2008-09-19 Thread David Cournapeau
Anne Archibald wrote: > > Well, for example, you might ask that all the non-nan elements be in > order, even if you don't specify where the nan goes. Ah, there are two problems, then: - sort - how median use sort. For sort, I don't know how sort speed would be influenced by treating nan.

Re: [Numpy-discussion] Medians that ignore values

2008-09-18 Thread Anne Archibald
2008/9/19 David Cournapeau <[EMAIL PROTECTED]>: > Anne Archibald wrote: >> >> That was in amax/amin. Pretty much every other function that does >> comparisons needs to be fixed to work with nans. In some cases it's >> not even clear how: where should a sort put the nans in an array? > > The problem

Re: [Numpy-discussion] Medians that ignore values

2008-09-18 Thread David Cournapeau
Anne Archibald wrote: > > That was in amax/amin. Pretty much every other function that does > comparisons needs to be fixed to work with nans. In some cases it's > not even clear how: where should a sort put the nans in an array? The problem is more on how the functions use sort than sort itself i

Re: [Numpy-discussion] Medians that ignore values

2008-09-18 Thread Anne Archibald
2008/9/18 David Cournapeau <[EMAIL PROTECTED]>: > Anne Archibald wrote: >> >> I don't think I agree: >> >> In [4]: np.median([1,3,nan]) >> Out[4]: 3.0 >> >> In [5]: np.median([1,nan,3]) >> Out[5]: nan >> >> In [6]: np.median([nan,1,3]) >> Out[6]: 1.0 >> > > I was referring to the fact that if you h

Re: [Numpy-discussion] Medians that ignore values

2008-09-18 Thread David Cournapeau
Anne Archibald wrote: > > I don't think I agree: > > In [4]: np.median([1,3,nan]) > Out[4]: 3.0 > > In [5]: np.median([1,nan,3]) > Out[5]: nan > > In [6]: np.median([nan,1,3]) > Out[6]: 1.0 > I was referring to the fact that if you have nan in your array, you should use nanmean if you want to i

Re: [Numpy-discussion] Medians that ignore values

2008-09-18 Thread Anne Archibald
2008/9/18 David Cournapeau <[EMAIL PROTECTED]>: > Peter Saffrey wrote: >> >> Is this the correct behavior for median with nan? > > That's the expected behavior, at least :) (this is also the expected > behavior of most math packages I know, including matlab and R, so this > should not be too surpri

Re: [Numpy-discussion] Medians that ignore values

2008-09-18 Thread David Cournapeau
Peter Saffrey wrote: > > Is this the correct behavior for median with nan? That's the expected behavior, at least :) (this is also the expected behavior of most math packages I know, including matlab and R, so this should not be too surprising if you have used those). > Is there a fix for > thi

Re: [Numpy-discussion] Medians that ignore values

2008-09-18 Thread Charles R Harris
On Thu, Sep 18, 2008 at 12:23 PM, Pierre GM <[EMAIL PROTECTED]> wrote: > On Thursday 18 September 2008 13:31:18 Peter Saffrey wrote: > > The version in the Ubuntu package repository. It says 1:1.0.4-6ubuntu3. > > So it's 1.0 ? It's fairly old, that would explain. > > > > if you don't give an axis

Re: [Numpy-discussion] Medians that ignore values

2008-09-18 Thread Pierre GM
On Thursday 18 September 2008 13:31:18 Peter Saffrey wrote: > The version in the Ubuntu package repository. It says 1:1.0.4-6ubuntu3. So it's 1.0 ? It's fairly old, that would explain. > > if you don't give an axis > > parameter, you should get the median of the flattened array, therefore a > > s

Re: [Numpy-discussion] Medians that ignore values

2008-09-18 Thread Charles R Harris
On Thu, Sep 18, 2008 at 11:31 AM, Peter Saffrey <[EMAIL PROTECTED]> wrote: > Pierre GM gmail.com> writes: > > > Mmh, typo? > > > > Yes, apologies. I was aiming for thorough, but ended up just careless. It's > been > a long day. > > > Ohoh. What version of numpy are you using ? > > The version in

Re: [Numpy-discussion] Medians that ignore values

2008-09-18 Thread Peter Saffrey
Pierre GM gmail.com> writes: > Mmh, typo? > Yes, apologies. I was aiming for thorough, but ended up just careless. It's been a long day. > Ohoh. What version of numpy are you using ? The version in the Ubuntu package repository. It says 1:1.0.4-6ubuntu3. > if you don't give an axis > param

Re: [Numpy-discussion] Medians that ignore values

2008-09-18 Thread Pierre GM
On Thursday 18 September 2008 10:59:12 Peter Saffrey wrote: > I had looked at masked arrays, but couldn't quite get them to work. That's unfortunate. > >>> from numeric import * Mmh, typo? > >>> from pylab import rand > >>> a = rand(10,3) > >>> a[a > 0.8] = nan > >>> m = ma.masked_array(a,

Re: [Numpy-discussion] Medians that ignore values

2008-09-18 Thread paul taney
Thu, 9/18/08, Peter Saffrey <[EMAIL PROTECTED]> wrote: > From: Peter Saffrey <[EMAIL PROTECTED]> > Subject: Re: [Numpy-discussion] Medians that ignore values > To: numpy-discussion@scipy.org > Date: Thursday, September 18, 2008, 10:59 AM > physics.ucf.edu> writes: >

Re: [Numpy-discussion] Medians that ignore values

2008-09-18 Thread Peter Saffrey
physics.ucf.edu> writes: > Currently the only way you can handle NaNs is by using masked arrays. > Create a mask by doing isfinite(a), then call the masked array > median(). There's an example here: > > http://sd-2116.dedibox.fr/pydocweb/doc/numpy.ma/ > I had looked at masked arrays, bu

Re: [Numpy-discussion] Medians that ignore values

2008-09-18 Thread jh
> You might want to try isfinite() to first remove nan, +/- infinity > before doing that. > numpy.median(a[numpy.isfinite(a)]) We just had this discussion a month or two ago, I think even on this list, and continued it at the SciPy conference. The problem with numpy.median(a[numpy.isfinite(a)])

Re: [Numpy-discussion] Medians that ignore values

2008-09-18 Thread Bruce Southey
Nadav Horesh wrote: > I think you need to use masked arrays. > > Nadav > > > -הודעה מקורית- > מאת: [EMAIL PROTECTED] בשם Peter Saffrey > נשלח: ה 18-ספטמבר-08 14:27 > אל: numpy-discussion@scipy.org > נושא: [Numpy-discussion] Medians that ignore values >

Re: [Numpy-discussion] Medians that ignore values

2008-09-18 Thread Nadav Horesh
I think you need to use masked arrays. Nadav -הודעה מקורית- מאת: [EMAIL PROTECTED] בשם Peter Saffrey נשלח: ה 18-ספטמבר-08 14:27 אל: numpy-discussion@scipy.org נושא: [Numpy-discussion] Medians that ignore values I have data from biological experiments that is represented as a list of

[Numpy-discussion] Medians that ignore values

2008-09-18 Thread Peter Saffrey
I have data from biological experiments that is represented as a list of about 5000 triples. I would like to convert this to a list of the median of each triple. I did some profiling and found that numpy was much about 12 times faster for this application than using regular Python lists and a l