Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Benjamin Root
On Thu, Jun 23, 2011 at 8:41 PM, Nathaniel Smith wrote: > Internally, there are > a bunch of different NA values -- for floats it's a particular NaN, > for integers it's INT_MIN, for booleans it's 2 (IIRC), etc. So, Bender runs on R? (Sorry, couldn't resist. Futurama season premiere is tonigh

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Nathaniel Smith
On Thu, Jun 23, 2011 at 5:56 PM, Benjamin Root wrote: > Lastly, I am not entirely familiar with R, so I am also very curious about > what this magical "NA" value is, and how it compares to how NaNs work. > Although, Pierre brought up the very good point that NaNs woulldn't work > anyway with integ

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Nathaniel Smith
On Thu, Jun 23, 2011 at 5:21 PM, Mark Wiebe wrote: > On Thu, Jun 23, 2011 at 7:00 PM, Nathaniel Smith wrote: >> It's should also be possible to accomplish a general solution at the >> dtype level. We could have a 'dtype factory' used like: >>  np.zeros(10, dtype=np.maybe(float)) >> where np.maybe

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Pierre GM
On Jun 24, 2011, at 2:56 AM, Benjamin Root wrote: > > As a heavy masked_array user, I regret not being able to participate more in > this discussion as I am madly cranking out matplotlib code. I would like to > say that I have always seen masked arrays as being the "next step up" from > using

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Pierre GM
On Jun 24, 2011, at 2:42 AM, Mark Wiebe wrote: > On Thu, Jun 23, 2011 at 7:28 PM, Pierre GM wrote: > Sorry y'all, I'm just commenting bits by bits: > > "One key problem is a lack of orthogonality with other features, for instance > creating a masked array with physical quantities can't be done

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Benjamin Root
On Thu, Jun 23, 2011 at 7:28 PM, Pierre GM wrote: > Sorry y'all, I'm just commenting bits by bits: > > "One key problem is a lack of orthogonality with other features, for > instance creating a masked array with physical quantities can't be done > because both are separate subclasses of ndarray.

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Mark Wiebe
On Thu, Jun 23, 2011 at 7:28 PM, Pierre GM wrote: > Sorry y'all, I'm just commenting bits by bits: > > "One key problem is a lack of orthogonality with other features, for > instance creating a masked array with physical quantities can't be done > because both are separate subclasses of ndarray.

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Mark Wiebe
On Thu, Jun 23, 2011 at 7:31 PM, Charles R Harris wrote: > On Thu, Jun 23, 2011 at 6:21 PM, Mark Wiebe wrote: > >> On Thu, Jun 23, 2011 at 7:00 PM, Nathaniel Smith wrote: >> >>> On Thu, Jun 23, 2011 at 2:44 PM, Robert Kern >>> wrote: >>> > On Thu, Jun 23, 2011 at 15:53, Mark Wiebe wrote: >>>

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Charles R Harris
On Thu, Jun 23, 2011 at 6:21 PM, Mark Wiebe wrote: > On Thu, Jun 23, 2011 at 7:00 PM, Nathaniel Smith wrote: > >> On Thu, Jun 23, 2011 at 2:44 PM, Robert Kern >> wrote: >> > On Thu, Jun 23, 2011 at 15:53, Mark Wiebe wrote: >> >> Enthought has asked me to look into the "missing data" problem an

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Pierre GM
On Jun 24, 2011, at 2:21 AM, Charles R Harris wrote: > > > On Thu, Jun 23, 2011 at 6:00 PM, Nathaniel Smith wrote: > On Thu, Jun 23, 2011 at 2:44 PM, Robert Kern wrote: > > On Thu, Jun 23, 2011 at 15:53, Mark Wiebe wrote: > >> Enthought has asked me to look into the "missing data" problem an

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Pierre GM
Sorry y'all, I'm just commenting bits by bits: "One key problem is a lack of orthogonality with other features, for instance creating a masked array with physical quantities can't be done because both are separate subclasses of ndarray. The only reasonable way to deal with this is to move the m

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Mark Wiebe
On Thu, Jun 23, 2011 at 7:00 PM, Nathaniel Smith wrote: > On Thu, Jun 23, 2011 at 2:44 PM, Robert Kern > wrote: > > On Thu, Jun 23, 2011 at 15:53, Mark Wiebe wrote: > >> Enthought has asked me to look into the "missing data" problem and how > NumPy > >> could treat it better. I've considered th

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Charles R Harris
On Thu, Jun 23, 2011 at 6:00 PM, Nathaniel Smith wrote: > On Thu, Jun 23, 2011 at 2:44 PM, Robert Kern > wrote: > > On Thu, Jun 23, 2011 at 15:53, Mark Wiebe wrote: > >> Enthought has asked me to look into the "missing data" problem and how > NumPy > >> could treat it better. I've considered th

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Charles R Harris
On Thu, Jun 23, 2011 at 5:51 PM, wrote: > On Thu, Jun 23, 2011 at 5:37 PM, Mark Wiebe wrote: > > On Thu, Jun 23, 2011 at 4:19 PM, Nathaniel Smith wrote: > >> > >> I'd like to see a statement of what the "missing data problem" is, and > >> how this solves it? Because I don't think this is entire

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Nathaniel Smith
On Thu, Jun 23, 2011 at 5:00 PM, Nathaniel Smith wrote: > Speaking as a user who's avoided numpy.ma, it wasn't actually because > of the behavior I pointed out (I never got far enough to notice it), > but because I got the distinct impression that it was a "second-class > citizen" in numpy-land. I

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Nathaniel Smith
On Thu, Jun 23, 2011 at 2:44 PM, Robert Kern wrote: > On Thu, Jun 23, 2011 at 15:53, Mark Wiebe wrote: >> Enthought has asked me to look into the "missing data" problem and how NumPy >> could treat it better. I've considered the different ideas of adding dtype >> variants with a special signal va

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Mark Wiebe
On Thu, Jun 23, 2011 at 6:51 PM, wrote: > On Thu, Jun 23, 2011 at 5:37 PM, Mark Wiebe wrote: > > On Thu, Jun 23, 2011 at 4:19 PM, Nathaniel Smith wrote: > >> > >> I'd like to see a statement of what the "missing data problem" is, and > >> how this solves it? Because I don't think this is entire

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Mark Wiebe
On Thu, Jun 23, 2011 at 6:39 PM, Christopher Barker wrote: > Nathaniel Smith wrote: > > IME > > it's important for the default behavior to be to fail loudly when > > things are wonky, not to silently patch them up, possibly > > incorrectly!) > > +1 for default behavior to fail, raise errors, or re

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread josef . pktd
On Thu, Jun 23, 2011 at 5:37 PM, Mark Wiebe wrote: > On Thu, Jun 23, 2011 at 4:19 PM, Nathaniel Smith wrote: >> >> I'd like to see a statement of what the "missing data problem" is, and >> how this solves it? Because I don't think this is entirely intuitive, >> or that everyone necessarily has th

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Christopher Barker
Nathaniel Smith wrote: > IME > it's important for the default behavior to be to fail loudly when > things are wonky, not to silently patch them up, possibly > incorrectly!) +1 for default behavior to fail, raise errors, or return "unknown values" is these situations. Though that behavior should

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Pierre GM
On Jun 24, 2011, at 12:43 AM, Mark Wiebe wrote: > On Thu, Jun 23, 2011 at 5:24 PM, Pierre GM wrote: > ... > Mmh, after timeseries, now masked arrays... Mark, I start to see a pattern > here ;) > > I think it speaks to what's on Enthought's mind, in any case. :) Eh... > > Anyhow, yes, there

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Mark Wiebe
On Thu, Jun 23, 2011 at 6:33 PM, Charles R Harris wrote: > > > On Thu, Jun 23, 2011 at 5:14 PM, Mark Wiebe wrote: > >> On Thu, Jun 23, 2011 at 6:02 PM, Charles R Harris < >> charlesr.har...@gmail.com> wrote: >> >>> On Thu, Jun 23, 2011 at 4:43 PM, Mark Wiebe wrote: >>> On Thu, Jun 23, 2011

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Charles R Harris
On Thu, Jun 23, 2011 at 5:14 PM, Mark Wiebe wrote: > On Thu, Jun 23, 2011 at 6:02 PM, Charles R Harris < > charlesr.har...@gmail.com> wrote: > >> On Thu, Jun 23, 2011 at 4:43 PM, Mark Wiebe wrote: >> >>> On Thu, Jun 23, 2011 at 5:24 PM, Pierre GM wrote: >>> On Jun 23, 2011, at 11:55 P

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Mark Wiebe
On Thu, Jun 23, 2011 at 6:02 PM, Charles R Harris wrote: > On Thu, Jun 23, 2011 at 4:43 PM, Mark Wiebe wrote: > >> On Thu, Jun 23, 2011 at 5:24 PM, Pierre GM wrote: >> >>> >>> On Jun 23, 2011, at 11:55 PM, Mark Wiebe wrote: >>> >>> > On Thu, Jun 23, 2011 at 4:46 PM, Charles R Harris < >>> charl

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Charles R Harris
On Thu, Jun 23, 2011 at 4:43 PM, Mark Wiebe wrote: > On Thu, Jun 23, 2011 at 5:24 PM, Pierre GM wrote: > >> >> On Jun 23, 2011, at 11:55 PM, Mark Wiebe wrote: >> >> > On Thu, Jun 23, 2011 at 4:46 PM, Charles R Harris < >> charlesr.har...@gmail.com> wrote: >> > On Thu, Jun 23, 2011 at 2:53 PM, Ma

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Mark Wiebe
On Thu, Jun 23, 2011 at 5:24 PM, Pierre GM wrote: > > On Jun 23, 2011, at 11:55 PM, Mark Wiebe wrote: > > > On Thu, Jun 23, 2011 at 4:46 PM, Charles R Harris < > charlesr.har...@gmail.com> wrote: > > On Thu, Jun 23, 2011 at 2:53 PM, Mark Wiebe wrote: > > Enthought has asked me to look into the "

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Mark Wiebe
On Thu, Jun 23, 2011 at 5:05 PM, Keith Goodman wrote: > On Thu, Jun 23, 2011 at 1:53 PM, Mark Wiebe wrote: > > Enthought has asked me to look into the "missing data" problem and how > NumPy > > could treat it better. I've considered the different ideas of adding > dtype > > variants with a speci

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Pierre GM
On Jun 23, 2011, at 11:55 PM, Mark Wiebe wrote: > On Thu, Jun 23, 2011 at 4:46 PM, Charles R Harris > wrote: > On Thu, Jun 23, 2011 at 2:53 PM, Mark Wiebe wrote: > Enthought has asked me to look into the "missing data" problem and how NumPy > could treat it better. I've considered the differe

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Charles R Harris
On Thu, Jun 23, 2011 at 4:08 PM, Robert Kern wrote: > On Thu, Jun 23, 2011 at 17:05, Charles R Harris > wrote: > > > > On Thu, Jun 23, 2011 at 4:04 PM, Robert Kern > wrote: > >> > >> On Thu, Jun 23, 2011 at 17:02, Charles R Harris > >> wrote: > >> > > >> > On Thu, Jun 23, 2011 at 3:48 PM, Gael

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Mark Wiebe
On Thu, Jun 23, 2011 at 4:54 PM, Eric Firing wrote: > On 06/23/2011 11:19 AM, Nathaniel Smith wrote: > > I'd like to see a statement of what the "missing data problem" is, and > > how this solves it? Because I don't think this is entirely intuitive, > > or that everyone necessarily has the same i

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Robert Kern
On Thu, Jun 23, 2011 at 17:05, Charles R Harris wrote: > > On Thu, Jun 23, 2011 at 4:04 PM, Robert Kern wrote: >> >> On Thu, Jun 23, 2011 at 17:02, Charles R Harris >> wrote: >> > >> > On Thu, Jun 23, 2011 at 3:48 PM, Gael Varoquaux >> > wrote: >> >> >> Ufuncs need work, but I have >> >> the im

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Mark Wiebe
On Thu, Jun 23, 2011 at 4:48 PM, Gael Varoquaux < gael.varoqu...@normalesup.org> wrote: > On Thu, Jun 23, 2011 at 03:53:31PM -0500, Mark Wiebe wrote: > >concluded that adding masks to the core ndarray appears is the best > way to > >deal with the problem in general. > > It seems to me that

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Charles R Harris
On Thu, Jun 23, 2011 at 4:04 PM, Robert Kern wrote: > On Thu, Jun 23, 2011 at 17:02, Charles R Harris > wrote: > > > > On Thu, Jun 23, 2011 at 3:48 PM, Gael Varoquaux > > wrote: > > >> Ufuncs need work, but I have > >> the impression that your proposal is simply to solve the special case of > >

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Keith Goodman
On Thu, Jun 23, 2011 at 1:53 PM, Mark Wiebe wrote: > Enthought has asked me to look into the "missing data" problem and how NumPy > could treat it better. I've considered the different ideas of adding dtype > variants with a special signal value and masked arrays, and concluded that > adding masks

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Robert Kern
On Thu, Jun 23, 2011 at 17:02, Charles R Harris wrote: > > On Thu, Jun 23, 2011 at 3:48 PM, Gael Varoquaux > wrote: >> Ufuncs need work, but I have >> the impression that your proposal is simply to solve the special case of >> masked data in the ufunc by breaking the simple numpy array model. >

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Charles R Harris
On Thu, Jun 23, 2011 at 3:48 PM, Gael Varoquaux < gael.varoqu...@normalesup.org> wrote: > On Thu, Jun 23, 2011 at 03:53:31PM -0500, Mark Wiebe wrote: > >concluded that adding masks to the core ndarray appears is the best > way to > >deal with the problem in general. > > It seems to me that

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Mark Wiebe
On Thu, Jun 23, 2011 at 4:46 PM, Charles R Harris wrote: > On Thu, Jun 23, 2011 at 2:53 PM, Mark Wiebe wrote: > >> Enthought has asked me to look into the "missing data" problem and how >> NumPy could treat it better. I've considered the different ideas of adding >> dtype variants with a special

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Eric Firing
On 06/23/2011 11:19 AM, Nathaniel Smith wrote: > I'd like to see a statement of what the "missing data problem" is, and > how this solves it? Because I don't think this is entirely intuitive, > or that everyone necessarily has the same idea. > >> Reduction operations like 'sum', 'prod', 'min', and

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Mark Wiebe
On Thu, Jun 23, 2011 at 4:44 PM, Robert Kern wrote: > On Thu, Jun 23, 2011 at 15:53, Mark Wiebe wrote: > > Enthought has asked me to look into the "missing data" problem and how > NumPy > > could treat it better. I've considered the different ideas of adding > dtype > > variants with a special s

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Charles R Harris
On Thu, Jun 23, 2011 at 3:44 PM, Robert Kern wrote: > On Thu, Jun 23, 2011 at 15:53, Mark Wiebe wrote: > > Enthought has asked me to look into the "missing data" problem and how > NumPy > > could treat it better. I've considered the different ideas of adding > dtype > > variants with a special s

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Gael Varoquaux
On Thu, Jun 23, 2011 at 03:53:31PM -0500, Mark Wiebe wrote: >concluded that adding masks to the core ndarray appears is the best way to >deal with the problem in general. It seems to me that this is going to make the numpy array a way more complex object. Althought it is currently quite si

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Charles R Harris
On Thu, Jun 23, 2011 at 2:53 PM, Mark Wiebe wrote: > Enthought has asked me to look into the "missing data" problem and how > NumPy could treat it better. I've considered the different ideas of adding > dtype variants with a special signal value and masked arrays, and concluded > that adding mask

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Robert Kern
On Thu, Jun 23, 2011 at 15:53, Mark Wiebe wrote: > Enthought has asked me to look into the "missing data" problem and how NumPy > could treat it better. I've considered the different ideas of adding dtype > variants with a special signal value and masked arrays, and concluded that > adding masks t

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Mark Wiebe
On Thu, Jun 23, 2011 at 4:19 PM, Nathaniel Smith wrote: > I'd like to see a statement of what the "missing data problem" is, and > how this solves it? Because I don't think this is entirely intuitive, > or that everyone necessarily has the same idea. > I agree it represents different problems in

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Nathaniel Smith
I'd like to see a statement of what the "missing data problem" is, and how this solves it? Because I don't think this is entirely intuitive, or that everyone necessarily has the same idea. > Reduction operations like 'sum', 'prod', 'min', and 'max' will operate as if > the values weren't there F

[Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Mark Wiebe
Enthought has asked me to look into the "missing data" problem and how NumPy could treat it better. I've considered the different ideas of adding dtype variants with a special signal value and masked arrays, and concluded that adding masks to the core ndarray appears is the best way to deal with th

[Numpy-discussion] Parallelize repeated operations

2011-06-23 Thread Jonathan Tu
Hi, I have found a very strange bug that I cannot understand. I would like to do something like this: Given 4 pairs of numpy arrays (x1, y1, x2, y2, x3, y3, x4, y4), I would like to compute each corresponding inner product ip = np.dot(yi.T, xj), for i = 1,...4 and j = 1,...,4. (Something lik

Re: [Numpy-discussion] review/test pull request for tightening default ufunc casting rule

2011-06-23 Thread Mark Wiebe
This is now merged. -Mark On Wed, Jun 22, 2011 at 9:24 AM, Mark Wiebe wrote: > Pull request is here: > > https://github.com/numpy/numpy/pull/95 > > -Mark > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/list

Re: [Numpy-discussion] A bit of advice?

2011-06-23 Thread josef . pktd
On Thu, Jun 23, 2011 at 8:20 AM, Neal Becker wrote: > Olivier Delalleau wrote: > >> What about : >> dict((k, [e for e in arr if (e['x0'], e['x1']) == k]) for k in cases) >> ? > > Not bad!  Thanks! > > BTW, is there an easier way to get the unique keys, then this: > > cases = tuple (set (tuple((e['

Re: [Numpy-discussion] A bit of advice?

2011-06-23 Thread Neal Becker
Olivier Delalleau wrote: > What about : > dict((k, [e for e in arr if (e['x0'], e['x1']) == k]) for k in cases) > ? Not bad! Thanks! BTW, is there an easier way to get the unique keys, then this: cases = tuple (set (tuple((e['a'],e['b'])) for e in u)) > > (note: it is inefficient written thi

Re: [Numpy-discussion] A bit of advice?

2011-06-23 Thread Olivier Delalleau
What about : dict((k, [e for e in arr if (e['x0'], e['x1']) == k]) for k in cases) ? (note: it is inefficient written this way though) -=- Olivier 2011/6/23 Neal Becker > I have a set of experiments that I want to plot. There will be many plots. > Each will show different test conditions. > >

[Numpy-discussion] A bit of advice?

2011-06-23 Thread Neal Becker
I have a set of experiments that I want to plot. There will be many plots. Each will show different test conditions. Suppose I put each of the test conditions and results into a recarray. The recarray could be: arr = np.empty ((#experiments,), dtype=[('x0',int), ('x1',int), ('y0',int)] wher