Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Laurent Gautier
On 2011-06-24 17:30, Robert Kern wrote: > On Fri, Jun 24, 2011 at 10:07, Laurent Gautier wrote: >> > On 2011-06-24 16:43, Robert Kern wrote: >>> >> >>> >> On Fri, Jun 24, 2011 at 09:33, Charles R Harris >>> >>wrote: >>> > >>> > > >>> > ?On Fri, Jun 24, 2011 at 8:06 AM, Robe

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Charles R Harris
On Fri, Jun 24, 2011 at 10:06 PM, Wes McKinney wrote: > On Fri, Jun 24, 2011 at 11:59 PM, Nathaniel Smith wrote: > > On Fri, Jun 24, 2011 at 6:57 PM, Benjamin Root wrote: > >> On Fri, Jun 24, 2011 at 8:11 PM, Nathaniel Smith wrote: > >>> This is a situation where I would just... use an array a

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Wes McKinney
On Fri, Jun 24, 2011 at 11:59 PM, Nathaniel Smith wrote: > On Fri, Jun 24, 2011 at 6:57 PM, Benjamin Root wrote: >> On Fri, Jun 24, 2011 at 8:11 PM, Nathaniel Smith wrote: >>> This is a situation where I would just... use an array and a mask, >>> rather than a masked array. Then lots of things -

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Nathaniel Smith
On Fri, Jun 24, 2011 at 6:57 PM, Benjamin Root wrote: > On Fri, Jun 24, 2011 at 8:11 PM, Nathaniel Smith wrote: >> This is a situation where I would just... use an array and a mask, >> rather than a masked array. Then lots of things -- changing fill >> values, temporarily masking/unmasking things

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Benjamin Root
On Fri, Jun 24, 2011 at 8:11 PM, Nathaniel Smith wrote: > On Fri, Jun 24, 2011 at 2:09 PM, Benjamin Root wrote: > > Another example of how we use masks in matplotlib is in pcolor(). We > have > > to combine the possible masks of X, Y, and V in both the x and y > directions > > to find the final

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Benjamin Root
On Fri, Jun 24, 2011 at 8:00 PM, Mark Wiebe wrote: > On Fri, Jun 24, 2011 at 6:22 PM, Wes McKinney wrote: > >> On Fri, Jun 24, 2011 at 7:10 PM, Charles R Harris >> wrote: >> > >> > >> > On Fri, Jun 24, 2011 at 4:21 PM, Matthew Brett > > >> > wrote: >> >> >> >> Hi, >> >> >> >> On Fri, Jun 24, 2

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Nathaniel Smith
On Fri, Jun 24, 2011 at 2:09 PM, Benjamin Root wrote: > Another example of how we use masks in matplotlib is in pcolor().  We have > to combine the possible masks of X, Y, and V in both the x and y directions > to find the final mask to use for the final output result (because each > facet needs v

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 7:02 PM, Matthew Brett wrote: > Hi, > > On Sat, Jun 25, 2011 at 12:22 AM, Wes McKinney > wrote: > ... > > Perhaps we should make a wiki page someplace summarizing pros and cons > > of the various implementation approaches? > > But - we should do this if it really is an ope

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 6:22 PM, Wes McKinney wrote: > On Fri, Jun 24, 2011 at 7:10 PM, Charles R Harris > wrote: > > > > > > On Fri, Jun 24, 2011 at 4:21 PM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> On Fri, Jun 24, 2011 at 10:09 PM, Benjamin Root > wrote: > >> ... > >> > Again, there

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 6:10 PM, Charles R Harris wrote: > > > On Fri, Jun 24, 2011 at 4:21 PM, Matthew Brett wrote: > >> Hi, >> >> On Fri, Jun 24, 2011 at 10:09 PM, Benjamin Root wrote: >> ... >> > Again, there are pros and cons either way and I see them very orthogonal >> and >> > complementar

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 5:21 PM, Matthew Brett wrote: > Hi, > > On Fri, Jun 24, 2011 at 10:09 PM, Benjamin Root wrote: > ... > > Again, there are pros and cons either way and I see them very orthogonal > and > > complementary. > > That may be true, but I imagine only one of them will be implement

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 4:24 PM, Nathaniel Smith wrote: > On Fri, Jun 24, 2011 at 12:26 PM, Mark Wiebe wrote: > > For the maybe dtype, it would need to gain access to the ufunc loop of > the > > underlying dtype, and call it appropriately during the inner loop. This > > appears to require some m

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 4:09 PM, Benjamin Root wrote: > > > On Fri, Jun 24, 2011 at 10:40 AM, Mark Wiebe wrote: > >> On Thu, Jun 23, 2011 at 7:56 PM, Benjamin Root wrote: >> >>> On Thu, Jun 23, 2011 at 7:28 PM, Pierre GM wrote: >>> Sorry y'all, I'm just commenting bits by bits:

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Charles R Harris
On Fri, Jun 24, 2011 at 6:11 PM, Wes McKinney wrote: > On Fri, Jun 24, 2011 at 8:02 PM, Charles R Harris > wrote: > > > > > > On Fri, Jun 24, 2011 at 5:22 PM, Wes McKinney > wrote: > >> > >> On Fri, Jun 24, 2011 at 7:10 PM, Charles R Harris > >> wrote: > >> > > >> > > >> > On Fri, Jun 24, 2011

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 3:38 PM, Lluís wrote: > Mark Wiebe writes: > > > It's should also be possible to accomplish a general > > solution at the dtype level. We could have a 'dtype > > factory' used like: np.zeros(10, dtype=np.maybe(float)) > > wh

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Wes McKinney
On Fri, Jun 24, 2011 at 8:02 PM, Charles R Harris wrote: > > > On Fri, Jun 24, 2011 at 5:22 PM, Wes McKinney wrote: >> >> On Fri, Jun 24, 2011 at 7:10 PM, Charles R Harris >> wrote: >> > >> > >> > On Fri, Jun 24, 2011 at 4:21 PM, Matthew Brett >> > wrote: >> >> >> >> Hi, >> >> >> >> On Fri, Jun

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Charles R Harris
On Fri, Jun 24, 2011 at 5:22 PM, Wes McKinney wrote: > On Fri, Jun 24, 2011 at 7:10 PM, Charles R Harris > wrote: > > > > > > On Fri, Jun 24, 2011 at 4:21 PM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> On Fri, Jun 24, 2011 at 10:09 PM, Benjamin Root > wrote: > >> ... > >> > Again, there

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Matthew Brett
Hi, On Sat, Jun 25, 2011 at 12:22 AM, Wes McKinney wrote: ... > Perhaps we should make a wiki page someplace summarizing pros and cons > of the various implementation approaches? But - we should do this if it really is an open question which one we go for. If not then, we're just slowing Mark

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Wes McKinney
On Fri, Jun 24, 2011 at 7:10 PM, Charles R Harris wrote: > > > On Fri, Jun 24, 2011 at 4:21 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Fri, Jun 24, 2011 at 10:09 PM, Benjamin Root wrote: >> ... >> > Again, there are pros and cons either way and I see them very orthogonal >> > and >> > complem

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Charles R Harris
On Fri, Jun 24, 2011 at 4:21 PM, Matthew Brett wrote: > Hi, > > On Fri, Jun 24, 2011 at 10:09 PM, Benjamin Root wrote: > ... > > Again, there are pros and cons either way and I see them very orthogonal > and > > complementary. > > That may be true, but I imagine only one of them will be implement

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Gael Varoquaux
On Thu, Jun 23, 2011 at 07:51:25PM -0400, josef.p...@gmail.com wrote: > From the perspective of statistical analysis, I don't see much > advantage of this. What to do with nans depends on the analysis, and > needs to be looked at for each case. >From someone who actually sometimes does statistics

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Matthew Brett
Hi, On Fri, Jun 24, 2011 at 10:09 PM, Benjamin Root wrote: ... > Again, there are pros and cons either way and I see them very orthogonal and > complementary. That may be true, but I imagine only one of them will be implemented. @Mark - I don't have a clear idea whether you consider the nafloat

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Nathaniel Smith
On Fri, Jun 24, 2011 at 12:26 PM, Mark Wiebe wrote: > For the maybe dtype, it would need to gain access to the ufunc loop of the > underlying dtype, and call it appropriately during the inner loop. This > appears to require some more invasive upheaval within the ufunc code than > the masking appro

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Benjamin Root
On Fri, Jun 24, 2011 at 10:40 AM, Mark Wiebe wrote: > On Thu, Jun 23, 2011 at 7:56 PM, Benjamin Root wrote: > >> On Thu, Jun 23, 2011 at 7:28 PM, Pierre GM wrote: >> >>> Sorry y'all, I'm just commenting bits by bits: >>> >>> "One key problem is a lack of orthogonality with other features, for >

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Lluís
Mark Wiebe writes: > It's should also be possible to accomplish a general > solution at the dtype level. We could have a 'dtype > factory' used like:  np.zeros(10, dtype=np.maybe(float)) > where np.maybe(x) returns a new dtype whose storage size >

Re: [Numpy-discussion] How to avoid extra copying when forming an array from an iterator

2011-06-24 Thread Robert Kern
On Fri, Jun 24, 2011 at 14:38, srean wrote: > A valiant exercise in hope: > > Is this possible to do it without a loop or extra copying. What I have is an > iterator that yields a fixed with string on every call to next(). Now I want > to create a numpy array of ints out of the last 4 chars of tha

[Numpy-discussion] [ANN] Numerical integration with guaranteed precision by interalg

2011-06-24 Thread Dmitrey
Hi all, some ideas implemented in the solver interalg (INTERval ALGorithm) that already turn out to be more effective than its competitors in numerical optimization (benchmark) appears to be extremely effective in numerical integration with guaranteed precision. Here are some exampl

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 1:18 PM, Matthew Brett wrote: > Hi, > > On Fri, Jun 24, 2011 at 5:45 PM, Mark Wiebe wrote: > > On Fri, Jun 24, 2011 at 6:59 AM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> On Fri, Jun 24, 2011 at 2:32 AM, Nathaniel Smith wrote: > ... > >> and the fact that 'missing_

Re: [Numpy-discussion] How to avoid extra copying when forming an array from an iterator

2011-06-24 Thread srean
A valiant exercise in hope: Is this possible to do it without a loop or extra copying. What I have is an iterator that yields a fixed with string on every call to next(). Now I want to create a numpy array of ints out of the last 4 chars of that string. My plan was to pass the iterator through a

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 1:04 PM, Matthew Brett wrote: > Hi, > > Just as a use case, if I do this: > > a = np.zeros((big_number,), dtype=np.int32) > a[0,0] = np.NA > > I think I'm right in saying that, with the array.mask implementation > my array memory usage with grow by new big_number bytes, whe

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 12:06 PM, Wes McKinney wrote: > On Fri, Jun 24, 2011 at 12:33 PM, Mark Wiebe wrote: > > On Thu, Jun 23, 2011 at 8:32 PM, Nathaniel Smith wrote: > >> > >> On Thu, Jun 23, 2011 at 5:21 PM, Mark Wiebe wrote: > >> > On Thu, Jun 23, 2011 at 7:00 PM, Nathaniel Smith > wrote:

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 11:54 AM, Nathaniel Smith wrote: > On Fri, Jun 24, 2011 at 9:33 AM, Mark Wiebe wrote: > > On Thu, Jun 23, 2011 at 8:32 PM, Nathaniel Smith wrote: > >> But on the other hand, we gain: > >> -- simpler implementation: no need to be checking and tracking the > >> mask buffe

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Matthew Brett
Hi, On Fri, Jun 24, 2011 at 5:45 PM, Mark Wiebe wrote: > On Fri, Jun 24, 2011 at 6:59 AM, Matthew Brett > wrote: >> >> Hi, >> >> On Fri, Jun 24, 2011 at 2:32 AM, Nathaniel Smith wrote: ... >> and the fact that 'missing_value' could be any type would make the >> code more complicated than the cu

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 11:25 AM, Christopher Barker wrote: > Robert Kern wrote: > > > It's worth noting that this is not a replacement for masked arrays, > > nor is it intended to be the be-all, end-all solution to missing data > > problems. It's mostly just intended to be a focused tool to fill

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 11:25 AM, Robert Kern wrote: > On Fri, Jun 24, 2011 at 11:13, Christopher Barker > wrote: > > Nathaniel Smith wrote: > > > >> If we think that the memory overhead for floating point types is too > >> high, it would be easy to add a special case where maybe(float) used a >

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 11:13 AM, Christopher Barker wrote: > Nathaniel Smith wrote: > >> The 'dtype factory' idea builds on the way I've structured datetime as a > >> parameterized type, > > ... > > Another disadvantage is that we get further from Gael Varoquaux's point: > >> Right now, the nump

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Matthew Brett
Hi, Just as a use case, if I do this: a = np.zeros((big_number,), dtype=np.int32) a[0,0] = np.NA I think I'm right in saying that, with the array.mask implementation my array memory usage with grow by new big_number bytes, whereas with the np.naint32 implementation you'd get something like: Err

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 10:07 AM, Matthew Brett wrote: > Hi, > > On Fri, Jun 24, 2011 at 3:43 PM, Robert Kern > wrote: > > On Fri, Jun 24, 2011 at 09:33, Charles R Harris > > wrote: > >> > >> On Fri, Jun 24, 2011 at 8:06 AM, Robert Kern > wrote: > > > >>> The alternative proposal would be to ad

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 10:02 AM, Pierre GM wrote: > On Jun 24, 2011, at 4:44 PM, Robert Kern wrote: > > > On Fri, Jun 24, 2011 at 09:35, Robert Kern > wrote: > >> On Fri, Jun 24, 2011 at 09:24, Keith Goodman > wrote: > >>> On Fri, Jun 24, 2011 at 7:06 AM, Robert Kern > wrote: > >>> > The

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 9:27 AM, Bruce Southey wrote: > ** > On 06/24/2011 09:06 AM, Robert Kern wrote: > > On Fri, Jun 24, 2011 at 07:30, Laurent Gautier > wrote: > > On 2011-06-24 13:59, Nathaniel Smith > wrote: > > On Thu, Jun 23, 2011 at 5:56 PM, Benjamin Root > wrote: > > Lastly

[Numpy-discussion] correlation in fourier domain

2011-06-24 Thread Alex Flint
I would like to perform 2d correlations between a large matrix A and a bunch of small matrices B1...Bn. Is there anything built into numpy to help me do this efficiently in the fourier domain (i.e. DFT of A -> multiplication with DFT of B1..Bn)? Cheers, Alex ___

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Wes McKinney
On Fri, Jun 24, 2011 at 12:33 PM, Mark Wiebe wrote: > On Thu, Jun 23, 2011 at 8:32 PM, Nathaniel Smith wrote: >> >> On Thu, Jun 23, 2011 at 5:21 PM, Mark Wiebe wrote: >> > On Thu, Jun 23, 2011 at 7:00 PM, Nathaniel Smith wrote: >> >> It's should also be possible to accomplish a general solution

Re: [Numpy-discussion] How to avoid extra copying when forming an array from an iterator

2011-06-24 Thread srean
On Fri, Jun 24, 2011 at 9:12 AM, Robert Kern wrote: > On Fri, Jun 24, 2011 at 04:03, srean wrote: > > To answer my own question, I guess I can keep appending to a > array.array() > > object and get a numpy.array from its buffer if possible. Is that the > > efficient way. > > It's one of the mos

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 8:57 AM, Keith Goodman wrote: > On Thu, Jun 23, 2011 at 3:24 PM, Mark Wiebe wrote: > > On Thu, Jun 23, 2011 at 5:05 PM, Keith Goodman > wrote: > >> > >> On Thu, Jun 23, 2011 at 1:53 PM, Mark Wiebe wrote: > >> > Enthought has asked me to look into the "missing data" prob

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Nathaniel Smith
On Fri, Jun 24, 2011 at 9:33 AM, Mark Wiebe wrote: > On Thu, Jun 23, 2011 at 8:32 PM, Nathaniel Smith wrote: >> But on the other hand, we gain: >>  -- simpler implementation: no need to be checking and tracking the >> mask buffer everywhere. The needed infrastructure is already built in. > > I do

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 8:01 AM, Neal Becker wrote: > Just 1 question before I look more closely. What is the cost to the non-MA > user > of this addition? > I'm following the idea that you don't pay for what you don't use. All the existing stuff will perform the same. -Mark > __

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 7:30 AM, Laurent Gautier wrote: > On 2011-06-24 13:59, Nathaniel Smith wrote: > > On Thu, Jun 23, 2011 at 5:56 PM, Benjamin Root wrote: > >> Lastly, I am not entirely familiar with R, so I am also very curious > about > >> what this magical "NA" value is, and how it com

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Fri, Jun 24, 2011 at 6:59 AM, Matthew Brett wrote: > Hi, > > On Fri, Jun 24, 2011 at 2:32 AM, Nathaniel Smith wrote: > ... > > If we think that the memory overhead for floating point types is too > > high, it would be easy to add a special case where maybe(float) used a > > distinguished NaN i

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Thu, Jun 23, 2011 at 8:32 PM, Nathaniel Smith wrote: > On Thu, Jun 23, 2011 at 5:21 PM, Mark Wiebe wrote: > > On Thu, Jun 23, 2011 at 7:00 PM, Nathaniel Smith wrote: > >> It's should also be possible to accomplish a general solution at the > >> dtype level. We could have a 'dtype factory' us

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Nathaniel Smith
On Fri, Jun 24, 2011 at 7:06 AM, Robert Kern wrote: > The alternative proposal would be to add a few new dtypes that are > NA-aware. E.g. an nafloat64 would reserve a particular NaN value > (there are lots of different NaN bit patterns, we'd just reserve one) > that would represent NA. An naint32

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Christopher Barker
Robert Kern wrote: > It's worth noting that this is not a replacement for masked arrays, > nor is it intended to be the be-all, end-all solution to missing data > problems. It's mostly just intended to be a focused tool to fill in > the gaps where masked arrays are less convenient for whatever rea

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Robert Kern
On Fri, Jun 24, 2011 at 11:13, Christopher Barker wrote: > Nathaniel Smith wrote: > >> If we think that the memory overhead for floating point types is too >> high, it would be easy to add a special case where maybe(float) used a >> distinguished NaN instead of a separate boolean. > > That would  

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Thu, Jun 23, 2011 at 8:00 PM, Pierre GM wrote: > > On Jun 24, 2011, at 2:42 AM, Mark Wiebe wrote: > > > On Thu, Jun 23, 2011 at 7:28 PM, Pierre GM wrote: > > Sorry y'all, I'm just commenting bits by bits: > > > > "One key problem is a lack of orthogonality with other features, for > instance

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Christopher Barker
Nathaniel Smith wrote: >> The 'dtype factory' idea builds on the way I've structured datetime as a >> parameterized type, ... Another disadvantage is that we get further from Gael Varoquaux's point: >> Right now, the numpy array can be seen as an extension of the C >> array, basically a pointer

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Nathaniel Smith
On Fri, Jun 24, 2011 at 8:30 AM, Robert Kern wrote: > I would suggest following R's lead and letting ((NA==NA) == True) > unlike NaNs. In R, NA and NaN do behave differently with respect to ==, but not the way you're saying: > NA == NA [1] NA > if (NA == NA) 1; Error in if (NA == NA) 1 : missing

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Robert Kern
On Fri, Jun 24, 2011 at 11:05, Nathaniel Smith wrote: > On Fri, Jun 24, 2011 at 8:14 AM, Robert Kern wrote: >> On Fri, Jun 24, 2011 at 10:07, Laurent Gautier wrote: >>> May be there is not so much need for reservation over the string NA, when >>> making the distinction between: >>> a- the intern

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Nathaniel Smith
On Fri, Jun 24, 2011 at 8:14 AM, Robert Kern wrote: > On Fri, Jun 24, 2011 at 10:07, Laurent Gautier wrote: >> May be there is not so much need for reservation over the string NA, when >> making the distinction between: >> a- the internal representation of a "missing string" (what is stored in >>

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Mark Wiebe
On Thu, Jun 23, 2011 at 7:56 PM, Benjamin Root wrote: > On Thu, Jun 23, 2011 at 7:28 PM, Pierre GM wrote: > >> Sorry y'all, I'm just commenting bits by bits: >> >> "One key problem is a lack of orthogonality with other features, for >> instance creating a masked array with physical quantities ca

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Robert Kern
On Fri, Jun 24, 2011 at 10:02, Pierre GM wrote: > > On Jun 24, 2011, at 4:44 PM, Robert Kern wrote: > >> On Fri, Jun 24, 2011 at 09:35, Robert Kern wrote: >>> On Fri, Jun 24, 2011 at 09:24, Keith Goodman wrote: On Fri, Jun 24, 2011 at 7:06 AM, Robert Kern wrote: > The alternative

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Robert Kern
On Fri, Jun 24, 2011 at 10:07, Laurent Gautier wrote: > On 2011-06-24 16:43, Robert Kern wrote: >> >> On Fri, Jun 24, 2011 at 09:33, Charles R Harris >> wrote: >>> >>> > >>> >  On Fri, Jun 24, 2011 at 8:06 AM, Robert Kern >>> >  wrote: >>  The alternative proposal would be to add a few

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Matthew Brett
Hi, On Fri, Jun 24, 2011 at 3:43 PM, Robert Kern wrote: > On Fri, Jun 24, 2011 at 09:33, Charles R Harris > wrote: >> >> On Fri, Jun 24, 2011 at 8:06 AM, Robert Kern wrote: > >>> The alternative proposal would be to add a few new dtypes that are >>> NA-aware. E.g. an nafloat64 would reserve a p

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Laurent Gautier
On 2011-06-24 16:43, Robert Kern wrote: > On Fri, Jun 24, 2011 at 09:33, Charles R Harris > wrote: >> > >> > On Fri, Jun 24, 2011 at 8:06 AM, Robert Kern >> > wrote: >>> >> The alternative proposal would be to add a few new dtypes that are >>> >> NA-aware. E.g. an nafloat64 would reserve a

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Pierre GM
On Jun 24, 2011, at 4:44 PM, Robert Kern wrote: > On Fri, Jun 24, 2011 at 09:35, Robert Kern wrote: >> On Fri, Jun 24, 2011 at 09:24, Keith Goodman wrote: >>> On Fri, Jun 24, 2011 at 7:06 AM, Robert Kern wrote: >>> The alternative proposal would be to add a few new dtypes that are N

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Charles R Harris
On Fri, Jun 24, 2011 at 8:44 AM, Robert Kern wrote: > On Fri, Jun 24, 2011 at 09:35, Robert Kern wrote: > > On Fri, Jun 24, 2011 at 09:24, Keith Goodman > wrote: > >> On Fri, Jun 24, 2011 at 7:06 AM, Robert Kern > wrote: > >> > >>> The alternative proposal would be to add a few new dtypes that

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Robert Kern
On Fri, Jun 24, 2011 at 09:35, Robert Kern wrote: > On Fri, Jun 24, 2011 at 09:24, Keith Goodman wrote: >> On Fri, Jun 24, 2011 at 7:06 AM, Robert Kern wrote: >> >>> The alternative proposal would be to add a few new dtypes that are >>> NA-aware. E.g. an nafloat64 would reserve a particular NaN

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Robert Kern
On Fri, Jun 24, 2011 at 09:33, Charles R Harris wrote: > > On Fri, Jun 24, 2011 at 8:06 AM, Robert Kern wrote: >> The alternative proposal would be to add a few new dtypes that are >> NA-aware. E.g. an nafloat64 would reserve a particular NaN value >> (there are lots of different NaN bit pattern

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Robert Kern
On Fri, Jun 24, 2011 at 09:24, Keith Goodman wrote: > On Fri, Jun 24, 2011 at 7:06 AM, Robert Kern wrote: > >> The alternative proposal would be to add a few new dtypes that are >> NA-aware. E.g. an nafloat64 would reserve a particular NaN value >> (there are lots of different NaN bit patterns, w

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Charles R Harris
On Fri, Jun 24, 2011 at 8:06 AM, Robert Kern wrote: > On Fri, Jun 24, 2011 at 07:30, Laurent Gautier wrote: > > On 2011-06-24 13:59, Nathaniel Smith wrote: > >> On Thu, Jun 23, 2011 at 5:56 PM, Benjamin Root wrote: > >>> Lastly, I am not entirely familiar with R, so I am also very curious > a

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Bruce Southey
On 06/24/2011 09:06 AM, Robert Kern wrote: On Fri, Jun 24, 2011 at 07:30, Laurent Gautier wrote: On 2011-06-24 13:59, Nathaniel Smith wrote: On Thu, Jun 23, 2011 at 5:56 PM, Benjamin Rootwrote: Lastly, I am not entirely familiar with R, so I am also very curious about what this magical

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Keith Goodman
On Fri, Jun 24, 2011 at 7:06 AM, Robert Kern wrote: > The alternative proposal would be to add a few new dtypes that are > NA-aware. E.g. an nafloat64 would reserve a particular NaN value > (there are lots of different NaN bit patterns, we'd just reserve one) > that would represent NA. An naint32

Re: [Numpy-discussion] How to avoid extra copying when forming an array from an iterator

2011-06-24 Thread Robert Kern
On Fri, Jun 24, 2011 at 04:03, srean wrote: > To answer my own question, I guess I can keep appending to a array.array() > object and get a numpy.array from its buffer if possible.  Is that the > efficient way. It's one of the most efficient ways to do it, yes, especially for 1D arrays. -- Robe

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Robert Kern
On Fri, Jun 24, 2011 at 07:30, Laurent Gautier wrote: > On 2011-06-24 13:59,  Nathaniel Smith wrote: >> On Thu, Jun 23, 2011 at 5:56 PM, Benjamin Root  wrote: >>> Lastly, I am not entirely familiar with R, so I am also very curious about >>> what this magical "NA" value is, and how it compares to

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Robert Kern
On Fri, Jun 24, 2011 at 06:47, Matthew Brett wrote: > Hi, > > On Thu, Jun 23, 2011 at 10:44 PM, Robert Kern wrote: >> On Thu, Jun 23, 2011 at 15:53, Mark Wiebe wrote: >>> Enthought has asked me to look into the "missing data" problem and how NumPy >>> could treat it better. I've considered the d

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Keith Goodman
On Thu, Jun 23, 2011 at 3:24 PM, Mark Wiebe wrote: > On Thu, Jun 23, 2011 at 5:05 PM, Keith Goodman wrote: >> >> On Thu, Jun 23, 2011 at 1:53 PM, Mark Wiebe wrote: >> > Enthought has asked me to look into the "missing data" problem and how >> > NumPy >> > could treat it better. I've considered t

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Charles R Harris
On Fri, Jun 24, 2011 at 6:30 AM, Laurent Gautier wrote: > On 2011-06-24 13:59, Nathaniel Smith wrote: > > On Thu, Jun 23, 2011 at 5:56 PM, Benjamin Root wrote: > >> Lastly, I am not entirely familiar with R, so I am also very curious > about > >> what this magical "NA" value is, and how it com

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Neal Becker
Just 1 question before I look more closely. What is the cost to the non-MA user of this addition? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Laurent Gautier
On 2011-06-24 13:59, Nathaniel Smith wrote: > On Thu, Jun 23, 2011 at 5:56 PM, Benjamin Root wrote: >> Lastly, I am not entirely familiar with R, so I am also very curious about >> what this magical "NA" value is, and how it compares to how NaNs work. >> Although, Pierre brought up the very good

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Matthew Brett
Hi, On Fri, Jun 24, 2011 at 2:32 AM, Nathaniel Smith wrote: ... > If we think that the memory overhead for floating point types is too > high, it would be easy to add a special case where maybe(float) used a > distinguished NaN instead of a separate boolean. The extra complexity > would be isolat

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Matthew Brett
Hi, On Thu, Jun 23, 2011 at 10:44 PM, Robert Kern wrote: > On Thu, Jun 23, 2011 at 15:53, Mark Wiebe wrote: >> Enthought has asked me to look into the "missing data" problem and how NumPy >> could treat it better. I've considered the different ideas of adding dtype >> variants with a special sig

[Numpy-discussion] Pleasantly surprised by recarray

2011-06-24 Thread Neal Becker
I was pleasantly surprised to find that recarrays can have a recursive structure, and can be defined using a nice syntax: dtype=[('deltaf', float),('eq', bool),('filt', [('pulse', 'a10'), ('alpha', float)])] However, this I discovered just by guessing. It would be good to mention this in the

Re: [Numpy-discussion] A bit of advice?

2011-06-24 Thread Neal Becker
josef.p...@gmail.com wrote: > On Thu, Jun 23, 2011 at 8:20 AM, Neal Becker wrote: >> Olivier Delalleau wrote: >> >>> What about : >>> dict((k, [e for e in arr if (e['x0'], e['x1']) == k]) for k in cases) >>> ? >> >> Not bad! Thanks! >> >> BTW, is there an easier way to get the unique keys, then

Re: [Numpy-discussion] How to avoid extra copying when forming an array from an iterator

2011-06-24 Thread srean
To answer my own question, I guess I can keep appending to a array.array() object and get a numpy.array from its buffer if possible. Is that the efficient way. On Fri, Jun 24, 2011 at 2:35 AM, srean wrote: > Hi, > > I have an iterator that yields a complex object. I want to make an array > out

[Numpy-discussion] How to avoid extra copying when forming an array from an iterator

2011-06-24 Thread srean
Hi, I have a iterator that yields a complex object. I want to make an array out of a numerical attribute that the yielded object possesses and that too very efficiently. My initial plan was to keep writing the numbers to a StringIO object and when done generate the numpy array using StringIO's bu