Re: [Numpy-discussion] Concepts for masked/missing data

On Sat, Jun 25, 2011 at 3:51 PM, Nathaniel Smith wrote: > On Sat, Jun 25, 2011 at 11:32 AM, Benjamin Root wrote: >> On Sat, Jun 25, 2011 at 12:05 PM, Nathaniel Smith wrote: >>> I guess that is a difference, but I'm trying to get at something more >>> fundamental -- not just what operations are a

Re: [Numpy-discussion] Concepts for masked/missing data

On 06/25/2011 09:09 AM, Benjamin Root wrote: > > > On Sat, Jun 25, 2011 at 1:57 PM, Nathaniel Smith > wrote: > > On Sat, Jun 25, 2011 at 11:50 AM, Eric Firing > wrote: > > On 06/25/2011 07:05 AM, Nathaniel Smith wrote: > >> On Sa

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 3:25 PM, Gael Varoquaux < gael.varoqu...@normalesup.org> wrote: > On Sat, Jun 25, 2011 at 03:16:39PM -0500, Mark Wiebe wrote: > >This is why I'm also proposing to add a 'mask=' parameter to ufuncs, > for > >example, to expose the implementation details of the masked

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 9:44 AM, Wes McKinney wrote: > On Sat, Jun 25, 2011 at 10:25 AM, Charles R Harris > wrote: > > On Sat, Jun 25, 2011 at 8:14 AM, Wes McKinney > wrote: > >> > >> On Sat, Jun 25, 2011 at 12:42 AM, Charles R Harris > >> wrote: > >> > > >> > > >> > On Fri, Jun 24, 2011 at 1

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 9:21 AM, Charles R Harris wrote: > On Sat, Jun 25, 2011 at 5:29 AM, Pierre GM wrote: > >> This thread is getting quite long, innit ? >> And I think it's getting a tad confusing, because we're mixing two >> different concepts: missing values and masks. >> There should be s

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 03:16:39PM -0500, Mark Wiebe wrote: >This is why I'm also proposing to add a 'mask=' parameter to ufuncs, for >example, to expose the implementation details of the masked array system >to people who need masks but need them to be a bit different. There may be >

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 7:00 AM, Gael Varoquaux < gael.varoqu...@normalesup.org> wrote: > On Sat, Jun 25, 2011 at 01:02:07AM +0100, Matthew Brett wrote: > > I'm personally worried that the memory overhead of array.masks will > > make many of us tend to avoid them. I work with images that can > >

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 9:14 AM, Wes McKinney wrote: > On Sat, Jun 25, 2011 at 12:42 AM, Charles R Harris > wrote: > > > > > > On Fri, Jun 24, 2011 at 10:06 PM, Wes McKinney > wrote: > >> > >> On Fri, Jun 24, 2011 at 11:59 PM, Nathaniel Smith > wrote: > >> > On Fri, Jun 24, 2011 at 6:57 PM, Be

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 6:29 AM, Pierre GM wrote: > This thread is getting quite long, innit ? > It's tiring, yeah! > And I think it's getting a tad confusing, because we're mixing two > different concepts: missing values and masks. > There should be support for missing values in numpy.core, I

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 6:17 AM, Matthew Brett wrote: > Hi, > > On Sat, Jun 25, 2011 at 2:10 AM, Mark Wiebe wrote: > > On Fri, Jun 24, 2011 at 7:02 PM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> On Sat, Jun 25, 2011 at 12:22 AM, Wes McKinney > >> wrote: > >> ... > >> > Perhaps we should m

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 6:00 AM, Matthew Brett wrote: > Hi, > > On Sat, Jun 25, 2011 at 1:54 AM, Mark Wiebe wrote: > > On Fri, Jun 24, 2011 at 5:21 PM, Matthew Brett > ... > >> @Mark - I don't have a clear idea whether you consider the nafloat64 > >> option to be still in play as the first thing

Re: [Numpy-discussion] Concepts for masked/missing data

On Sat, Jun 25, 2011 at 11:32 AM, Benjamin Root wrote: > On Sat, Jun 25, 2011 at 12:05 PM, Nathaniel Smith wrote: >> I guess that is a difference, but I'm trying to get at something more >> fundamental -- not just what operations are allowed, but what >> operations people *expect* to be allowed.

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Fri, Jun 24, 2011 at 11:06 PM, Wes McKinney wrote: > On Fri, Jun 24, 2011 at 11:59 PM, Nathaniel Smith wrote: > > On Fri, Jun 24, 2011 at 6:57 PM, Benjamin Root wrote: > >> On Fri, Jun 24, 2011 at 8:11 PM, Nathaniel Smith wrote: > >>> This is a situation where I would just... use an array a

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Fri, Jun 24, 2011 at 10:59 PM, Nathaniel Smith wrote: > On Fri, Jun 24, 2011 at 6:57 PM, Benjamin Root wrote: > > On Fri, Jun 24, 2011 at 8:11 PM, Nathaniel Smith wrote: > >> This is a situation where I would just... use an array and a mask, > >> rather than a masked array. Then lots of thin

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 9:21 AM, Charles R Harris wrote: > > I think he aims to support both. One complication with masks is keeping > them tied to the data on disk. With na values one file can contain both the > data and the missing data markers, whereas with masks, two files would be > required

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Fri, Jun 24, 2011 at 8:25 PM, Benjamin Root wrote: > On Fri, Jun 24, 2011 at 8:00 PM, Mark Wiebe wrote: > >> On Fri, Jun 24, 2011 at 6:22 PM, Wes McKinney wrote: >> >>> On Fri, Jun 24, 2011 at 7:10 PM, Charles R Harris >>> wrote: >>> > >>> > >>> > On Fri, Jun 24, 2011 at 4:21 PM, Matthew Br

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Fri, Jun 24, 2011 at 8:11 PM, Nathaniel Smith wrote: > On Fri, Jun 24, 2011 at 2:09 PM, Benjamin Root wrote: > > Another example of how we use masks in matplotlib is in pcolor(). We > have > > to combine the possible masks of X, Y, and V in both the x and y > directions > > to find the final

Re: [Numpy-discussion] Concepts for masked/missing data

On Sat, Jun 25, 2011 at 1:18 PM, Alan G Isaac wrote: > On 6/25/2011 2:06 PM, Benjamin Root wrote: > > Note that "np.sum([])" also returns 0.0. I think the > > reason why it has been returning zero instead of NaN was > > because there wasn't a NaN-equivalent for integers. > > > http://en.wikipedi

Re: [Numpy-discussion] Concepts for masked/missing data

On Sat, Jun 25, 2011 at 1:57 PM, Nathaniel Smith wrote: > On Sat, Jun 25, 2011 at 11:50 AM, Eric Firing wrote: > > On 06/25/2011 07:05 AM, Nathaniel Smith wrote: > >> On Sat, Jun 25, 2011 at 9:26 AM, Matthew Brett > wrote: > >>> To clarify, you're proposing for: > >>> > >>> a = np.sum(np.array(

Re: [Numpy-discussion] Concepts for masked/missing data

On Sat, Jun 25, 2011 at 12:17 PM, Wes McKinney wrote: > > Agree. My basic observation about numpy.ma is that it's a finely > crafted solution for a different set of problems than the ones I have. > I just don't want the same thing to happen here so I'm stuck writing > code (like I am now) that lo

Re: [Numpy-discussion] Concepts for masked/missing data

On Sat, Jun 25, 2011 at 11:50 AM, Eric Firing wrote: > On 06/25/2011 07:05 AM, Nathaniel Smith wrote: >> On Sat, Jun 25, 2011 at 9:26 AM, Matthew Brett   >> wrote: >>> To clarify, you're proposing for: >>> >>> a = np.sum(np.array([np.NA, np.NA]) >>> >>> 1) ->  np.NA >>> 2) ->  0.0 >> >> Yes -- and

Re: [Numpy-discussion] Concepts for masked/missing data

On 06/25/2011 07:05 AM, Nathaniel Smith wrote: > On Sat, Jun 25, 2011 at 9:26 AM, Matthew Brett > wrote: >> So far I see the difference between 1) and 2) being that you cannot >> unmask. So, if you didn't even know you could unmask data, then it >> would not matter that 1) was being implemented

Re: [Numpy-discussion] Concepts for masked/missing data

On Sat, Jun 25, 2011 at 12:05 PM, Nathaniel Smith wrote: > On Sat, Jun 25, 2011 at 9:26 AM, Matthew Brett > wrote: > > So far I see the difference between 1) and 2) being that you cannot > > unmask. So, if you didn't even know you could unmask data, then it > > would not matter that 1) was bein

Re: [Numpy-discussion] Concepts for masked/missing data

On 6/25/2011 2:06 PM, Benjamin Root wrote: > Note that "np.sum([])" also returns 0.0. I think the > reason why it has been returning zero instead of NaN was > because there wasn't a NaN-equivalent for integers. http://en.wikipedia.org/wiki/Empty_sum fwiw, Alan Isaac

Re: [Numpy-discussion] Concepts for masked/missing data

On Sat, Jun 25, 2011 at 11:26 AM, Matthew Brett wrote: > Hi, > > On Sat, Jun 25, 2011 at 5:05 PM, Nathaniel Smith wrote: > > So obviously there's a lot of interest in this question, but I'm > > losing track of all the different issues that've being raised in the > > 150-post thread of doom. I thi

Re: [Numpy-discussion] Concepts for masked/missing data

On Sat, Jun 25, 2011 at 1:05 PM, Nathaniel Smith wrote: > On Sat, Jun 25, 2011 at 9:26 AM, Matthew Brett > wrote: >> So far I see the difference between 1) and 2) being that you cannot >> unmask.  So, if you didn't even know you could unmask data, then it >> would not matter that 1) was being im

Re: [Numpy-discussion] Concepts for masked/missing data

Hi, On Sat, Jun 25, 2011 at 6:05 PM, Nathaniel Smith wrote: > Yes, absolutely memory and speed are important. But a really fast > solution to the wrong problem isn't so useful either :-). Would you be happy with me summarizing your idea as 1) = NA logic / API 2) = mask logic / API ? It might

Re: [Numpy-discussion] Concepts for masked/missing data

On Sat, Jun 25, 2011 at 9:26 AM, Matthew Brett wrote: > So far I see the difference between 1) and 2) being that you cannot > unmask.  So, if you didn't even know you could unmask data, then it > would not matter that 1) was being implemented by masks? I guess that is a difference, but I'm trying

Re: [Numpy-discussion] Concepts for masked/missing data

On Sat, Jun 25, 2011 at 10:26 AM, Matthew Brett wrote: > Hi, > > On Sat, Jun 25, 2011 at 5:05 PM, Nathaniel Smith wrote: > > So obviously there's a lot of interest in this question, but I'm > > losing track of all the different issues that've being raised in the > > 150-post thread of doom. I thi

Re: [Numpy-discussion] Concepts for masked/missing data

Hi, On Sat, Jun 25, 2011 at 5:05 PM, Nathaniel Smith wrote: > So obviously there's a lot of interest in this question, but I'm > losing track of all the different issues that've being raised in the > 150-post thread of doom. I think I'll find this easier if we start by > putting aside the questio

Re: [Numpy-discussion] Concepts for masked/missing data

On Sat, Jun 25, 2011 at 10:05 AM, Nathaniel Smith wrote: > So obviously there's a lot of interest in this question, but I'm > losing track of all the different issues that've being raised in the > 150-post thread of doom. I think I'll find this easier if we start by > putting aside the questions

[Numpy-discussion] Concepts for masked/missing data

So obviously there's a lot of interest in this question, but I'm losing track of all the different issues that've being raised in the 150-post thread of doom. I think I'll find this easier if we start by putting aside the questions about implementation and such and focus for now on the *conceptual

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

Hi, On Sat, Jun 25, 2011 at 3:27 PM, Charles R Harris wrote: > > > On Sat, Jun 25, 2011 at 6:00 AM, Gael Varoquaux > wrote: >> >> On Sat, Jun 25, 2011 at 01:02:07AM +0100, Matthew Brett wrote: >> > I'm personally worried that the memory overhead of array.masks will >> > make many of us tend to a

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011/6/25 Charles R Harris > I think what we really need to see are the use cases and work flow. The > ones that hadn't occurred to me before were memory mapped files and data > stored on disk in general. I think we may need some standard format for > masked data on disk if we don't go the NA val

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

Hi, On Sat, Jun 25, 2011 at 4:05 PM, Charles R Harris wrote: > > > On Sat, Jun 25, 2011 at 8:52 AM, Matthew Brett > wrote: >> >> Hi, >> >> On Sat, Jun 25, 2011 at 3:46 PM, Charles R Harris >> wrote: >> > >> > >> > On Sat, Jun 25, 2011 at 8:31 AM, Matthew Brett >> > wrote: >> >> >> >> Hi, >> >>

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

Hi, On Sat, Jun 25, 2011 at 3:44 PM, Wes McKinney wrote: ... > Here are some things I can think of that would be affected by any changes here > > 1) Right now users of pandas can type pandas.isnull(series[5]) and > that will yield True if the value is NA for any dtype. This might be > hard to sup

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 8:52 AM, Matthew Brett wrote: > Hi, > > On Sat, Jun 25, 2011 at 3:46 PM, Charles R Harris > wrote: > > > > > > On Sat, Jun 25, 2011 at 8:31 AM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> On Sat, Jun 25, 2011 at 3:21 PM, Charles R Harris > >> wrote: > >> > > >> > >

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 8:44 AM, Wes McKinney wrote: > On Sat, Jun 25, 2011 at 10:25 AM, Charles R Harris > wrote: > > > > > > On Sat, Jun 25, 2011 at 8:14 AM, Wes McKinney > wrote: > >> > >> On Sat, Jun 25, 2011 at 12:42 AM, Charles R Harris > >> wrote: > >> > > >> > > >> > On Fri, Jun 24, 20

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 08:27:57AM -0600, Charles R Harris wrote: >Could you expand a bit on what sort of data you have and how you deal with >it. Where does it come from, how is it stored on disk, what do you do with >it? That sort of thing. 3D and 4D images. Mostly stored on disk in

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

Hi, On Sat, Jun 25, 2011 at 3:46 PM, Charles R Harris wrote: > > > On Sat, Jun 25, 2011 at 8:31 AM, Matthew Brett > wrote: >> >> Hi, >> >> On Sat, Jun 25, 2011 at 3:21 PM, Charles R Harris >> wrote: >> > >> > >> > On Sat, Jun 25, 2011 at 5:29 AM, Pierre GM wrote: >> >> >> >> This thread is get

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 8:31 AM, Matthew Brett wrote: > Hi, > > On Sat, Jun 25, 2011 at 3:21 PM, Charles R Harris > wrote: > > > > > > On Sat, Jun 25, 2011 at 5:29 AM, Pierre GM wrote: > >> > >> This thread is getting quite long, innit ? > >> And I think it's getting a tad confusing, because we'

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 10:25 AM, Charles R Harris wrote: > > > On Sat, Jun 25, 2011 at 8:14 AM, Wes McKinney wrote: >> >> On Sat, Jun 25, 2011 at 12:42 AM, Charles R Harris >> wrote: >> > >> > >> > On Fri, Jun 24, 2011 at 10:06 PM, Wes McKinney >> > wrote: >> >> >> >> On Fri, Jun 24, 2011 at 1

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

Hi, On Sat, Jun 25, 2011 at 3:14 PM, Wes McKinney wrote: ... > I hope you're right. So far it seems that anyone who has spent real > time with R (e.g. myself, Nathaniel) has expressed serious concerns > about the masked approach. I'm sorry - I have been distracted. For my sake, and because this

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

Hi, On Sat, Jun 25, 2011 at 3:21 PM, Charles R Harris wrote: > > > On Sat, Jun 25, 2011 at 5:29 AM, Pierre GM wrote: >> >> This thread is getting quite long, innit ? >> And I think it's getting a tad confusing, because we're mixing two >> different concepts: missing values and masks. >> There sh

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 6:00 AM, Gael Varoquaux < gael.varoqu...@normalesup.org> wrote: > On Sat, Jun 25, 2011 at 01:02:07AM +0100, Matthew Brett wrote: > > I'm personally worried that the memory overhead of array.masks will > > make many of us tend to avoid them. I work with images that can > >

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 8:14 AM, Wes McKinney wrote: > On Sat, Jun 25, 2011 at 12:42 AM, Charles R Harris > wrote: > > > > > > On Fri, Jun 24, 2011 at 10:06 PM, Wes McKinney > wrote: > >> > >> On Fri, Jun 24, 2011 at 11:59 PM, Nathaniel Smith > wrote: > >> > On Fri, Jun 24, 2011 at 6:57 PM, Be

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 5:29 AM, Pierre GM wrote: > This thread is getting quite long, innit ? > And I think it's getting a tad confusing, because we're mixing two > different concepts: missing values and masks. > There should be support for missing values in numpy.core, I think we all > agree on

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 12:42 AM, Charles R Harris wrote: > > > On Fri, Jun 24, 2011 at 10:06 PM, Wes McKinney wrote: >> >> On Fri, Jun 24, 2011 at 11:59 PM, Nathaniel Smith wrote: >> > On Fri, Jun 24, 2011 at 6:57 PM, Benjamin Root wrote: >> >> On Fri, Jun 24, 2011 at 8:11 PM, Nathaniel Smith

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

On Sat, Jun 25, 2011 at 01:02:07AM +0100, Matthew Brett wrote: > I'm personally worried that the memory overhead of array.masks will > make many of us tend to avoid them. I work with images that can > easily get large enough that I would not want an array-items size byte > array added to my storag

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

This thread is getting quite long, innit ? And I think it's getting a tad confusing, because we're mixing two different concepts: missing values and masks. There should be support for missing values in numpy.core, I think we all agree on that. * What's been suggested of adding new dtypes (naflo

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

Hi, On Sat, Jun 25, 2011 at 2:10 AM, Mark Wiebe wrote: > On Fri, Jun 24, 2011 at 7:02 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Sat, Jun 25, 2011 at 12:22 AM, Wes McKinney >> wrote: >> ... >> > Perhaps we should make a wiki page someplace summarizing pros and cons >> > of the various implem

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

Hi, On Sat, Jun 25, 2011 at 1:54 AM, Mark Wiebe wrote: > On Fri, Jun 24, 2011 at 5:21 PM, Matthew Brett ... >> @Mark - I don't have a clear idea whether you consider the nafloat64 >> option to be still in play as the first thing to be implemented >> (before array.mask).   If it is, what kind of

[Numpy-discussion] [ANN] Euroscipy 2011 - registration now open

Dear all, After some delay due to technical problems, registration for Euroscipy 2011 is now open! Please go to http://www.euroscipy.org/conference/euroscipy2011, login to your account if you have one, or create a new account (right side of the upper banner of the Euroscipy webpage), then