Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

Hi, instead of putting up a pull request that reverts all the 25000 lines of code than have been written to support an NA mask, why won't you set up a pull request that uses the current code base to implement your own ideas on how it should work? ___ NumP

Re: [Numpy-discussion] consensus

On 10/28/11 10:38 PM, Benjamin Root wrote: > I might like the idea of a NA to "always" mean absent (and destroys - > even through views), and MA (or some other name) which always means > ignore (and has the masking behavior with views). I should point out that if I'm dictating code to someone (e.g

[Numpy-discussion] consensus (was: NA masks in the next numpy release?)

Matt, On Friday, October 28, 2011, Matthew Brett wrote: > >> Forget about rudeness or decision processes. > > No, that's a common mistake, which is to assume that any conversation > about things which aren't technical, is not important. Nathaniel's > point is important. Rudeness is important.

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris wrote: > Matthew, the problem I have is that it seems that you and Nathaniel won't be > satisfied unless things are done *your* way. Hi Charles, I'm sorry if I've given this impression, and I know it's easy to feel this way in a contentious discu

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

On Fri, Oct 28, 2011 at 6:45 PM, Wes McKinney wrote: > On Fri, Oct 28, 2011 at 7:53 PM, Benjamin Root wrote: > > > > > > On Friday, October 28, 2011, Matthew Brett > wrote: > >> Hi, > >> > >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers > >> wrote: > >>> > >>> > >>> On Sat, Oct 29, 2011 at 12

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

Hi, On Fri, Oct 28, 2011 at 4:53 PM, Benjamin Root wrote: > > > On Friday, October 28, 2011, Matthew Brett wrote: >> Hi, >> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers >> wrote: >>> >>> >>> On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett >>> wrote: Hi, On Fri, Oct 28,

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

On Fri, Oct 28, 2011 at 7:53 PM, Benjamin Root wrote: > > > On Friday, October 28, 2011, Matthew Brett wrote: >> Hi, >> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers >> wrote: >>> >>> >>> On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett >>> wrote: Hi, On Fri, Oct 28, 2011

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

On Friday, October 28, 2011, Matthew Brett wrote: > Hi, > > On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers > wrote: >> >> >> On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett >> wrote: >>> >>> Hi, >>> >>> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris >>> wrote: >>> > >>> > >>> > On Fri, Oct 28

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

Hi, On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers wrote: > > > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett > wrote: >> >> Hi, >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris >> wrote: >> > >> > >> > On Fri, Oct 28, 2011 at 3:56 PM, Matthew Brett >> > wrote: >> >> >> >> Hi, >> >> >>

Re: [Numpy-discussion] NA masks in the next numpy release?

2011/10/28 Stéfan van der Walt > On Fri, Oct 28, 2011 at 4:19 PM, Charles R Harris > wrote: > > Memory use is a known problem. One way to start addressing it might be to > > implement a "bit" arraytype. It might even be possible to prototype that > on > > top of the existing types. Views make bi

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

Hi, On Fri, Oct 28, 2011 at 4:21 PM, Charles R Harris wrote: > > > On Fri, Oct 28, 2011 at 5:09 PM, Matthew Brett > wrote: >> >> On Fri, Oct 28, 2011 at 3:49 PM, Charles R Harris >> wrote: >> > >> > >> > 2011/10/28 Stéfan van der Walt >> >> >> >> On Fri, Oct 28, 2011 at 3:21 PM, Benjamin Root

Re: [Numpy-discussion] NA masks in the next numpy release?

On Fri, Oct 28, 2011 at 4:19 PM, Charles R Harris wrote: > Memory use is a known problem. One way to start addressing it might be to > implement a "bit" arraytype. It might even be possible to prototype that on > top of the existing types. Views make bit arrays a bit more interesting ;) Since 1/8

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett wrote: > Hi, > > On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris > wrote: > > > > > > On Fri, Oct 28, 2011 at 3:56 PM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett > > >> wrote: > >> > Hi, > >>

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

On Fri, Oct 28, 2011 at 5:09 PM, Matthew Brett wrote: > On Fri, Oct 28, 2011 at 3:49 PM, Charles R Harris > wrote: > > > > > > 2011/10/28 Stéfan van der Walt > >> > >> On Fri, Oct 28, 2011 at 3:21 PM, Benjamin Root wrote: > >> > The space issues was never ignored and Mark left room for that to

Re: [Numpy-discussion] NA masks in the next numpy release?

On Fri, Oct 28, 2011 at 5:05 PM, Chris.Barker wrote: > On 10/28/11 11:37 AM, Matthew Brett wrote: > > The main motivation for the alterNEP was our strong feeling that > > separating ABSENT and IGNORE was easier to comprehend and cleaner. > > I don't know about easier to comprehend, or cleaner, bu

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

On Fri, Oct 28, 2011 at 3:49 PM, Charles R Harris wrote: > > > 2011/10/28 Stéfan van der Walt >> >> On Fri, Oct 28, 2011 at 3:21 PM, Benjamin Root wrote: >> > The space issues was never ignored and Mark left room for that to be >> > addressed.  Parameterized dtypes can still be added (and isn't

Re: [Numpy-discussion] NA masks in the next numpy release?

On 10/28/11 11:37 AM, Matthew Brett wrote: > The main motivation for the alterNEP was our strong feeling that > separating ABSENT and IGNORE was easier to comprehend and cleaner. I don't know about easier to comprehend, or cleaner, but it is more feature-full. I see two issues here: 1) being ab

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011/10/28 Stéfan van der Walt > On Fri, Oct 28, 2011 at 3:21 PM, Benjamin Root wrote: > > The space issues was never ignored and Mark left room for that to be > > addressed. Parameterized dtypes can still be added (and isn't all that > > different from multi-na). Perhaps I could be convinced o

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

On Fri, Oct 28, 2011 at 3:21 PM, Benjamin Root wrote: > The space issues was never ignored and Mark left room for that to be > addressed.  Parameterized dtypes can still be added (and isn't all that > different from multi-na). Perhaps I could be convinced of a having np.MA > assignments mean "igno

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

Hi, On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris wrote: > > > On Fri, Oct 28, 2011 at 3:56 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett >> wrote: >> > Hi, >> > >> > On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris >> > wrote: >> >> >> >> >> >>

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

On Friday, October 28, 2011, Matthew Brett wrote: > Hi, > > On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett wrote: >> Hi, >> >> On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris >> wrote: >>> >>> >>> On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith wrote: On Tue, Oct 25, 2011 at 2:56 P

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

On Fri, Oct 28, 2011 at 3:56 PM, Matthew Brett wrote: > Hi, > > On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett > wrote: > > Hi, > > > > On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris > > wrote: > >> > >> > >> On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith wrote: > >>> > >>> On Tue, Oct 25,

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

Hi, On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett wrote: > Hi, > > On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris > wrote: >> >> >> On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith wrote: >>> >>> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant >>> wrote: >>> > I think Nathaniel and Matthe

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

Hi, On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris wrote: > > > On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith wrote: >> >> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant >> wrote: >> > I think Nathaniel and Matthew provided very >> > specific feedback that was helpful in understanding o

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith wrote: > On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant > wrote: > > I think Nathaniel and Matthew provided very > > specific feedback that was helpful in understanding other perspectives of > a > > difficult problem. In particular, I really

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

On Fri, Oct 28, 2011 at 2:32 PM, Matthew Brett wrote: > Hi, > > On Fri, Oct 28, 2011 at 2:16 PM, Nathaniel Smith wrote: >> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant >> wrote: >>> I think Nathaniel and Matthew provided very >>> specific feedback that was helpful in understanding other per

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

Hi, On Fri, Oct 28, 2011 at 2:16 PM, Nathaniel Smith wrote: > On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant > wrote: >> I think Nathaniel and Matthew provided very >> specific feedback that was helpful in understanding other perspectives of a >> difficult problem.     In particular, I really

[Numpy-discussion] consensus (was: NA masks in the next numpy release?)

On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant wrote: > I think Nathaniel and Matthew provided very > specific feedback that was helpful in understanding other perspectives of a > difficult problem.     In particular, I really wanted bit-patterns > implemented.    However, I also understand that

Re: [Numpy-discussion] NA masks in the next numpy release?

Hi, On Fri, Oct 28, 2011 at 1:52 PM, Benjamin Root wrote: > > > On Fri, Oct 28, 2011 at 3:22 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Fri, Oct 28, 2011 at 1:14 PM, Benjamin Root wrote: >> > >> > >> > On Fri, Oct 28, 2011 at 3:02 PM, Matthew Brett >> > wrote: >> >> >> >> You and I know tha

Re: [Numpy-discussion] NA masks in the next numpy release?

On Fri, Oct 28, 2011 at 3:22 PM, Matthew Brett wrote: > Hi, > > On Fri, Oct 28, 2011 at 1:14 PM, Benjamin Root wrote: > > > > > > On Fri, Oct 28, 2011 at 3:02 PM, Matthew Brett > > wrote: > >> > >> You and I know that I've got an array with values [99, 100, 3] and a > >> mask with values [False,

Re: [Numpy-discussion] NA masks in the next numpy release?

2011/10/28 Stéfan van der Walt > On Fri, Oct 28, 2011 at 12:47 PM, Benjamin Root wrote: > > > > 2011/10/28 Stéfan van der Walt > >> The > >> implementation as it stands essentially gives us a faster and more > >> integrated version of numpy.ma; but it has become clear from this > >> conversatio

Re: [Numpy-discussion] NA masks in the next numpy release?

Hi, On Fri, Oct 28, 2011 at 1:14 PM, Benjamin Root wrote: > > > On Fri, Oct 28, 2011 at 3:02 PM, Matthew Brett > wrote: >> >> You and I know that I've got an array with values [99, 100, 3] and a >> mask with values [False, False, True].  So maybe I'd like to see what >> happens if I take off the

Re: [Numpy-discussion] NA masks in the next numpy release?

On Fri, Oct 28, 2011 at 1:14 PM, Benjamin Root wrote: > Considering that you have admitted before to not regularly using masked > arrays, I seriously doubt that you would be able to judge whether this is a > significant detriment or not. Let's not be unreasonable; Matthew has a valid concern (may

Re: [Numpy-discussion] NA masks in the next numpy release?

On Fri, Oct 28, 2011 at 3:02 PM, Matthew Brett wrote: > > You and I know that I've got an array with values [99, 100, 3] and a > mask with values [False, False, True]. So maybe I'd like to see what > happens if I take off the mask from the second value. I know that's > what I want to do, but I

Re: [Numpy-discussion] NA masks in the next numpy release?

On Fri, Oct 28, 2011 at 12:47 PM, Benjamin Root wrote: > > 2011/10/28 Stéfan van der Walt >> The >> implementation as it stands essentially gives us a faster and more >> integrated version of numpy.ma; but it has become clear from this >> conversation that such an approach overlooks a very common

Re: [Numpy-discussion] skip lines at the end of file with loadtxt

On Tue, Oct 25, 2011 at 12:28 PM, Massimo Di Stefano wrote: > urllib.urlretrieve('http://www.cdc.noaa.gov/Correlation/amon.us.long.data', > 'file.txt') a = np.loadtxt('file.txt', skiprows=1) > > but it fails becouse of the txt description at the end of the file, It's always hard to stop reading b

Re: [Numpy-discussion] NA masks in the next numpy release?

Hi, On Fri, Oct 28, 2011 at 12:15 PM, Lluís wrote: > Summarizing: let's forget for a moment that "mask" has a meaning in english: This is at the core of the problem. You and I know what's really going on - there's a mask over the data. But in what follows we're going to try and pretend that

Re: [Numpy-discussion] NA masks in the next numpy release?

2011/10/28 Stéfan van der Walt > On Fri, Oct 28, 2011 at 11:16 AM, Benjamin Root wrote: > > this by making missing data front-and-center. However, my belief is that > > Mark's approach is easier to comprehend and is cleaner. Cleaner features > > means that it is more likely to be used. > > Cle

Re: [Numpy-discussion] NA masks in the next numpy release?

On Fri, Oct 28, 2011 at 11:16 AM, Benjamin Root wrote: > this by making missing data front-and-center.  However, my belief is that > Mark's approach is easier to comprehend and is cleaner.  Cleaner features > means that it is more likely to be used. Cleaner features may be easier to adopt, but wh

Re: [Numpy-discussion] NA masks in the next numpy release?

I haven't actually tested the code, but AFAIK the following is a short overview with examples of how the two orthogonal feature axis (ABSENT/IGNORE and PROPAGATE/SKIP) are related and how it all is supposed to work. I have never talked to Mark or anybody else in this list (that is, outside of this

Re: [Numpy-discussion] NA masks in the next numpy release?

Hi, On Fri, Oct 28, 2011 at 11:16 AM, Benjamin Root wrote: > On Fri, Oct 28, 2011 at 12:39 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Thu, Oct 27, 2011 at 10:56 PM, Benjamin Root wrote: >> > >> > >> > On Thursday, October 27, 2011, Charles R Harris >> > >> > wrote: >> >> >> >> >> >> On Thu,

Re: [Numpy-discussion] NA masks in the next numpy release?

On Fri, Oct 28, 2011 at 12:58 PM, Gary Strangman wrote: > > >> I wonder if that might be handled as a scikits-image extension, rather > >> than core numpy? > > > > I think Stefan and Nathaniel and Gary Strangman and others are saying > > we don't want to pay the price of a large memory hike for m

Re: [Numpy-discussion] NA masks in the next numpy release?

On Fri, Oct 28, 2011 at 12:39 PM, Matthew Brett wrote: > Hi, > > On Thu, Oct 27, 2011 at 10:56 PM, Benjamin Root wrote: > > > > > > On Thursday, October 27, 2011, Charles R Harris < > charlesr.har...@gmail.com> > > wrote: > >> > >> > >> On Thu, Oct 27, 2011 at 7:16 PM, Travis Oliphant < > oliph..

Re: [Numpy-discussion] NA masks in the next numpy release?

>> I wonder if that might be handled as a scikits-image extension, rather >> than core numpy? > > I think Stefan and Nathaniel and Gary Strangman and others are saying > we don't want to pay the price of a large memory hike for masking. I > suspect that Nathaniel is right, and that a large major

Re: [Numpy-discussion] NA masks in the next numpy release?

Hi, On Fri, Oct 28, 2011 at 9:21 AM, Chris.Barker wrote: > On 10/27/11 7:51 PM, Travis Oliphant wrote: >> As I mentioned. I find the ability to separate an ABSENT idea from an >> IGNORED idea convincing. In other words, I think distinguishing between >> masks and bit-patterns is not just an imple

[Numpy-discussion] [ANN] SciPy India 2011 Abstracts due November 2nd

== SciPy 2011 Call for Papers == The third `SciPy India Conference `_ will be held from December 4th through the 7th at the `Indian Institute of Technology, Bombay (IITB) `_ in Mumbai, Maharashtra India. At t

Re: [Numpy-discussion] NA masks in the next numpy release?

Hi, On Thu, Oct 27, 2011 at 10:56 PM, Benjamin Root wrote: > > > On Thursday, October 27, 2011, Charles R Harris > wrote: >> >> >> On Thu, Oct 27, 2011 at 7:16 PM, Travis Oliphant >> wrote: >>> >>> That is a pretty good explanation.   I find myself convinced by Matthew's >>> arguments.    I thi

Re: [Numpy-discussion] NA masks in the next numpy release?

On 10/27/11 7:51 PM, Travis Oliphant wrote: > As I mentioned. I find the ability to separate an ABSENT idea from an > IGNORED idea convincing. In other words, I think distinguishing between > masks and bit-patterns is not just an implementation detail, but > provides a useful concept for multiple u

Re: [Numpy-discussion] NA masks in the next numpy release?

On Thu, Oct 27, 2011 at 8:51 PM, Travis Oliphant wrote: > As I mentioned. I find the ability to separate an ABSENT idea from an > IGNORED idea convincing.In other words, I think distinguishing between > masks and bit-patterns is not just an implementation detail, but provides a > useful conc

Re: [Numpy-discussion] NA masks in the next numpy release?

On Fri, Oct 28, 2011 at 10:45:09AM +0200, Han Genuit wrote: > Also, I like the short and concise abbreviation for 'Not Applicable', > NA. It has more common uses than IGNORE. > (See also here: > http://www.johndcook.com/R_language_for_programmers.html#missing) That's a very R centric point a view:

Re: [Numpy-discussion] NA masks in the next numpy release?

Hi all, On Thu, Oct 27, 2011 at 7:51 PM, Travis Oliphant wrote: > I understand exactly what it would take to add bit-patterns to NumPy.  I > also understand what Mark did and agree that it is possible to add Matthew's > idea to the current code-base.  I think it is worth exploring Another data p

Re: [Numpy-discussion] NA masks in the next numpy release?

Yes, to further iterate on that, you can also create multiple masked views with each its own mask properties. It would be ambiguous to mix a bit-pattern NA together with standard NA's in the same mask, but you can make different specialized masked views on the same data. Also, I like the short and