[Numpy-discussion] New version of "scipy lecture notes"

2015-09-28 Thread Gael Varoquaux
Dear Pythonistas, We have just released a new version of the "scipy lecture notes": http://www.scipy-lectures.org/ These are a consistent set of materials to learn the core aspects of the scientific Python ecosystem, from beginner to expert. They are written and maintained by a set of volunteers

Re: [Numpy-discussion] Proposal of new function: iteraxis()

2013-04-26 Thread Gael Varoquaux
On Thu, Apr 25, 2013 at 08:10:32PM +0100, Robert Kern wrote: > In my opinion, duplicating functionality under different aliases just > so people can supposedly find things without reading the documentation > is not a viable strategy for building out an API. +1. It's been my experience over and ove

Re: [Numpy-discussion] 1.7.0 release

2012-11-13 Thread Gael Varoquaux
On Mon, Nov 12, 2012 at 02:27:02PM -0800, Ondřej Čertík wrote: > I successfully defended my Ph.D. thesis last Thursday, I just need to > do some changes to it and submit it and I am done. Yey! Tag and release (the thesis, I mean)! Congratulation. G PS: It seems to me that it was yesterday that

Re: [Numpy-discussion] Scipy dot

2012-11-09 Thread Gael Varoquaux
On Fri, Nov 09, 2012 at 03:12:42PM +, Nathaniel Smith wrote: > But what if someone compiles numpy against an optimized blas (mkl, > say) and then compiles SciPy against the reference blas? What do you > do then!? ;-) This could happen. But the converse happens very often. What happens is that

Re: [Numpy-discussion] Scipy dot

2012-11-09 Thread Gael Varoquaux
On Thu, Nov 08, 2012 at 11:29:19AM -0600, Anthony Scopatz wrote: > Indeed, there is no reason not to make this available in NumPy. > +1, I agree, this should be a fix in numpy, not scipy. I agree. My point was a bit of a side issue: given a user's computer, I have no garantee that numpy will

Re: [Numpy-discussion] Scipy dot

2012-11-08 Thread Gael Varoquaux
On Thu, Nov 08, 2012 at 11:28:21AM +, Nathaniel Smith wrote: > I think everyone would be very happy to see numpy.dot modified to do > this automatically. But adding a scipy.dot IMHO would be fixing things > in the wrong place and just create extra confusion. I am not sure I agree: numpy is oft

Re: [Numpy-discussion] A change with minor compatibility questions

2012-10-17 Thread Gael Varoquaux
On Wed, Oct 17, 2012 at 10:22:28AM -0500, Travis Oliphant wrote: > There is a small chance this will break someone's code if they relied on this > behavior. I don't believe anyone is currently relying on this behavior -- but > I've been proven wrong before. What do people on this list think?

Re: [Numpy-discussion] Behavior of .base

2012-09-30 Thread Gael Varoquaux
On Sun, Sep 30, 2012 at 10:30:52PM +0200, Han Genuit wrote: > Also, considering that this behaviour already exists in past versions > of NumPy, namely 1.6, I just checked: in numpy 1.6.1, the behaviour is to create an endless chain of base.base.base... In some sens, what Travis is proposing is go

Re: [Numpy-discussion] Memory order of array copies

2012-09-30 Thread Gael Varoquaux
On Sun, Sep 30, 2012 at 07:17:42PM +0100, Nathaniel Smith wrote: > Is there anything better to do than simply revert np.copy() to its > traditional behaviour and accept that np.copy(a) and a.copy() will > continue to have different semantics indefinitely? Have np.copy take an 'order=None', which w

Re: [Numpy-discussion] Making numpy sensible: backward compatibility please

2012-09-30 Thread Gael Varoquaux
On Sat, Sep 29, 2012 at 03:09:08PM -0700, Ondřej Čertík wrote: > Chuck, Gael, here is my todo list for the 1.7.0 release: > https://github.com/numpy/numpy/issues/396 I have created issues and mentionned them in the comments on your issue. Cheers, Gaël ___

Re: [Numpy-discussion] Making numpy sensible: backward compatibility please

2012-09-29 Thread Gael Varoquaux
Hi Nathaniel, First of all, thanks for your investment in numpy. You have really been moving the project forward lately. On Sat, Sep 29, 2012 at 03:03:01AM +0100, Nathaniel Smith wrote: > > I have just fixed a fairly nasty bug in scikit-learn that was > > introduced by change of semantics in orde

Re: [Numpy-discussion] Making numpy sensible: backward compatibility please

2012-09-29 Thread Gael Varoquaux
On Fri, Sep 28, 2012 at 07:37:35PM -0600, Charles R Harris wrote: > +1, I think we should endeavor to have a respectful and welcoming > community. > With a bit of humour now and then among the old timers, no? Look, I've been > outright insulted on the list and I don't recall anyone weighin

[Numpy-discussion] Making numpy sensible: backward compatibility please

2012-09-28 Thread Gael Varoquaux
Hi numpy developers, First of all, thanks a lot for the hard work you put in numpy. I know very well that maintaining such a core library is a lot of effort and a service to the community. But "with great dedication, comes great responsibility" :). I find that Numpy is a bit of a wild horse, a mo

Re: [Numpy-discussion] Memory leak with numpy master

2012-09-24 Thread Gael Varoquaux
On Mon, Sep 24, 2012 at 07:59:11PM +0100, Nathaniel Smith wrote: > > which means I probably forgot a DECREF while doing the > > PyArray_Diagonal changes... > Yep: https://github.com/numpy/numpy/pull/457 Awesome. I can confirm that this fixes the problem. Script below to check. You are my hero!

Re: [Numpy-discussion] Memory leak with numpy master

2012-09-24 Thread Gael Varoquaux
Hi Fred, On Mon, Sep 24, 2012 at 02:17:16PM -0400, Frédéric Bastien wrote: > with numpy '1.6.1', I have no problem. > With numpy 1.7.0b2, I can reproduce the problem. OK, thanks. I think that I'll start a bisect to figure out when it crept in. Gael __

[Numpy-discussion] Memory leak with numpy master

2012-09-24 Thread Gael Varoquaux
Hi list, I think that I am hit a memory leak with numpy master. The following code enables to reproduce it: import numpy as np n = 100 m = np.eye(n) for i in range(3): #np.linalg.slogdet(m) t, result_t =

Re: [Numpy-discussion] Views of memmaps and offset

2012-09-23 Thread Gael Varoquaux
On Sat, Sep 22, 2012 at 12:19:53PM -0600, Charles R Harris wrote: > But you loose the pointer to the filename and the offset. In previous > versions of numpy c.base used to be the np.memmap instance from which > c is an array view. That allowed to make efficient pickling without >

Re: [Numpy-discussion] Views of memmaps and offset

2012-09-23 Thread Gael Varoquaux
On Sat, Sep 22, 2012 at 12:30:46PM -0600, Charles R Harris wrote: > As to the 1.7 release, I've been thinking we are violating the release > early, release often maxim. Bugs trickle in at a constant rate and if we > wait to fix them all we wait forever. So while it would be nice to h

Re: [Numpy-discussion] Views of memmaps and offset

2012-09-22 Thread Gael Varoquaux
On Sat, Sep 22, 2012 at 11:16:27AM -0600, Charles R Harris wrote: >I think this is a bug, taking a view should probably update the offset. OK, we can include a fix for that alongside with the patch to keep track of the filename. Cheers, Gaël ___ Nu

Re: [Numpy-discussion] Views of memmaps and offset

2012-09-22 Thread Gael Varoquaux
On Sat, Sep 22, 2012 at 06:30:27PM +0200, Olivier Grisel wrote: > The only clean solution would be to make `numpy.memmap` use a wrapper > buffer object that would keep track of the filename and offset > attributes instead of using a `mmap.mmap` instance directly. Indeed, Olivier and I have been st

Re: [Numpy-discussion] Views of memmaps and offset

2012-09-22 Thread Gael Varoquaux
On Sat, Sep 22, 2012 at 10:15:52AM -0600, Charles R Harris wrote: >Would some sort of 'dup' method be useful? What do you mean by dup? G ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discus

[Numpy-discussion] Views of memmaps and offset

2012-09-22 Thread Gael Varoquaux
Hi list, I am struggling with offsets on the view of a memmaped array. Consider the following: import numpy as np a = np.memmap('tmp.mmap', dtype=np.float64, shape=50, mode='w+') a[:] = np.arange(50) b = a[10:] Here, I have a.offset == 0 and b.offset == 0. In practice, the data in b is offset c

Re: [Numpy-discussion] Licensing question

2012-08-02 Thread Gael Varoquaux
On Thu, Aug 02, 2012 at 09:44:53PM +0100, Damon McDougall wrote: > I have a question about the licence for NumPy's codebase. I am currently > writing a library and I'd like to release under some BSD-type licence. > Unfortunately, my choice to link against MIT's FFTW library (released > under the GP

Re: [Numpy-discussion] Lazy imports again

2012-07-16 Thread Gael Varoquaux
On Mon, Jul 16, 2012 at 10:28:19AM -0600, Charles R Harris wrote: >Working lazy imports would be useful to have. Ralf is opposed to the idea >because it caused all sorts of problems on different platforms when it was >tried in scipy. I thought I'd open the topic for discussion so that f

Re: [Numpy-discussion] Good way to develop numpy as popular choice!

2012-06-21 Thread Gael Varoquaux
On Thu, Jun 21, 2012 at 08:59:09PM -0400, Benjamin Root wrote: > > munkres seems to be a pure python implementation ;-). > Oops! I could have sworn that I once tried one named munkres that used > numpy. But that was several years ago. >There is a development branch of sk-learn

Re: [Numpy-discussion] NA-mask interactions with existing C code

2012-05-11 Thread Gael Varoquaux
On Fri, May 11, 2012 at 09:59:16AM -0500, Travis Oliphant wrote: > > Is there a advantage for users by making it a subclass? Nobody is saying > > you couldn't 'inherit' the struct (make the ndmask struct be castable to > > a PyArrayObject*) even if that is not declared in the Python type object.

Re: [Numpy-discussion] Missing data wrap-up and request for comments

2012-05-10 Thread Gael Varoquaux
On Wed, May 09, 2012 at 02:35:26PM -0500, Travis Oliphant wrote: >  Basically it buys not forcing *all* NumPy users (on the C-API level) to >now deal with a masked array.    I know this push is a feature that is >part of Mark's intention (as it pushes downstream libraries to think about

Re: [Numpy-discussion] [SciPy-Dev] Announce: scikit-learn v0.11

2012-05-09 Thread Gael Varoquaux
Thank you very much, it is my pleasure. But it's really a team that you need to thank: the number of active contributors is huge. Cheers, Gael > On Tue, May 8, 2012 at 1:13 AM, Gael Varoquaux > wrote: > >   On behalf of Andy Mueller, our release manager, I am happy to announce

[Numpy-discussion] [SciPy-Dev] Announce: scikit-learn v0.11

2012-05-07 Thread Gael Varoquaux
On behalf of Andy Mueller, our release manager, I am happy to announce the 0.11 release of scikit-learn. This release includes some major new features such as randomized sparse models, gradient boosted regression trees, label propagation and many more. The release also has major imp

[Numpy-discussion] Announce: scikit-learn v0.11

2012-05-07 Thread Gael Varoquaux
On behalf of Andy Mueller, our release manager, I am happy to announce the 0.11 release of scikit-learn. This release includes some major new features such as randomized sparse models, gradient boosted regression trees, label propagation and many more. The release also has major imp

[Numpy-discussion] ANN Scikit-learn 0.11-beta

2012-05-06 Thread Gael Varoquaux
On behalf of our release manager, Andreas Mueller, and all the scikit-learn contributors, I am happy to announce the 0.11 beta. We are doing a quick beta and will hopefuly be releasing the final version tomorrow. The purpose of this beta is to get feedback on any release-critical bugs such as buil

Re: [Numpy-discussion] What is consensus anyway

2012-04-25 Thread Gael Varoquaux
On Wed, Apr 25, 2012 at 06:03:25AM -0600, Charles R Harris wrote: > Well, you have already appealed to the authority of greater experience, so > it's a bit late to declare disinterest in the subject ;) I mean, at this > point I really would like to see how big your FOSS is. Chuck, I am not sure th

Re: [Numpy-discussion] What is consensus anyway

2012-04-24 Thread Gael Varoquaux
On Tue, Apr 24, 2012 at 05:59:09PM -0600, Charles R Harris wrote: > Travis, if you are playing the BDFL role, then just make the darn decision > and remove the code so we can get on with life. As it is you go back and > forth and that does none of us any good, you're a big guy and you're > rocking

Re: [Numpy-discussion] Removing masked arrays for 1.7? (Was 1.7 blockers)

2012-04-16 Thread Gael Varoquaux
On Mon, Apr 16, 2012 at 10:40:53PM -0500, Travis Oliphant wrote: > > The objectors object to any binary ABI change, but not specifically > > three pointers rather than two or one? > Adding pointers is not really an ABI change (but removing them after > they were there would be...) It's really jus

Re: [Numpy-discussion] MemoryError : with scipy.spatial.distance

2012-04-05 Thread Gael Varoquaux
On Thu, Apr 05, 2012 at 01:05:01PM -0700, Abhishek Pratap wrote: > Also in my case I dont really have a good approximate on value of K in > K-means. That's a hard problem, for which I have no answer, sorry :$ G ___ NumPy-Discussion mailing list NumPy-D

Re: [Numpy-discussion] MemoryError : with scipy.spatial.distance

2012-04-04 Thread Gael Varoquaux
On Wed, Apr 04, 2012 at 04:41:51PM -0700, Abhishek Pratap wrote: > Thanks Chris. So I guess the question becomes how can I efficiently > cluster 1 million x,y coordinates. Did you try the scikit-learn's implementation of DBSCAN: http://scikit-learn.org/stable/modules/clustering.html#dbscan ? I am

Re: [Numpy-discussion] creating/working NumPy-ndarrays in C++

2012-04-04 Thread Gael Varoquaux
On Wed, Apr 04, 2012 at 11:34:13AM -0700, Dag Sverre Seljebotn wrote: > On 04/03/2012 04:45 PM, srean wrote: > > This makes me ask something that I always wanted to know: why is weave > > not the preferred or encouraged way ? > > Is it because no developer has interest in maintaining it or is it t

Re: [Numpy-discussion] Crashes and test failures

2012-03-22 Thread Gael Varoquaux
On Thu, Mar 22, 2012 at 06:24:13PM -0600, Charles R Harris wrote: >Since you are running on Ubuntu, it's probably a good idea to >python setup.py install --user >which will put everything in .local and avoid the mess with dist-packages >and site-packages. Maybe you do that already

Re: [Numpy-discussion] Crashes and test failures

2012-03-22 Thread Gael Varoquaux
On Thu, Mar 22, 2012 at 11:20:10AM -0600, Charles R Harris wrote: > This one was fixed a few days ago, so you aren't running the latest. Indeed, but if I run the latest, any operation on arrays segfaults on me, so I am running 0.6.1. But you are saying that it's known and fixed. Great! > That mig

[Numpy-discussion] Crashes and test failures

2012-03-22 Thread Gael Varoquaux
Hi list, I just got a new laptop, running Ubuntu 11.10 64bit on an Intel i7. I am a bit intriged by the test results of numpy on this box. First of all, master builds and imports OK, but the simplest test case crashes with a segfault: import numpy as np a = np.ones(10, dtype=np.bool)

Re: [Numpy-discussion] C++ Example

2012-03-04 Thread Gael Varoquaux
On Sat, Mar 03, 2012 at 04:38:53PM -0800, David Cournapeau wrote: > This is really the kind of code that should be done in cython, as it is > mostly about wrapping C code into the python C API. +1 Gael ___ NumPy-Discussion mailing list NumPy-Discussion@

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Gael Varoquaux
On Thu, Feb 23, 2012 at 04:07:04PM -0500, Wes McKinney wrote: > In this last case for example, around 500 MB of RAM is taken up for an > array that should only be about 80-90MB. If you're a data scientist > working in Python, this is _not good_. But why, oh why, are people storing big data in CSV?

Re: [Numpy-discussion] Proposed Roadmap Overview

2012-02-21 Thread Gael Varoquaux
On Sun, Feb 19, 2012 at 05:44:27AM -0500, David Warde-Farley wrote: > I think the comments about the developer audience NumPy will attract are > important. There may be lots of C++ developers out there, but the > intersection of (truly competent in C++) and (likely to involve oneself in > NumPy

Re: [Numpy-discussion] strange behavior of numpy.random.multivariate_normal, ticket:1842

2012-02-16 Thread Gael Varoquaux
On Thu, Feb 16, 2012 at 05:00:29PM +, Nathaniel Smith wrote: > I agree, but the behavior is still surprising -- people reasonably > expect something like svd to be deterministic. People are wrong then. Trust me, I work enough with ill-conditionned problems, including SVDs, to know that the alg

Re: [Numpy-discussion] Release management (was: Updated differences between 1.5.1 to 1.6.1)

2012-02-15 Thread Gael Varoquaux
On Tue, Feb 14, 2012 at 09:14:11PM -0600, Bruce Southey wrote: > Ralf, > I will miss you as a numpy release manager! > You have not only done an incredible job but also taken the role to a > higher level. > Your attitude and attention to details has been amazing. I definitely +1 that. I think that

Re: [Numpy-discussion] what is the best way to pass c, c++ array to numpy in cython?

2012-01-05 Thread Gael Varoquaux
On Thu, Jan 05, 2012 at 04:04:52PM +0900, 최원준 wrote: >can I pass the array without malloc? An array is a pointer in C, so yes you can do what you want. G ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/list

Re: [Numpy-discussion] what is the best way to pass c, c++ array to numpy in cython?

2012-01-04 Thread Gael Varoquaux
On Thu, Jan 05, 2012 at 11:26:05AM +0900, 최원준 wrote: >it seems like you recommend below way. >Cython example of exposing C-computed arrays in Python without data copies >[1]https://gist.github.com/1249305 >but it uses malloc. isn't it? In this example, the data can be allocated th

Re: [Numpy-discussion] what is the best way to pass c, c++ array to numpy in cython?

2012-01-03 Thread Gael Varoquaux
On Tue, Jan 03, 2012 at 07:56:52PM -0800, Wonjun, Choi wrote: > what is the best way to pass c, c++ array to numpy in cython? I don't know if it is the best way, but I wrote a self-contained example a little while ago, to explain to people one way of doing it: http://gael-varoquaux.info/blog/?p=15

Re: [Numpy-discussion] Functions vs Methods

2011-12-27 Thread Gael Varoquaux
Hi Jaidev, On Wed, Dec 28, 2011 at 12:09:45PM +0530, Jaidev Deshpande wrote: > Eg. x is a NumPy array, and I need its transpose > Should I use > >>> x.T > or > >>> numpy.transpose(T) ? If you are wondering for a timing reason, use IPython's '%timeit' magic to figure out if it does make a di

Re: [Numpy-discussion] Owndata flag

2011-12-23 Thread Gael Varoquaux
On Thu, Dec 15, 2011 at 04:36:24PM +, Robert Kern wrote: > > More explicitly, I have some temporary home-made C structure that holds > > a pointer to an array. I prepare (using Cython) an numpy.ndarray using > > the PyArray_NewFromDescr function. I can delete my temporary C structure > > withou

Re: [Numpy-discussion] Memory hungry reduce ops in Numpy

2011-11-15 Thread Gael Varoquaux
On Tue, Nov 15, 2011 at 05:57:14PM +, Robert Kern wrote: > Actually, last time I suggested it, it was brought up that the online > algorithms can be worse numerically. I'll try to find the thread. Indeed, especially for smallish datasets where the memory overhead is not an issue. I think that

Re: [Numpy-discussion] Memory hungry reduce ops in Numpy

2011-11-15 Thread Gael Varoquaux
On Tue, Nov 15, 2011 at 05:46:52PM +0100, Andreas Müller wrote: >My question was more along the lines of "why doesn't numpy do the online >algorithm". It's probably a matter of nobody having had the time and the urge to code it, _and_ do all the extra steps necessary to integrate to the co

Re: [Numpy-discussion] ANN: SciPy 0.10.0 released

2011-11-13 Thread Gael Varoquaux
On Sun, Nov 13, 2011 at 12:32:10PM -0700, Charles R Harris wrote: > Congratulations to all. This looks like a nice release. And congratulations to Ralf for moving this forward! G ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.s

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Gael Varoquaux
On Fri, Oct 28, 2011 at 10:45:09AM +0200, Han Genuit wrote: > Also, I like the short and concise abbreviation for 'Not Applicable', > NA. It has more common uses than IGNORE. > (See also here: > http://www.johndcook.com/R_language_for_programmers.html#missing) That's a very R centric point a view:

[Numpy-discussion] [ANN] ESCO2012 and PyHPC2011: Python in science in major computational science conference

2011-09-11 Thread Gael Varoquaux
ESCO 2012 - European Seminar on Coupled Problems = ESCO2012 http://esco2012.femhub.com/ is the 3rd event in a series of interdisciplineary meetings dedicated to computational science challenges in multi-physics and PDEs. I was invited as ESCO last y

[Numpy-discussion] Hiring a junior developer on the scikit-learn

2011-09-02 Thread Gael Varoquaux
Hi list, Once again, we are looking for a junior developer to work on the scikit-learn. Below is the official job posting. Unofficially I would like to stress that this is a unique opportunity to be payed for two years to work on learning and improving the scientific Python toolstack. Gael

Re: [Numpy-discussion] SVD does not converge on "clean" matrix

2011-08-20 Thread Gael Varoquaux
On Sun, Aug 14, 2011 at 09:15:35PM +0200, Charanpal Dhanjal wrote: > Incidentally, I am confused as to why numpy calls the lapack lite > routines - when I call numpy.show_config() it seems to have detected my > ATLAS libraries and I would have expected it to use those. My rule of thumb is to nev

Re: [Numpy-discussion] Efficient way to load a 1Gb file?

2011-08-10 Thread Gael Varoquaux
On Wed, Aug 10, 2011 at 04:01:37PM -0400, Anne Archibald wrote: > A 1 Gb text file is a miserable object anyway, so it might be desirable > to convert to (say) HDF5 and then throw away the text file. +1 G ___ NumPy-Discussion mailing list NumPy-Discussi

Re: [Numpy-discussion] dtype repr change?

2011-07-27 Thread Gael Varoquaux
On Wed, Jul 27, 2011 at 05:25:20PM -0600, Charles R Harris wrote: >Well, doc tests are just a losing proposition, no one should be using them >for writing tests. It's not like this is a new discovery, doc tests have >been known to be unstable for years. Untested documentation is broken

Re: [Numpy-discussion] dtype repr change?

2011-07-27 Thread Gael Varoquaux
On Wed, Jul 27, 2011 at 04:59:17PM -0500, Mark Wiebe wrote: >but ultimately NumPy needs the ability to change its repr and other >details like it in order to progress as a software project. You have to understand that numpy is the core layer on which people have build pretty huge scientifi

Re: [Numpy-discussion] code review/build & test for datetime business day API

2011-07-25 Thread Gael Varoquaux
On Mon, Jul 25, 2011 at 03:52:48PM -0500, Mark Wiebe wrote: >Can't use numpy.datetime, since that conflicts with Python's datetime >library, especially in pylab. I don't understand that: isn't the point of namespaces to avoid those naming conflicts. To me that's just like saying that numpy

Re: [Numpy-discussion] inverting and calculating eigenvalues for many small matrices

2011-07-12 Thread Gael Varoquaux
On Tue, Jul 12, 2011 at 12:16:29PM -0400, greg whittier wrote: > Gael, your code addresses inverses, but I take it something similar for > eigenvalues of a matrix bigger than 5x5 doesn't exists since a > closed-form solution doesn't exist for finding polynomials roots for > order > 5? I guess so :

Re: [Numpy-discussion] inverting and calculating eigenvalues for many small matrices

2011-07-12 Thread Gael Varoquaux
On Mon, Jul 11, 2011 at 05:01:07PM -0400, Daniel Wheeler wrote: > Hi, I am trying to find the eigenvalues and eigenvectors as well as > the inverse for a large number of small matrices. The matrix size > (MxM) will typically range from 2x2 to 8x8 at most. If you really care about speed, for matri

Re: [Numpy-discussion] HPC missing data - was: NA/Missing Data Conference Call Summary

2011-07-06 Thread Gael Varoquaux
On Wed, Jul 06, 2011 at 08:39:37PM +0200, Dag Sverre Seljebotn wrote: > As for myself, I'll admit that I'll almost certainly continue with > explicit masking without using any of the proposed NEPs -- I have to be > extremely aware of the masks in the statistical methods I use. My gut feeling is

Re: [Numpy-discussion] miniNEP1: where= argument for ufuncs

2011-07-06 Thread Gael Varoquaux
On Wed, Jul 06, 2011 at 12:26:24PM -0700, Nathaniel Smith wrote: > A mini-NEP for the where= argument to ufuncs I _love_ this proposal and it would probably be much more useful to me than the different masked array proposal that are too focused on a specific usage pattern to answer all my needs.

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-25 Thread Gael Varoquaux
On Sat, Jun 25, 2011 at 03:16:39PM -0500, Mark Wiebe wrote: >This is why I'm also proposing to add a 'mask=' parameter to ufuncs, for >example, to expose the implementation details of the masked array system >to people who need masks but need them to be a bit different. There may be >

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-25 Thread Gael Varoquaux
On Sat, Jun 25, 2011 at 08:27:57AM -0600, Charles R Harris wrote: >Could you expand a bit on what sort of data you have and how you deal with >it. Where does it come from, how is it stored on disk, what do you do with >it? That sort of thing. 3D and 4D images. Mostly stored on disk in

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-25 Thread Gael Varoquaux
On Sat, Jun 25, 2011 at 01:02:07AM +0100, Matthew Brett wrote: > I'm personally worried that the memory overhead of array.masks will > make many of us tend to avoid them. I work with images that can > easily get large enough that I would not want an array-items size byte > array added to my storag

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Gael Varoquaux
On Thu, Jun 23, 2011 at 07:51:25PM -0400, josef.p...@gmail.com wrote: > From the perspective of statistical analysis, I don't see much > advantage of this. What to do with nans depends on the analysis, and > needs to be looked at for each case. >From someone who actually sometimes does statistics

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-23 Thread Gael Varoquaux
On Thu, Jun 23, 2011 at 03:53:31PM -0500, Mark Wiebe wrote: >concluded that adding masks to the core ndarray appears is the best way to >deal with the problem in general. It seems to me that this is going to make the numpy array a way more complex object. Althought it is currently quite si

Re: [Numpy-discussion] Adjacent matrix

2011-06-14 Thread Gael Varoquaux
On Tue, Jun 14, 2011 at 10:29:38AM -0300, Thiago Franco Moraes wrote: > I don't know if I understand. The idea is to use _make_edges_3d to > give me the connectivity, isn't it? Like for example, a 3x3 image: > 0, 1, 2 > 3, 4, 5 > 6, 7, 8 > If I use _make_edges_3d(3, 3) I get: > array([[0, 1, 3,

Re: [Numpy-discussion] Adjacent matrix

2011-06-13 Thread Gael Varoquaux
Hi, You can probably find some inspiration from https://github.com/scikit-learn/scikit-learn/blob/master/scikits/learn/feature_extraction/image.py Gaël ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/n

Re: [Numpy-discussion] tighten up ufunc casting rule

2011-06-05 Thread Gael Varoquaux
On Sun, Jun 05, 2011 at 04:41:26PM -0600, Charles R Harris wrote: > Now, with regards to the actual failures induced by the new > branch, it took me a while to understand why they where happening, > and now I realise that we probably should have explicit coercions > at these loc

Re: [Numpy-discussion] tighten up ufunc casting rule

2011-06-05 Thread Gael Varoquaux
On Sun, Jun 05, 2011 at 10:43:45PM +0200, Ralf Gommers wrote: >four in scikit-learn (plus two that don't >look related), Yeah, some of these failures are due to numerical unstabilities in tests (nasty ones, still fighting) and some simply to crappy code (HMMs are hopeless :( ). Now, with

Re: [Numpy-discussion] tighten up ufunc casting rule

2011-06-02 Thread Gael Varoquaux
On Thu, Jun 02, 2011 at 03:06:58PM -0500, Mark Wiebe wrote: >Would anyone object to, at least temporarily, tightening up the default >ufunc casting rule to 'same_kind' in NumPy master? It's a one line change, >so would be easy to undo, but such a change is very desirable in my >opin

Re: [Numpy-discussion] Contribution

2011-05-29 Thread Gael Varoquaux
On Sun, May 29, 2011 at 07:09:42AM +0400, Dmitriy Rybalkin wrote: > Thank you, Gael, I really hope that I'll improve my programming skills. > Also I have a question: Will the page with my example be deleted in > case the code is not efficient or smth else has been found to be > wrong? I cannot rea

Re: [Numpy-discussion] Contribution

2011-05-28 Thread Gael Varoquaux
I Dmitriy, Welcome. This is a great initiative. You'll learn by doing, and I am willing to bet that contributing examples can make you a good programmer. On Fri, May 27, 2011 at 09:47:24PM +0400, Dmitriy Rybalkin wrote: > I thought I could contribute to the Numpy project by > showing some examp

Re: [Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm

2011-05-17 Thread Gael Varoquaux
On Tue, May 17, 2011 at 12:55:39PM -0500, Benjamin Root wrote: >Is this hungarian method in an official scikits package, or is this just >your own thing? Right now we are playing with the idea of integrating it in the scikits learn, as it would be useful in a couple of places. I don't know

Re: [Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm

2011-05-17 Thread Gael Varoquaux
On Tue, May 17, 2011 at 09:36:40AM -0700, Hoyt Koepke wrote: > > OK, your input is making my motivation to fight with Jonker-Volgenant go > > down. I'll see with the other people involved if we still target > > Jonger-Volgenant, or if we stick with the hungarian algorithm, in which > > case the pro

Re: [Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm

2011-05-17 Thread Gael Varoquaux
On Mon, May 16, 2011 at 10:03:09AM -0700, Hoyt Koepke wrote: > > I might go that way: I already have pure-Python code that implements it > > and that I have been using for a year or so. > Fair enough -- though you'd probably get a big speed up moving to cython. Indeed. If this is needed, we'll co

Re: [Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm

2011-05-16 Thread Gael Varoquaux
On Mon, May 16, 2011 at 09:15:21AM -0700, Hoyt Koepke wrote: > > Thanks for the suggestion. Unfortunately, I do not want to add compiled > > code (or an external dependency) to the scikit for this solver, as it is > > a fairly minor step for us. I particular chances are that it will never > > be a

Re: [Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm

2011-05-16 Thread Gael Varoquaux
Hey On Mon, May 16, 2011 at 08:49:58AM -0700, Hoyt Koepke wrote: > On Mon, May 16, 2011 at 12:18 AM, Gael Varoquaux > wrote: > > Jonker-Volgenant algorithm for best linear assignment in Python, using > > numpy. Unsuprisingly, it is proving time-costly. I cannot afford to spend

[Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm

2011-05-16 Thread Gael Varoquaux
Following a suggestion by Joseph, I am trying to implement the Jonker-Volgenant algorithm for best linear assignment in Python, using numpy. Unsuprisingly, it is proving time-costly. I cannot afford to spend too much time on this, as it not to solve a problem of mine, but for the scikits.learn. Thu

Re: [Numpy-discussion] optimizing ndarray.__setitem__

2011-05-05 Thread Gael Varoquaux
On Wed, May 04, 2011 at 08:57:42AM -0700, Mark Wiebe wrote: >You'd better time this in 1.6 too. ;) >  > [2]https://github.com/numpy/numpy/commit/f60797ba64ccf33597225d23b893b6eb11149860 Awesome work! That's all I have to say. I find very enjoyable the generalized use of the time machine i

Re: [Numpy-discussion] [Scikit-learn-general] random number generator, entropy and pickling

2011-04-25 Thread Gael Varoquaux
On Mon, Apr 25, 2011 at 01:23:12PM -0500, Robert Kern wrote: > > Yes, that's exactly why we want the different objects to able to recieve > > their own PRNG. > But seriously, they are running A+B, the combination of A and B. If > A+B changes to A+B', then the results may be different. That's to be

Re: [Numpy-discussion] random number generator, entropy and pickling

2011-04-25 Thread Gael Varoquaux
On Mon, Apr 25, 2011 at 11:05:05AM -0700, T J wrote: > If code A relies on code B (eg, some numpy function) and code B > changes, then the stream of random numbers will no longer be the same. > The point here is that the user wrote code A but depended on code B, > and even though code A was unchan

[Numpy-discussion] random number generator, entropy and pickling

2011-04-25 Thread Gael Varoquaux
Hi there, We are courrently having a discussion on the scikits learn mailing list about which patterns to adopt for random number generation. One thing that is absolutely clear is that making the stream of random numbers reproducible is critical. We have several objects that can serve as random va

Re: [Numpy-discussion] ATLAS - support letter

2011-04-20 Thread Gael Varoquaux
On Wed, Apr 20, 2011 at 10:49:10PM -0700, Matthew Brett wrote: > Well - thanks for the offer - Clint was asking for individual letters > too, you could email and ask him? I can do that, and ask around me. > Are you on the math-atlas list? No I am not. > If not I'll forward you his request...

Re: [Numpy-discussion] ATLAS - support letter

2011-04-20 Thread Gael Varoquaux
On Thu, Apr 21, 2011 at 07:25:18AM +0530, pratik wrote: > If the place where he is seeking tenure does not know his name (i.e > hasn't heard of ATLAS) then it is not a good place to seek tenure in :) . Scholars undervalue code and don't realise the difficulty and the amount of work it takes to pr

Re: [Numpy-discussion] Shared memory ndarrays (update)

2011-04-10 Thread Gael Varoquaux
Hey Sturla, It's really great that you are still working on that. I'll test the code under Linux. The scipy community has moved to github. If I create a repository under github and put the code on it, would you use it? If I find time, I'll add a setup.py. Gaël __

Re: [Numpy-discussion] Partial least squares

2011-03-24 Thread Gael Varoquaux
On Thu, Mar 24, 2011 at 01:57:46PM -0700, Nadav Horesh wrote: > The code doc refer to an excellent reference [Wegelin et al. 2000], so > no real problem here. If the reference is critical I would suggest on > of the following: > 1. Put a link to the document. Do you have one? I'd be happy to add

Re: [Numpy-discussion] Partial least squares

2011-03-24 Thread Gael Varoquaux
On Thu, Mar 24, 2011 at 08:15:12PM +0100, Olivier Grisel wrote: > 2011/3/24 Nadav Horesh : > > I am looking for a partial least sqaures code refactoring for two (X,Y) > > matrices. I found the following, but they not not work for me: > > 1. MDP: Factors only one matrix (am I wrong?) > > 2. pychem:

Re: [Numpy-discussion] Partial least squares

2011-03-24 Thread Gael Varoquaux
On Thu, Mar 24, 2011 at 12:06:56PM -0700, Nadav Horesh wrote: >I am looking for a partial least sqaures code refactoring for two (X,Y) >matrices. I found the following, but they not not work for me: We have had a PLS code contributed in the scikits learn very recently: http://scikit-learn.

Re: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule)

2011-03-04 Thread Gael Varoquaux
On Fri, Mar 04, 2011 at 11:54:07AM -0800, Christoph Gohlke wrote: > I also ran tests and/or examples of a few 3rd party packages that were > built against numpy 1.5.1: scipy, pygame, PyMOL, numexpr, matplotlib, > basemap, scikits.learn, ETS.mayavi, Bottleneck, pytables, and pandas. Wow, this is

Re: [Numpy-discussion] Condensing array...

2011-02-25 Thread Gael Varoquaux
On Fri, Feb 25, 2011 at 10:52:09AM +0100, Fred wrote: > > What exactly do you mean by 'decimating'. To me is seems that you are > > looking for matrix factorization or matrix completion techniques, which > > are trendy topics in machine learning currently. > By decimating, I mean this: > input arr

Re: [Numpy-discussion] Condensing array...

2011-02-25 Thread Gael Varoquaux
On Fri, Feb 25, 2011 at 10:36:42AM +0100, Fred wrote: > I have a big array (44 GB) I want to decimate. > But this array has a lot of NaN (only 1/3 has value, in fact, so 2/3 of > NaN). > If I "basically" decimate it (a la NumPy, ie data[::nx, ::ny, ::nz], for > instance), the decimated array wi

Re: [Numpy-discussion] NumPy speed tests by NASA

2011-02-22 Thread Gael Varoquaux
On Tue, Feb 22, 2011 at 09:59:26PM +, Pauli Virtanen wrote: > > Probably because the numpy binary that the author was using was compiled > > without a blas implementation, and just using numpy's internal > > lapack_lite. This is a common problem in real life. > It doesn't use blas_lite at the

Re: [Numpy-discussion] NumPy speed tests by NASA

2011-02-22 Thread Gael Varoquaux
On Tue, Feb 22, 2011 at 04:44:56PM -0500, Alan G Isaac wrote: > On 2/22/2011 3:45 PM, Sturla Molden wrote: > > I came accross some NumPy performance tests by NASA. Comparisons against > > pure Python, Matlab, gfortran, Intel Fortran, Intel Fortran with MKL, > > and Java. For those that are interest

Re: [Numpy-discussion] Inversion of near singular matrices.

2011-01-30 Thread Gael Varoquaux
On Sun, Jan 30, 2011 at 04:15:34PM +0100, Sturla Molden wrote: > Den 30.01.2011 07:28, skrev Algis Kabaila: > > Why not simply numply.linalg.cond? This gives the condition > > number directly (and presumably performs the inspection of > > sv's). Or do you think that sv's give more useful informatio

[Numpy-discussion] [ANN] FEMTEC: Trac on open source scientific software

2011-01-30 Thread Gael Varoquaux
Hi list, This is just a note that an extra track at FEMTEC, a conference for computational methods in engineering and science, is open for open source scientific software. The organisers have a taste for Python, so if you want to submit a paper on numerical methods with Python, this is an excellen

  1   2   3   4   5   6   >