Re: [Numpy-discussion] svd error checking vs. speed

2014-02-20 Thread alex
On Mon, Feb 17, 2014 at 1:24 PM, Sturla Molden wrote: > Sturla Molden wrote: >> Dave Hirschfeld wrote: >> >>> Even if lapack_lite always performed the isfinite check and threw a python >>> error if False, it would be much better than either hanging or segfaulting >>> and >>> people who care about

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-17 Thread Sturla Molden
Sturla Molden wrote: > wrote: > maybe -1 >> >> statsmodels is using np.linalg.pinv which uses svd >> I never ran heard of any crash (*), and the only time I compared with >> scipy I didn't like the slowdown. > > If you did care about speed in least-sqares fitting you would not call QR > or SVD

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-17 Thread Sturla Molden
wrote: maybe -1 > > statsmodels is using np.linalg.pinv which uses svd > I never ran heard of any crash (*), and the only time I compared with > scipy I didn't like the slowdown. If you did care about speed in least-sqares fitting you would not call QR or SVD directly, but use the builting LAPA

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-17 Thread Sturla Molden
Sturla Molden wrote: > Dave Hirschfeld wrote: > >> Even if lapack_lite always performed the isfinite check and threw a python >> error if False, it would be much better than either hanging or segfaulting >> and >> people who care about the isfinite cost probably would be linking to a fast >>

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-17 Thread Sturla Molden
Dave Hirschfeld wrote: > Even if lapack_lite always performed the isfinite check and threw a python > error if False, it would be much better than either hanging or segfaulting > and > people who care about the isfinite cost probably would be linking to a fast > lapack anyway. +1 (if I have

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-17 Thread Dave Hirschfeld
Sturla Molden gmail.com> writes: > > gmail.com> wrote: > > > I use official numpy release for development, Windows, 32bit python, > > i.e. MingW 3.5 and whatever old ATLAS the release includes. > > > > a constant 13% cpu usage is 1/8 th of my 8 virtual cores. > > Based on this and Alex' mess

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-17 Thread Sturla Molden
wrote: > I use official numpy release for development, Windows, 32bit python, > i.e. MingW 3.5 and whatever old ATLAS the release includes. > > a constant 13% cpu usage is 1/8 th of my 8 virtual cores. Based on this and Alex' message it seems the offender is the f2c generated lapack_lite librar

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-17 Thread josef . pktd
On Mon, Feb 17, 2014 at 10:03 AM, alex wrote: > On Mon, Feb 17, 2014 at 4:49 AM, Dave Hirschfeld wrote: >> alex ncsu.edu> writes: >> >>> >>> Hello list, >>> >>> Here's another idea resurrection from numpy github comments that I've >>> been advised could be posted here for re-discussion. >>> >>>

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-17 Thread alex
On Mon, Feb 17, 2014 at 4:49 AM, Dave Hirschfeld wrote: > alex ncsu.edu> writes: > >> >> Hello list, >> >> Here's another idea resurrection from numpy github comments that I've >> been advised could be posted here for re-discussion. >> >> The proposal would be to make np.linalg.svd more like scip

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-17 Thread Sturla Molden
Jason Grout wrote: > For what my vote is worth, -1. I thought this was pretty much the > designed difference between the scipy and numpy linalg routines. Scipy > does the checking, and numpy provides the raw speed. Maybe this is > better resolved as a note in the documentation for numpy ab

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-17 Thread Sturla Molden
Dave Hirschfeld wrote: > It certainly shouldn't crash or hang though and for me at least it doesn't - > it returns NaN which immediately suggests to me that I've got bad input > (maybe just because I've seen it before). It might be dependent on the BLAS or LAPACK version. Since you are on Anac

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-17 Thread Jason Grout
On 2/15/14 3:37 PM, alex wrote: > The proposal would be to make np.linalg.svd more like scipy.linalg.svd > with respect to input checking. The argument against the change is > raw speed; if you know that you will never feed non-finite input to > svd, then np.linalg.svd is a bit faster than scipy.l

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-17 Thread Dave Hirschfeld
alex ncsu.edu> writes: > > Hello list, > > Here's another idea resurrection from numpy github comments that I've > been advised could be posted here for re-discussion. > > The proposal would be to make np.linalg.svd more like scipy.linalg.svd > with respect to input checking. The argument aga

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-15 Thread alex
On Sat, Feb 15, 2014 at 6:34 PM, Sebastian Berg wrote: > On Sa, 2014-02-15 at 18:20 -0500, alex wrote: > >> >> I'm not sure exactly what you mean by this. You are suggesting that >> if the svd fails with some kind of exception (possibly poorly or >> misleadingly worded) then it could be cleaned-

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-15 Thread Sebastian Berg
On Sa, 2014-02-15 at 18:20 -0500, alex wrote: > > I'm not sure exactly what you mean by this. You are suggesting that > if the svd fails with some kind of exception (possibly poorly or > misleadingly worded) then it could be cleaned-up after the fact by > checking the input, and that this would

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-15 Thread alex
On Sat, Feb 15, 2014 at 6:06 PM, Sebastian Berg wrote: > On Sa, 2014-02-15 at 17:35 -0500, josef.p...@gmail.com wrote: >> On Sat, Feb 15, 2014 at 5:12 PM, Skipper Seabold wrote: >> > On Sat, Feb 15, 2014 at 5:08 PM, wrote: >> >> >> >> On Sat, Feb 15, 2014 at 4:56 PM, Sebastian Berg >> >> wrote:

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-15 Thread Sturla Molden
wrote: > copy of np.pinv used in linear regression > https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tools/tools.py#L348 > (it's a recent change to streamline some of the linalg in regression, > and master only) Why not call lapack routine DGELSS instead? It does exactly this,

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-15 Thread Sebastian Berg
On Sa, 2014-02-15 at 17:35 -0500, josef.p...@gmail.com wrote: > On Sat, Feb 15, 2014 at 5:12 PM, Skipper Seabold wrote: > > On Sat, Feb 15, 2014 at 5:08 PM, wrote: > >> > >> On Sat, Feb 15, 2014 at 4:56 PM, Sebastian Berg > >> wrote: > >> > On Sa, 2014-02-15 at 16:37 -0500, alex wrote: > >> >> H

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-15 Thread josef . pktd
On Sat, Feb 15, 2014 at 5:18 PM, alex wrote: > On Sat, Feb 15, 2014 at 5:08 PM, wrote: >> On Sat, Feb 15, 2014 at 4:56 PM, Sebastian Berg >> wrote: >>> On Sa, 2014-02-15 at 16:37 -0500, alex wrote: Hello list, Here's another idea resurrection from numpy github comments that I've

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-15 Thread alex
On Sat, Feb 15, 2014 at 5:08 PM, wrote: > On Sat, Feb 15, 2014 at 4:56 PM, Sebastian Berg > wrote: >> On Sa, 2014-02-15 at 16:37 -0500, alex wrote: >>> Hello list, >>> >>> Here's another idea resurrection from numpy github comments that I've >>> been advised could be posted here for re-discussio

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-15 Thread josef . pktd
On Sat, Feb 15, 2014 at 5:12 PM, Skipper Seabold wrote: > On Sat, Feb 15, 2014 at 5:08 PM, wrote: >> >> On Sat, Feb 15, 2014 at 4:56 PM, Sebastian Berg >> wrote: >> > On Sa, 2014-02-15 at 16:37 -0500, alex wrote: >> >> Hello list, >> >> >> >> Here's another idea resurrection from numpy github co

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-15 Thread alex
On Sat, Feb 15, 2014 at 5:08 PM, wrote: > On Sat, Feb 15, 2014 at 4:56 PM, Sebastian Berg > wrote: >> On Sa, 2014-02-15 at 16:37 -0500, alex wrote: >>> Hello list, >>> >>> Here's another idea resurrection from numpy github comments that I've >>> been advised could be posted here for re-discussio

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-15 Thread Skipper Seabold
On Sat, Feb 15, 2014 at 5:08 PM, wrote: > On Sat, Feb 15, 2014 at 4:56 PM, Sebastian Berg > wrote: > > On Sa, 2014-02-15 at 16:37 -0500, alex wrote: > >> Hello list, > >> > >> Here's another idea resurrection from numpy github comments that I've > >> been advised could be posted here for re-disc

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-15 Thread josef . pktd
On Sat, Feb 15, 2014 at 4:56 PM, Sebastian Berg wrote: > On Sa, 2014-02-15 at 16:37 -0500, alex wrote: >> Hello list, >> >> Here's another idea resurrection from numpy github comments that I've >> been advised could be posted here for re-discussion. >> >> The proposal would be to make np.linalg.sv

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-15 Thread Sebastian Berg
On Sa, 2014-02-15 at 16:37 -0500, alex wrote: > Hello list, > > Here's another idea resurrection from numpy github comments that I've > been advised could be posted here for re-discussion. > > The proposal would be to make np.linalg.svd more like scipy.linalg.svd > with respect to input checking.

[Numpy-discussion] svd error checking vs. speed

2014-02-15 Thread alex
Hello list, Here's another idea resurrection from numpy github comments that I've been advised could be posted here for re-discussion. The proposal would be to make np.linalg.svd more like scipy.linalg.svd with respect to input checking. The argument against the change is raw speed; if you know