Re: [Numpy-discussion] Efficient square distance computation

2013-10-10 Thread Ke Sun
> > > On a machine I had access to it took about 20 minutes. > > How? I am using matrix multiplication (the same code as > > http://stackoverflow.com/a/4856692) and it runs for around 18 hours. > > > > > make sure you are using an optimized BLAS library. > e.g. with openblas I could also do this si

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Matthew Brett
Hi, On Tue, Oct 8, 2013 at 12:44 PM, Matthew Brett wrote: > Hi, > > On Tue, Oct 8, 2013 at 4:38 AM, Ke Sun wrote: >> On Tue, Oct 08, 2013 at 01:49:14AM -0700, Matthew Brett wrote: >>> Hi, >>> >>> On Tue, Oct 8, 2013 at 1:06 AM, Ke Sun wrote: >>> > Dear all, >>> > >>> > I have written the follow

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Matthew Brett
Hi, On Tue, Oct 8, 2013 at 4:38 AM, Ke Sun wrote: > On Tue, Oct 08, 2013 at 01:49:14AM -0700, Matthew Brett wrote: >> Hi, >> >> On Tue, Oct 8, 2013 at 1:06 AM, Ke Sun wrote: >> > Dear all, >> > >> > I have written the following function to compute the square distances of a >> > large >> > matri

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Jaime Fernández del Río
On Tue, Oct 8, 2013 at 4:38 AM, Ke Sun wrote: > On Tue, Oct 08, 2013 at 01:49:14AM -0700, Matthew Brett wrote: > > Hi, > > > > On Tue, Oct 8, 2013 at 1:06 AM, Ke Sun wrote: > > > Dear all, > > > > > > I have written the following function to compute the square distances > of a large > > > matrix

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Julian Taylor
On Tue, Oct 8, 2013 at 1:38 PM, Ke Sun wrote: > > > On a machine I had access to it took about 20 minutes. > How? I am using matrix multiplication (the same code as > http://stackoverflow.com/a/4856692) and it runs for around 18 hours. > > make sure you are using an optimized BLAS library. e.g. w

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Ke Sun
On Tue, Oct 08, 2013 at 01:49:14AM -0700, Matthew Brett wrote: > Hi, > > On Tue, Oct 8, 2013 at 1:06 AM, Ke Sun wrote: > > Dear all, > > > > I have written the following function to compute the square distances of a > > large > > matrix (each sample a row). It compute row by row and print the ov

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Julian Taylor
> Out of interest, how did you do this with matrix multiplication? http://stackoverflow.com/a/4856692 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Henry Gomersall
On 08/10/13 09:49, Matthew Brett wrote: > On Tue, Oct 8, 2013 at 1:06 AM, Ke Sun wrote: >> >Dear all, >> > >> >I have written the following function to compute the square distances of a >> >large >> >matrix (each sample a row). It compute row by row and print the overall >> >progress. >> >The pr

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Henry Gomersall
On 08/10/13 09:06, Ke Sun wrote: > I give as input a 70,000x800 matrix. The output should be a 70,000x70,000 > matrix. The program runs really slow (16 hours for 1/3 progress). And it eats > 36G memory (fortunately I have enough). At this stage I'd be asking myself what I'm trying to achieve and w

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Julian Taylor
your computation is symmetric so you only need to compute the upper or lower triangle which will save both memory and time. On Tue, Oct 8, 2013 at 10:06 AM, Ke Sun wrote: > Dear all, > > I have written the following function to compute the square distances of a > large > matrix (each sample a r

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Matthew Brett
Hi, On Tue, Oct 8, 2013 at 1:06 AM, Ke Sun wrote: > Dear all, > > I have written the following function to compute the square distances of a > large > matrix (each sample a row). It compute row by row and print the overall > progress. > The progress output is important and I didn't use matrix m

[Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Ke Sun
Dear all, I have written the following function to compute the square distances of a large matrix (each sample a row). It compute row by row and print the overall progress. The progress output is important and I didn't use matrix multiplication. I give as input a 70,000x800 matrix. The output sh