I agree. Avoid the lines like: iv = c( iv, min(i, j) ) I had code that was sped up by 70 times after fixing the size of my output object before entering a loop.
Cheers David -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Patrick Burns Sent: 17 October 2007 15:57 To: jim holtman Cc: r-help@r-project.org; Dieter Best Subject: Re: [R] How to speed up multiple for loop over list of data frames I suspect the vast majority of time is because of growing objects. Preallocate 'iv', 'jv', 'rho_sv' and 'rho_pv' to be their final length and then subscript into them with their values. Patrick Burns [EMAIL PROTECTED] +44 (0)20 8525 0696 http://www.burns-stat.com (home of S Poetry and "A Guide for the Unwilling S User") jim holtman wrote: >First thing to do is to use Rprof (?Rprof) on a subset of your data to >see where time is being spent. My guess is that most of it is in the >calls to 'cor' and if this is the case, they you have to figure out >some other algorithm. > >Also if these dataframes all contain numeric information, convert them >to matrices intially because the subsetting that you are doing on the >dataframe (e.g., alist[[p]][i,"v"]) can be very expensive. The output >from Rprof will help determine what course of action you should take. > >On 10/16/07, Dieter Best <[EMAIL PROTECTED]> wrote: > > >>Hi there, >> >> I have a multiple for loop over a list of data frames >> >> for ( i in 1:(N-1) ) { >> for ( j in (i+1):N ) { >> for ( p in 1:M ) { >> v_i[p] = alist[[p]][i,"v"] >> v_j[p] = alist[[p]][j,"v"] >> } >> rho_s = cor(v_i, v_j, method = "spearman") >> rho_p = cor(v_i, v_j, method = "pearson" ) >> iv = c( iv, min(i, j) ) >> jv = c( jv, max(i, j) ) >> rho_sv = c( rho_sv, rho_s) >> rho_pv = c( rho_pv, rho_p) >> } >>} >> >> N is of the order of 400, M about 800. >> >> This takes me an entire day basically. Is there anything I could do to speed things up or is cor really that slow? >> >> -- D >> >> >> >>--------------------------------- >> >> >> [[alternative HTML version deleted]] >> >>______________________________________________ >>R-help@r-project.org mailing list >>https://stat.ethz.ch/mailman/listinfo/r-help >>PLEASE do read the posting guide >>http://www.R-project.org/posting-guide.html >>and provide commented, minimal, self-contained, reproducible code. >> >> >> > > > > ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. <DIV><FONT size="1" color="gray">This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom </FONT></DIV> ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
