Re: [R] efficient coding with foreach and bigmemory

2011-09-30 Thread Jay Emerson
First, we strongly recommend 64-bit R. Otherwise, you may not be able to scale up as far as you would like. Second, as I think you realize, with big objects you may have to do things in chunks. I generally recommend working a column at a time rather than in blocks of rows if possible (better per

[R] efficient coding with foreach and bigmemory

2011-09-29 Thread Jean V Adams
I recently learned about the bigmemory and foreach packages and am trying to use them to help me create a very large matrix. Without those packages, I can create the type of matrix that I want with 10 columns and 5e6 rows. I would like to be able to scale up to 5e9 rows, or more, if possible.

[R] Efficient coding

2009-08-04 Thread Daniele Amberti
I have a piece of code as the one at the bottom, unfortunately since it involves time series from a db it's not easy to give to mailing list a working script. It becomes very slow after few hundred iterations over variable sp (must process several thousands). The Rprof() indicates that the probl