This won't help with large memory issues, but just a pointer:
When you start to construct data_all with these commands
data_all = vector("list", 17);
data_all[[1993]] = data1993;
The first pre-allocates a list of length 17, but the second adds the
data to the 1993rd slot requiring a complete reallocation. Look at
length(data_all). You'd be better off in general with something like
this:
data_all <- vector("list", 17)
names(data_all) <- 1993: 2010
data_all[["1993"]] <- data1993
etc.
which creates a vector of length 17 with components named after the years.
If you want to automate that last bit over each year, this would work:
for( yr in 1993: 2010){
data_all[[as.character(yr)]] <- get(paste("data", yr, sep = ""))
}
It's also been pointed out to me that the Oarray package allows one to
start indexing at an arbitrary point (e.g., 1993 for the first slot)
which might be helpful for managing your data_all object.
Michael
On Mon, Jan 30, 2012 at 3:54 AM, Petr Kurtin <kur...@avast.com> wrote:
> Hi,
>
> I have got a lot of SPSS data for years 1993-2010. I load all data into
> lists so I can easily index the values over the years. Unfortunately loaded
> data occupy quite a lot of memory (10Gb) - so my question is, what's the
> best approach to work with big data files? Can R get a value from the file
> data without full loading into memory? How can a slower computer with not
> enough memory work with such data?
>
> I use the following commands:
>
> data1993 = vector("list", 4);
> data1993[[1]] = read.spss(...) # first trimester
> data1993[[2]] = read.spss(...) # second trimester
> ...
> data_all = vector("list", 17);
> data_all[[1993]] = data1993;
> ...
>
> and indexing, e.g.: data_all[[1993]][[1]]$DISTRICT, etc.
>
> Thanks,
> Petr Kurtin
>
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
