Re: [R] handling a lot of data

2012-01-30 Thread Paul Bivand
If you do not need all the variables in the SPSS files, use package 'memisc'. spss.system.file() and it's subset() allow you to just load the variables needed. You will need to transform into data.frame as the memisc data.set includes the SPSS attributes, user-missings etc. Paul Bivand Centre for

Re: [R] handling a lot of data

2012-01-30 Thread R. Michael Weylandt
This won't help with large memory issues, but just a pointer: When you start to construct data_all with these commands data_all = vector("list", 17); data_all[[1993]] = data1993; The first pre-allocates a list of length 17, but the second adds the data to the 1993rd slot requiring a complete rea

Re: [R] handling a lot of data

2012-01-30 Thread Milan Bouchet-Valat
Le lundi 30 janvier 2012 à 09:54 +0100, Petr Kurtin a écrit : > Hi, > > I have got a lot of SPSS data for years 1993-2010. I load all data into > lists so I can easily index the values over the years. Unfortunately loaded > data occupy quite a lot of memory (10Gb) - so my question is, what's the >

[R] handling a lot of data

2012-01-30 Thread Petr Kurtin
Hi, I have got a lot of SPSS data for years 1993-2010. I load all data into lists so I can easily index the values over the years. Unfortunately loaded data occupy quite a lot of memory (10Gb) - so my question is, what's the best approach to work with big data files? Can R get a value from the fil