Hi, I have a data set that consists of about 2 million observations and several high dimensional fixed effects (2 factors at around 1000 levels each, and others with a few hundred levels). I'm looking to run linear and logit regressions. I've tried packages such as filehash and biglm to store some of the big matrices on the hard drive, but I still get errors like "Cannot allocate vector of length ...".
I've read about some iterative methods for coefficient estimation in STATA that would probably work for this, but I'm wondering if there is an R package out there meant for situations like mine. I'm running a XP x64 machine with an AMD 2.8Ghz dual core processor and 6GB of RAM, and I'm not really concerned with memory- and time-intensive solutions as long as they work. Thanks, Dan -- View this message in context: http://www.nabble.com/Large-data-sets-with-high-dimensional-fixed-effects-tp25894824p25894824.html Sent from the R help mailing list archive at Nabble.com. ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
