McCall, Ken (CMG-Dayton <Ken.McCall <at> coxinc.com> writes:
> I'm trying to run a linear mixed effects analysis on fairly large
> datasets with lmer (from the lme4 package) on a 32-bit Windows
> machine running XP with 3 GB of RAM. It's not working. (details
> below)
> I've researched the ff and bigmemory packages, but it appears they
> won't handle the mixed mode dataset I'm analyzing. It has some
> character fields for the categorical variables. It's also not clear
> a linear mixed effect regression can be run with those packages. Can
> anyone point me to a lme solution on larger datasets that can
> address the 32-bit memory limitations?
> The smallest dataframe I'm trying to run is about 120,000
> observations and 7 variables, but I'd rather run a for loop script
> on 620K+ observations. I've seen several recommendations (Kabacoff
> in 'R in Action' and others) that when possible, run R in a 64-bit
> build. Problem is I'm on a deadline, and procuring a new computer
> takes time, and approvals up the food chain. Suggestions? Is 64-bit
> my only option?
> > install.packages("lme4")
(only needs to be done once)
> > library(lme4)
> > math07g4 <- sqlQuery(conn, "select ssid, ss_chg,
> campus2, district_id, pblack, pfreelnch, pmob
> FROM codemob0607ma WHERE grade2 = 4")
Why are you attach()ing? Probably unnecessary ...
> > attach(math07g4)
> > fit07ma4 <- lmer(ss_chg ~ 1 + factor(campus2) + factor(district_id) +
> pblack + pfreelnch + pmob +
> (1 | campus2) + (1 | district_id), data=math07g4)
>
> And I get this:
> Error: cannot allocate vector of size 2.5 Gb
> In addition: Warning messages:
> 1: In model.matrix.default(mt, mf, contrasts) :
> Reached total allocation of 2187Mb: see help(memory.size)
Once upon a time there may have been an option for sparse
model matrices, but not now (I think).
Depending on whether you have any budget at all,
I wonder if you could use Amazon ... google "r amazon ec2 instance"
for more information ...
If you need more info, I would suggest posting to
r-sig-mixed-models <at> r-project.org (a specialty mailing
list for mixed models).
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
