On 23.12.2011 14:54, Brendan Halpin wrote:
I've been running a glmer logit on a very large data set (600k obs).
Running on a 10% subset works correctly, but for the complete data set,
R completes apparently without error, but does not display the results.
Given these jobs take about 200 hours, it's very hard to make progress
by trial and error.
I append the code and the sample and complete output. As is apparent, I
upgraded R during the complete run, but I recall testing on the
subsample with the earlier version too. I am also assuming that
upgrading R will not affect the running process -- is this true?
Err, this depends on the platform and the way you are using R.
If you change some parts of R while that is running, it may result into
unexpected behaviour or a crash if R accesses the files after the
upgrade, of course.
Best wishes,
Uwe Ligges
I'd be grateful for any leads. In the meantime I'll be running with
larger subsamples!
Regards,
Brendan Halpin
- code ---------------------------------------------------------------
library(arm)
library(foreign)
mlm<- read.dta("../workingdata.dta")
attach(mlm)
gender<- as.factor(stu_gend)
yr<- year - 1998
failure<- (lmer(fail ~
1 + cao + subj1 + subj2 + subj3 + gender + yr + ageentry +
as.factor(yrs5)
+ modsize + meancao + depfemr + (1|deptno) + (1|modinst) +
(1|ulid) ,
na.action = na.exclude, family = binomial (link="logit")))
display(failure, digits=5, detail=TRUE)
----------------------------------------------------------------------
- output with 10% sample data ----------------------------------------
R version 2.14.0 (2011-10-31)
Copyright (C) 2011 The R Foundation for Statistical Computing
ISBN 3-900051-07-0
Platform: i486-pc-linux-gnu (32-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
library(arm)
arm (Version 1.4-13, built: 2011-6-19)
Working directory is /home/brendan/work/mlmmarks/genderECSR
library(foreign)
mlm<- read.dta("../worksample-random1.dta")
attach(mlm)
gender<- as.factor(stu_gend)
yr<- year - 1998
failure<- (lmer(fail ~
+ 1 + cao + subj1 + subj2 + subj3 + gender + yr + ageentry +
as.factor(yrs5)
+ + modsize + meancao + depfemr + (1|deptno) + (1|modinst) + (1|ulid) ,
na.action = na.exclude, family = binomial (link="logit")))
display(failure, digits=5, detail=TRUE)
glmer(formula = fail ~ 1 + cao + subj1 + subj2 + subj3 + gender +
yr + ageentry + as.factor(yrs5) + modsize + meancao + depfemr +
(1 | deptno) + (1 | modinst) + (1 | ulid), family = binomial(link =
"logit"),
na.action = na.exclude)
coef.est coef.se z value Pr(>|z|)
(Intercept) 2.63826 0.97870 2.69568 0.00702
cao -2.08963 0.11987 -17.43314 0.00000
subj1 0.02608 0.23573 0.11064 0.91190
subj2 -0.55668 0.32759 -1.69932 0.08926
subj3 -1.57120 0.30664 -5.12400 0.00000
genderM 0.36368 0.09188 3.95845 0.00008
yr 0.06067 0.01658 3.65996 0.00025
ageentry -0.00720 0.04338 -0.16598 0.86817
as.factor(yrs5)1 -0.25181 0.05712 -4.40806 0.00001
as.factor(yrs5)2 -0.54725 0.07601 -7.20005 0.00000
as.factor(yrs5)3 -1.07483 0.08660 -12.41184 0.00000
as.factor(yrs5)4 -1.22447 0.14373 -8.51932 0.00000
as.factor(yrs5)5 -1.55032 0.31342 -4.94653 0.00000
modsize 0.03387 0.02533 1.33733 0.18112
meancao 1.08747 0.10748 10.11780 0.00000
depfemr -1.49097 0.49350 -3.02122 0.00252
Error terms:
Groups Name Std.Dev.
modinst (Intercept) 1.14308
ulid (Intercept) 1.54030
deptno (Intercept) 0.52497
Residual 1.00000
---
number of obs: 63254, groups: modinst, 9076; ulid, 2275; deptno, 26
AIC = 30275.2, DIC = 30237.2
deviance = 30237.2
Loading required package: MASS
Loading required package: Matrix
Loading required package: lattice
Attaching package: ‘Matrix’
The following object(s) are masked from ‘package:base’:
det
Loading required package: lme4
Attaching package: ‘lme4’
The following object(s) are masked from ‘package:stats’:
AIC, BIC
Loading required package: R2WinBUGS
Loading required package: coda
Attaching package: ‘coda’
The following object(s) are masked from ‘package:lme4’:
HPDinterval
Loading required package: abind
Loading required package: foreign
Attaching package: ‘arm’
The following object(s) are masked from ‘package:coda’:
traceplot
----------------------------------------------------------------------
- output with complete data ------------------------------------------
R version 2.13.1 (2011-07-08)
Copyright (C) 2011 The R Foundation for Statistical Computing
ISBN 3-900051-07-0
Platform: i486-pc-linux-gnu (32-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
library(arm)
arm (Version 1.4-13, built: 2011-6-19)
Working directory is /home/brendan/work/mlmmarks/genderECSR
library(foreign)
mlm<- read.dta("../workingdata.dta")
attach(mlm)
gender<- as.factor(stu_gend)
yr<- year - 1998
failure<- (lmer(fail ~
+ 1 + cao + subj1 + subj2 + subj3 + gender + yr + ageentry +
as.factor(yrs5)
+ + modsize + meancao + depfemr + (1|deptno) + (1|modinst) + (1|ulid) ,
na.action = na.exclude, family = binomial (link="logit")))
Loading required package: MASS
Loading required package: Matrix
Loading required package: lattice
Attaching package: ‘Matrix’
The following object(s) are masked from ‘package:base’:
det
Loading required package: lme4
Attaching package: ‘lme4’
The following object(s) are masked from ‘package:stats’:
AIC, BIC
Loading required package: R2WinBUGS
Loading required package: coda
Attaching package: ‘coda’
The following object(s) are masked from ‘package:lme4’:
HPDinterval
Loading required package: abind
Loading required package: foreign
Attaching package: ‘arm’
The following object(s) are masked from ‘package:coda’:
traceplot
----------------------------------------------------------------------
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
