Hi Simon, Thanks for your reply. That is very helpful.
However, in the logistic regression example, there also appears to be a large difference in the p-values and estimates when comparing summary(model) # mgcv 1.7-11 summary(model,freq=F) # mgcv 1.7-17 These should be the same, right? But why is this not the case? The results are shown below, and are also explained in this post: http://r.789695.n4.nabble.com/mgcv-bam-very-large-standard-error-difference-between-versions-1-7-11-and-1-7-17-bug-tp4632173p4632491.html) With kind regards, Martijn # call in all versions model = bam(NoStandard3 ~ te(GeoX,GeoY,WordFreqLog.z,by=OldSpeakerContrast,d=c(2,1)) + OldSpeakerContrast + PopCntLog.z + s(Word,bs="re") + s(Placename,bs="re") + s(Word,PopAvgIncomeLog.z,bs="re") + s(Word,PopAvgAgeLog_residIncome.z,bs="re") + s(Word,PopCntLog.z,bs="re") + s(Word,YearRec.z,bs="re"),data=lexdst,family="binomial",method="REML") ### mgcv, version 1.7-11 #Family: binomial #Link function: logit # #Formula: #NoStandard3 ~ te(GeoX, GeoY, WordFreqLog.z, by = OldSpeakerContrast, # d = c(2, 1)) + OldSpeakerContrast + PopCntLog.z + s(Word, # bs = "re") + s(Placename, bs = "re") + s(Word, PopAvgIncomeLog.z, # bs = "re") + s(Word, PopAvgAgeLog_residIncome.z, bs = "re") + # s(Word, PopCntLog.z, bs = "re") + s(Word, YearRec.z, bs = "re") # #Parametric coefficients: # Estimate Std. Error z value Pr(>|z|) #(Intercept) -0.41610 0.14027 -2.966 0.00301 ** #OldSpeakerContrast1 0.44508 0.01934 23.009 < 2e-16 *** #PopCntLog.z -0.11673 0.02725 -4.284 1.83e-05 *** #--- #Signif. codes: 0 â***â 0.001 â**â 0.01 â*â 0.05 â.â 0.1 â â 1 # #Approximate significance of smooth terms: # edf Ref.df Chi.sq p-value #te(GeoX,GeoY,WordFreqLog.z):OldSpeakerContrast0 54.05 69.06 223.3 < 2e-16 #te(GeoX,GeoY,WordFreqLog.z):OldSpeakerContrast1 59.17 74.83 327.6 < 2e-16 #s(Word) 166.41 167.84 13756.0 < 2e-16 #s(Placename) 157.64 188.02 692.7 < 2e-16 #s(Word,PopAvgIncomeLog.z) 137.83 160.05 829.9 < 2e-16 #s(Word,PopAvgAgeLog_residIncome.z) 121.13 151.13 435.5 < 2e-16 #s(Word,PopCntLog.z) 97.12 133.97 205.5 7.01e-05 #s(Word,YearRec.z) 128.98 155.27 585.7 < 2e-16 # #R-sq.(adj) = 0.372 Deviance explained = 32.4% #REML score = 97450 Scale est. = 1 n = 69259 ### mgcv 1.7-17, summary(model,freq=F) #Family: binomial #Link function: logit # #Formula: #NoStandard3 ~ te(GeoX, GeoY, WordFreqLog.z, by = OldSpeakerContrast, # d = c(2, 1)) + OldSpeakerContrast + PopCntLog.z + s(Word, # bs = "re") + s(Placename, bs = "re") + s(Word, PopAvgIncomeLog.z, # bs = "re") + s(Word, PopAvgAgeLog_residIncome.z, bs = "re") + # s(Word, PopCntLog.z, bs = "re") + s(Word, YearRec.z, bs = "re") # #Parametric coefficients: # Estimate Std. Error z value Pr(>|z|) #(Intercept) -0.98150 0.52485 -1.870 0.0615 . #OldSpeakerContrast1 0.65497 0.68628 0.954 0.3399 #PopCntLog.z -0.11724 0.02726 -4.300 1.71e-05 *** #--- #Signif. codes: 0 â***â 0.001 â**â 0.01 â*â 0.05 â.â 0.1 â â 1 # #Approximate significance of smooth terms: # edf Ref.df Chi.sq p-value #te(GeoX,GeoY,WordFreqLog.z):OldSpeakerContrast0 53.62 68.48 221.4 <2e-16 #te(GeoX,GeoY,WordFreqLog.z):OldSpeakerContrast1 58.64 74.14 324.9 <2e-16 #s(Word) 166.41 168.00 14351.7 <2e-16 #s(Placename) 157.79 209.00 1160.0 <2e-16 #s(Word,PopAvgIncomeLog.z) 137.84 170.00 1062.1 <2e-16 #s(Word,PopAvgAgeLog_residIncome.z) 121.15 170.00 686.7 <2e-16 #s(Word,PopCntLog.z) 97.10 169.00 434.0 <2e-16 #s(Word,YearRec.z) 128.98 170.00 867.4 <2e-16 # #R-sq.(adj) = 0.372 Deviance explained = 32.4% #REML score = 97449 Scale est. = 1 n = 69259 Met vriendelijke groet, Martijn -- ******************************************* Martijn Wieling http://www.martijnwieling.nl wiel...@gmail.com +31(0)614108622 ******************************************* University of Groningen http://www.rug.nl/staff/m.b.wieling ******************************************* On Mon, Jun 11, 2012 at 10:34 AM, Simon Wood <s.w...@bath.ac.uk> wrote: > Dear Martijn, > > You are right that the change in the summary defaults was not a sensible > thing to do, and I'll change it back. However in the mean time you can use > 1.7-17 with summary(...,freq=FALSE), when you have random effects in the > model (it really shouldn't make a statistically meaningful difference > otherwise). > > I made the change because it leads to better p-value performance when > parametric model component are penalized using gam's `paraPen' argument, but > as you have demonstrated this then uses a fixed effects distribution that is > not really appropriate when there are random effects present. I should > probably just deal with the paraPen case in the paraPen documentation. > > Technically the difference here is that with freq=TRUE the covariance matrix > for the model coefficient estimators/predictions is taken to be > > (X'X+S)^{-1}X'X(X'X+S)^{-1}\sigma^2 > > where X is the model matrix and S the penalty matrix for the model (Gaussian > case, non-Gaussian also has weights). This is based on the fact that the > coefficient estimates/predictions are > \hat \beta = (X'X+S)^{-1}X'y > where y is the response data, and y_i iid N(0,\sigma^2). > > When freq=FALSE the covariance matrix is the one that comes from the > Bayesian/random effects view of the smoothing process, and is > > (X'X+S)^{-1}\sigma^2 > > The latter is usually what you want when you have true random effects in the > model (rather than smooths which are usually not in the model as true random > effects). > > best, > Simon > > > > On 03/06/12 18:45, Martijn Wieling wrote: >> >> Dear useRs, >> >> I've ran some additional analyses (see below), which strongly suggest >> the standard errors of the bam (and gam) function are much too low in >> mgcv version 1.7-17, at least when including an s(X,bs="re") term. >> Until this issue has been clarified, it's perhaps best to use an older >> version of mgcv (unfortunately, however, in earlier versions the >> p-value calculation of s(X,bs="re") is not correct). All analyses were >> conducted in R 2.15.0. >> >> My approach was the following: I created a mixed-effects regression >> model with a single random intercept and only linear predictors. In my >> view, the results using lmer (lme4) should be comparable to those of >> bam and gam (mgcv). This was the case when using an older version of >> mgcv (version 1.7-13), but this is not the case anymore in version >> 1.7-17. In version 1.7-17, the standard errors and p-values are much >> lower and very similar to those of a linear model (which does not take >> the random-effects structure into account). The R-code and results are >> shown below. (The results using gam are not shown, but show the same >> pattern.) >> >> Furthermore, note that the differences in standard errors become less >> severe (but still noticeable) when less data is involved (e.g., using >> only 500 rows as opposed to>100.000). Finally, when not including an >> s(X,bs="re") term, but another non-random-effect smooth, the standard >> errors do not appear to be structurally lower (only for some >> variables, but not by a great deal - see also below). >> >> With kind regards, >> Martijn Wieling >> University of Groningen >> >> #### lme4 model (most recent version of lme4) >> modelLMER<- lmer(RefPMIdistMeanLog.c ~ SpYearBirth.z*IsAragon + >> SpIsMale + (1|Key), data=wrddst) >> # Estimate Std. Error t value >> #SpYearBirth.z -0.012084 0.004577 -2.640 >> #IsAragon 0.138959 0.010040 13.840 >> #SpIsMale -0.003087 0.008290 -0.372 >> #SpYearBirth.z:IsAragon 0.015429 0.010159 1.519 >> >> >> #### mgcv 1.7-13, default (method = "REML") - almost identical to >> modelLMER >> modelBAMold<- bam(RefPMIdistMeanLog.c ~ SpYearBirth.z*IsAragon + >> SpIsMale + s(Key,bs="re"), data=wrddst) >> # Estimate Std. Error t value Pr(>|t|) >> #SpYearBirth.z -0.012084 0.004578 -2.640 0.00829 ** >> #IsAragon 0.138959 0.010042 13.838< 2e-16 *** >> #SpIsMale -0.003087 0.008292 -0.372 0.70968 >> #SpYearBirth.z:IsAragon 0.015429 0.010160 1.519 0.12886 >> >> >> #### mgcv 1.7-17, method = "REML" - standard errors greatly reduced >> # (comparable to standard errors of LM without random intercept) >> modelBAMnew<- bam(RefPMIdistMeanLog.c ~ SpYearBirth.z*IsAragon + >> SpIsMale + s(Key,bs="re"), data=wrddst); print(testje,cor=F) >> # Estimate Std. Error t value Pr(>|t|) >> #SpYearBirth.z -0.012084 0.001159 -10.428< 2e-16 *** >> #IsAragon 0.138959 0.002551 54.472< 2e-16 *** >> #SpIsMale -0.003087 0.002098 -1.471 0.141 >> #SpYearBirth.z:IsAragon 0.015429 0.002587 5.965 2.45e-09 *** >> >> #### lm results, standard errors comparable to mgcv 1.7-17 >> modelLM<- lm(RefPMIdistMeanLog.c ~ SpYearBirth.z*IsAragon + SpIsMale, >> data=wrddst) >> # Estimate Std. Error t value Pr(>|t|) >> #(Intercept) -0.025779 0.001653 -15.595< 2e-16 *** >> #SpYearBirth.z -0.011906 0.001182 -10.070< 2e-16 *** >> #IsAragon 0.139323 0.002603 53.531< 2e-16 *** >> #SpIsMale -0.003076 0.002140 -1.437 0.151 >> #SpYearBirth.z:IsAragon 0.015252 0.002639 5.780 7.49e-09 *** >> >> >> #### mgcv 1.7-17, default (method = "fREML") - completely different >> from previous models >> modelBAMfREML<- bam(RefPMIdistMeanLog.c ~ SpYearBirth.z*IsAragon + >> SpIsMale + s(Key,bs="re"), data=wrddst); print(testje,cor=F) >> # Estimate Std. Error t value Pr(>|t|) >> #(Intercept) -0.025391 0.106897 -0.238 0.812 >> #SpYearBirth.z -0.012084 0.076300 -0.158 0.874 >> #IsAragon 0.138959 0.166697 0.834 0.405 >> #SpIsMale -0.003087 0.138291 -0.022 0.982 >> #SpYearBirth.z:IsAragon 0.015429 0.168260 0.092 0.927 >> # >> #Approximate significance of smooth terms: >> # edf Ref.df F p-value >> #s(Key) -38.95 310 15.67<2e-16 *** >> >> >> #### differences w.r.t. standard smooths >> #### mgcv version 1.7-13 >> m2old<- bam(RefPMIdistMeanLog.c ~ s(GeoX,GeoY) + >> SpYearBirth.z*IsAragon + SpIsMale, data=wrddst, method="REML") >> ## RESULTS >> #Family: gaussian >> #Link function: identity >> # >> #Formula: >> #RefPMIdistMeanLog.c ~ s(GeoX, GeoY) + SpYearBirth.z * IsAragon + >> # SpIsMale >> # >> #Parametric coefficients: >> # Estimate Std. Error t value Pr(>|t|) >> #(Intercept) -0.001386 0.004982 -0.278 0.7809 >> #SpYearBirth.z -0.012950 0.001167 -11.097< 2e-16 *** >> #IsAragon 0.020532 0.023608 0.870 0.3845 >> #SpIsMale -0.004788 0.002219 -2.158 0.0309 * >> #SpYearBirth.z:IsAragon 0.015611 0.002600 6.005 1.92e-09 *** >> #--- >> #Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 >> # >> #Approximate significance of smooth terms: >> # edf Ref.df F p-value >> #s(GeoX,GeoY) 27.11 28.14 126.2<2e-16 *** >> #--- >> #Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 >> # >> #R-sq.(adj) = 0.0555 Deviance explained = 5.58% >> #REML score = 39232 Scale est. = 0.11734 n = 112608 >> >> >> #### mgcv version 1.7-17 >> m2new<- bam(RefPMIdistMeanLog.c ~ s(GeoX,GeoY) + >> SpYearBirth.z*IsAragon + SpIsMale, data=wrddst, method="REML") >> #Family: gaussian >> #Link function: identity >> # >> #Formula: >> #RefPMIdistMeanLog.c ~ s(GeoX, GeoY) + SpYearBirth.z * IsAragon + >> # SpIsMale >> # >> #Parametric coefficients: >> # Estimate Std. Error t value Pr(>|t|) >> #(Intercept) -0.001388 0.003938 -0.352 0.7245 >> #SpYearBirth.z -0.012950 0.001167 -11.098< 2e-16 *** >> #IsAragon 0.020543 0.018055 1.138 0.2552 >> #SpIsMale -0.004788 0.002215 -2.161 0.0307 * >> #SpYearBirth.z:IsAragon 0.015611 0.002600 6.005 1.92e-09 *** >> #--- >> #Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 >> # >> #Approximate significance of smooth terms: >> # edf Ref.df F p-value >> #s(GeoX,GeoY) 27.11 28.14 126.2<2e-16 *** >> #--- >> #Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 >> # >> #R-sq.(adj) = 0.0555 Deviance explained = 5.58% >> #REML score = 39232 Scale est. = 0.11734 n = 112608 >> >> >> On Sat, Jun 2, 2012 at 6:25 PM, Martijn Wieling<wiel...@gmail.com> wrote: >>> >>> Dear useRs, >>> >>> I reran an analysis with bam (mgcv, version 1.7-17) originally >>> conducted using an older version of bam (mgcv, version 1.7-11) and >>> this resulted in the same estimates, but much lower standard errors >>> (in some cases 20 times as low) and lower p-values. This obviously >>> results in a larger set of significant predictors. Is this result >>> expected given the improvements in the new version? Or this a bug and >>> are the p-values of bam in mgcv 1.7-17 too low? The summaries of both >>> versions are shown below to enable a comparison. >>> >>> In addition, applying the default method="fREML" (mgcv version 1.7-17) >>> on the same dataset yields only non-significant results, while all >>> results are highly significant using method="REML". Furthermore, it >>> also results in large negative (e.g., -8757) edf values linked to >>> s(X,bs="RE") terms. Is this correct, or is this a bug? The summary of >>> the model using method="fREML" is also shown below. >>> >>> I hope someone can shed some light on this. >>> >>> With kind regards, >>> Martijn Wieling, >>> University of Groningen >>> >>> ################################# >>> ### mgcv version 1.7-11 >>> ################################# >>> >>> Family: gaussian >>> Link function: identity >>> >>> Formula: >>> RefPMIdistMeanLog.c ~ s(GeoX, GeoY) + RefVratio.z + >>> IsSemiwordOrDemonstrative + >>> RefSoundCnt.z + SpYearBirth.z * IsAragon + PopCntLog_residGeo.z + >>> s(Word, bs = "re") + s(Key, bs = "re") >>> >>> Parametric coefficients: >>> Estimate Std. Error t value Pr(>|t|) >>> (Intercept) -0.099757 0.020234 -4.930 8.23e-07 *** >>> RefVratio.z 0.105705 0.013328 7.931 2.19e-15 *** >>> IsSemiwordOrDemonstrative 0.289828 0.046413 6.245 4.27e-10 *** >>> RefSoundCnt.z 0.119981 0.021202 5.659 1.53e-08 *** >>> SpYearBirth.z -0.011396 0.002407 -4.734 2.21e-06 *** >>> IsAragon 0.055678 0.033137 1.680 0.09291 . >>> PopCntLog_residGeo.z -0.006504 0.003279 -1.984 0.04731 * >>> SpYearBirth.z:IsAragon 0.015871 0.005459 2.907 0.00365 ** >>> --- >>> Signif. codes: 0 â***â 0.001 â**â 0.01 â*â 0.05 â.â 0.1 â â 1 >>> >>> Approximate significance of smooth terms: >>> edf Ref.df F p-value >>> s(GeoX,GeoY) 24.01 24.21 31.16<2e-16 *** >>> s(Word) 352.29 347.00 501.57<2e-16 *** >>> s(Key) 269.75 289.25 10.76<2e-16 *** >>> --- >>> Signif. codes: 0 â***â 0.001 â**â 0.01 â*â 0.05 â.â 0.1 â â 1 >>> >>> R-sq.(adj) = 0.693 Deviance explained = 69.4% >>> REML score = -22476 Scale est. = 0.038177 n = 112608 >>> >>> >>> ################################# >>> ### mgcv version 1.7-17, much lower p-values and standard errors than >>> version 1.7-11 >>> ################################# >>> >>> Family: gaussian >>> Link function: identity >>> >>> Formula: >>> RefPMIdistMeanLog.c ~ s(GeoX, GeoY) + RefVratio.z + >>> IsSemiwordOrDemonstrative + >>> RefSoundCnt.z + SpYearBirth.z * IsAragon + PopCntLog_residGeo.z + >>> s(Word, bs = "re") + s(Key, bs = "re") >>> >>> Parametric coefficients: >>> Estimate Std. Error t value Pr(>|t|) >>> (Intercept) -0.0997566 0.0014139 -70.552< 2e-16 *** >>> RefVratio.z 0.1057049 0.0006565 161.010< 2e-16 *** >>> IsSemiwordOrDemonstrative 0.2898285 0.0020388 142.155< 2e-16 *** >>> RefSoundCnt.z 0.1199813 0.0009381 127.905< 2e-16 *** >>> SpYearBirth.z -0.0113956 0.0006508 -17.509< 2e-16 *** >>> IsAragon 0.0556777 0.0057143 9.744< 2e-16 *** >>> PopCntLog_residGeo.z -0.0065037 0.0007938 -8.193 2.58e-16 *** >>> SpYearBirth.z:IsAragon 0.0158712 0.0014829 10.703< 2e-16 *** >>> --- >>> Signif. codes: 0 â***â 0.001 â**â 0.01 â*â 0.05 â.â 0.1 â â 1 >>> >>> Approximate significance of smooth terms: >>> edf Ref.df F p-value >>> s(GeoX,GeoY) 24.01 24.21 31.15<2e-16 *** >>> s(Word) 352.29 347.00 587.26<2e-16 *** >>> s(Key) 269.75 313.00 4246.76<2e-16 *** >>> --- >>> Signif. codes: 0 â***â 0.001 â**â 0.01 â*â 0.05 â.â 0.1 â â 1 >>> >>> R-sq.(adj) = 0.693 Deviance explained = 69.4% >>> REML score = -22476 Scale est. = 0.038177 n = 112608 >>> >>> >>> ################################# >>> ### mgcv version 1.7-17, default: method="fREML", all p-values >>> non-significant and negative edf's of s(X,bs="re") >>> ################################# >>> >>> Family: gaussian >>> Link function: identity >>> >>> Formula: >>> RefPMIdistMeanLog.c ~ s(GeoX, GeoY) + RefVratio.z + >>> IsSemiwordOrDemonstrative + >>> RefSoundCnt.z + SpYearBirth.z * IsAragon + PopCntLog_residGeo.z + >>> s(Word, bs = "re") + s(Key, bs = "re") >>> >>> Parametric coefficients: >>> Estimate Std. Error t value Pr(>|t|) >>> (Intercept) -0.099757 1.730235 -0.058 0.954 >>> RefVratio.z 0.105705 1.145329 0.092 0.926 >>> IsSemiwordOrDemonstrative 0.289828 4.167237 0.070 0.945 >>> RefSoundCnt.z 0.119981 1.901158 0.063 0.950 >>> SpYearBirth.z -0.011396 0.034236 -0.333 0.739 >>> IsAragon 0.055678 0.298629 0.186 0.852 >>> PopCntLog_residGeo.z -0.006504 0.041981 -0.155 0.877 >>> SpYearBirth.z:IsAragon 0.015871 0.077142 0.206 0.837 >>> >>> Approximate significance of smooth terms: >>> edf Ref.df F p-value >>> s(GeoX,GeoY) -1376 1 7.823 0.00516 ** >>> s(Word) -8298 347 577.910< 2e-16 *** >>> s(Key) -1421 316 13.512< 2e-16 *** >>> --- >>> Signif. codes: 0 â***â 0.001 â**â 0.01 â*â 0.05 â.â 0.1 â â 1 >>> >>> R-sq.(adj) = 0.741 Deviance explained = 69.4% >>> fREML score = -22476 Scale est. = 0.038177 n = 112608 >> >> >> > > > -- > Simon Wood, Mathematical Science, University of Bath BA2 7AY UK > +44 (0)1225 386603 http://people.bath.ac.uk/sw283 ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
