Tal Galili wrote:
Hi Ben,
I just wished to give a small remark about your claim:
"it's best not to consider hypothesis testing (statistical significance) and
AIC in the same analysis."
Since in the case of forward selection for orthogonal matrix's, it can be
shown that AIC is like using a P to enter rule of 0.16. For further
reference see:page 3 of: "A SIMPLE FORWARD SELECTION PROCEDURE BASED
ONFALSE DISCOVERY RATE CONTROL" BY YOAV BENJAMINI AND YULIA GAVRILOV,
http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aoas/1239888367
Cheers,
Tal Galili
Tal,
That is not limited to orthogonal designs. When used for one variable
at a time variable selection. AIC is just a restatement of the P-value,
and as such, doesn't solve the severe problems with stepwise variable
selection other than forcing us to use slightly more sensible alpha
values. As an aside, some statisticians try to deal with multiplicity
problems caused by stepwise variable selection by making alpha smaller
than 0.05. This increases bias by giving variables whose effects are
estimated with error a greater relative chance of being selected. alpha
typically needs to be 0.5 or greater to avoid problems with stepwise
variable selection.
AIC was designed to compare two pre-specified models.
Variable selection does not compete well with shrinkage methods that
simultaneously model all potential predictors.
Frank
On Sat, Jul 4, 2009 at 1:46 AM, Ben Bolker <bol...@ufl.edu> wrote:
alexander russell-2 wrote:
Hello,
I'd like to say that it's clear when an independent variable can be ruled
out generally speaking; on the other hand in R's AIC with bbmle, if one
finds a better AIC value for a model without the given independent
variable,
versus the same model with, can we say that the independent variable is
not
likely to be significant(in the ordinary sense!)?
That is, having made a lot of models from a data set, then the best two
are
say 78.2 and 79.3 without and with (a second independent variable
respectively) should we say it's better to judge the influence of the 2nd
IV
as insignificant?
regards,
-shfets
_____________________________________
Without meaning to sound snarky, it's best not to consider hypothesis
testing (statistical significance) and AIC in the same analysis.
If you want to decide whether predictor variables have a significant
effect on a response, you should consider their effect in the full model,
via Wald test, likelihood ratio test, etc.. If you want to find the model
with the best expected predictive capability (i.e. lowest expected
Kullback-Leibler distance), you should use AIC.
Burnham and Anderson, among others, say this repeatedly.
In general, for a one-parameter difference, hypothesis testing
is "more conservative" than AIC (e.g., critical log-likelihood difference
for a p-value of 0.05 under the LRT test is 1.92, while the log-likelihood
difference required to say that a model is expected to have better
predictive capability/lower AIC is 1) -- but since they are designed to
answer
such different questions, it's not even a fair comparison.
Ben Bolker
--
View this message in context:
http://www.nabble.com/is-AIC-always-100--in-evaluating-a-model--tp24323538p24329622.html
Sent from the R help mailing list archive at Nabble.com.
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
--
Frank E Harrell Jr Professor and Chair School of Medicine
Department of Biostatistics Vanderbilt University
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
