Noah Silverman wrote:
Thanks for the suggestion.
You explained that Briar combines both accuracy and discrimination
ability. If I understand you right, that is in relation to binary
classification.
I'm not concerned with binary classification, but the accuracy of the
probability predictions.
Is there some kind of score that measures just the accuracy?
Thanks!
-N
The Brier score has nothing to do with classification. It is a
probability accuracy score.
Frank
On 8/19/09 10:42 AM, Frank E Harrell Jr wrote:
Noah Silverman wrote:
Hello,
I working on a model to predict probabilities.
I don't really care about binary prediction accuracy.
I do really care about the accuracy of my probability predictions.
Frank was nice enough to point me to the val.prob function from the
Design library. It looks very promising for my needs.
I've put together some tests and run the val.prob analysis. It
produces some very informative graphs along with a bunch of
performance measures.
Unfortunately, I'm not sure which measure, if any, is the "best"
one. I'm comparing hundreds of different models/parameter
combinations/etc. So Ideally I'd like a single value or two as the
"performance measure" for each one. That way I can pick the "best"
model from all my experiments.
As mentioned above, I'm mainly interested in the accuracy of my
probability predictions.
Does anyone have an opinion about which measure I should look at??
(I see Dxy, C, R2, D, U, Briar, Emax, Eavg, etc.)
Thanks!!
-N
It all depends on the goal, i.e., the relative value you place on
absolute accuracy vs. discrimination ability. The Brier score combines
both and other than interpretability has many advantages.
Frank
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
--
Frank E Harrell Jr Professor and Chair School of Medicine
Department of Biostatistics Vanderbilt University
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
