On Jan 22, 2010, at 12:01 PM, Na'im R. Tyson wrote:
Thank you for your very prompt response. The authors of the ROCR
package informed me the package works as stated in the documentation
as long as you use R version 2.9.0--and indeed, it does! I do not
mind using a slightly older version of R to get the results I need.
It is useful to have the 'spread.estimate' feature for plotting, but
I wanted a numerical confidence interval. In short, I am comparing
a few binary classifiers, and I want to show that the confidence
intervals for the average AUC overlap. You can see this graphically
with the 'spread.estimate' option, but my dissertation committee
prefers numbers.
Ouch. I'm pretty sure you can find discussion on R-help offered by Dr
Harrell regarding why overlap of CI's for ROC curves would have low
power to detect important differences. And there is also the fact that
overlap of 95% CI's is not an appropriate test even in the more
straightforward situation of comparing group means.
--
David.
Again, thank you for all of your help. It has led me in the right
direction.
Regards,
Na'im
On Jan 22, 2010, at 8:51 AM, David Winsemius wrote:
On Jan 22, 2010, at 8:31 AM, Frank E Harrell Jr wrote:
David Winsemius wrote:
On Jan 22, 2010, at 3:53 AM, Na'im R. Tyson wrote:
Dear R-philes,
I am plotting ROC curves for several cross-validation runs of a
classifier (using the function below). In addition to the
average AUC, I am interested in obtaining a confidence interval
for the average AUC. Is there a straightforward way to do this
via the ROCR package?
You should probably contact the authors. When I tried using that
package a few weeks ago, several of the annotation features were
broken. I contacted the author who said there had been problems
after converting to S4 method. He also said there would be a fix
but not immediately. There has been a release since that time and
I tried it, but it did not appear to fix the problems I
encountered. All I was able to get were very simple ROC curves
without any confidence intervals or marking of levels. I ended up
turning to the Epi package for what I needed ( but I did not need
confidence intervals so cannot comment on that aspect.)
I'm wondering what was broken with the S3 implementation that made
them change to S4.
I was typing from memory and may not have conveyed accurately what
was in the message. He mention changing versions but my attribution
of that problem as switching from S3 to S4 methods seems to have
been a manufactured memory. Furthermore, on loading the package in
its current form, I am no longer having the problems I earlier
experienced.
So now my question to Tyson would be, what you were hoping to see
with your request for confidence intervals? The "spread estimate"
feature seems to have been fixed in version 1.0-4.
Frank
--
Frank E Harrell Jr Professor and Chairman School of
Medicine
Department of Biostatistics Vanderbilt
University
David Winsemius, MD
Heritage Laboratories
West Hartford, CT
David Winsemius, MD
Heritage Laboratories
West Hartford, CT
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
