Greetings, Regarding "NRI requires cutoff values" or not, the more recent Pencina et al's paper suggests the use of a category-free NRI, i.e. without cutoffs. Pencina, D'Agostino, Steyerberg. Statist Med 2011,30:11-21. For computations, I find Hmisc's improveProb is very flexible. Allows use for category-based or category-free NRI, and also gives IDI results. Vincent
> Message: 48 > Date: Tue, 17 Jan 2012 16:55:51 +0000 > From: "Essers, Jonah" <jonah.ess...@childrens.harvard.edu> > To: "Kevin E. Thorpe" <kevin.tho...@utoronto.ca> > Cc: "r-help@R-project.org" <r-help@r-project.org> > Subject: Re: [R] net classification improvement? > Message-ID: <cb3b10e7.b41c%jonah.ess...@childrens.harvard.edu> > Content-Type: text/plain; charset="iso-8859-1" > > Actually, I don't think I made myself clear and I wrote this late last > night....Sorry. More the issue is that the raw model predictions (from 0 > to 1) have no inherent clinical value to them. I.e. They aren't "risk of > disease" or "risk of outcome". They are raw scores that are specific to > each model and are meant to discriminate one disease from another disease. > Trying to compare models is impossible because the NRI requires cutoff > values. The cutoffs are different for each model. > > So, as I've done more reading, it appears the the IRI--Integrated > Discrimination Improvement Index--which is na?ve to cutoff values--may be > more what I'm looking for. Does this make sense? I guess I just need a > sanity check. > > I have been toying with the PredictABEL package and this seems to like my > data inputs just fine and relies on HMISC and ROCR, both packages I know > well. > > Thanks > jonah > > On 1/17/12 11:49 AM, "Kevin E. Thorpe" <kevin.tho...@utoronto.ca> wrote: > >>On 01/17/2012 07:16 AM, Essers, Jonah wrote: >>> Thanks for the reply. I think more the issue is whether it can be >>>applied >>> to cross-sectional data. This I'm not sure. This method is heavily cited >>> in the New England Journal of Medicine, but thus far I've only seen it >>> used with longitudinal data. >> >>As I recall, the Pencina et al paper does not suggest it cannot be used >>outside of longitudinal data. In fact, I don't remember them using >>longitudinal data at all. So, unless I'm misunderstanding your >>question, I think the function in Hmisc (whose name I always forget) >>should be fine. >> >>> >>> On 1/16/12 10:23 PM, "Kevin E. Thorpe"<kevin.tho...@utoronto.ca> wrote: >>> >>>> On 01/16/2012 08:10 PM, Essers, Jonah wrote: >>>>> Greetings, >>>>> >>>>> I have generated several ROC curves and would like to compare the >>>>>AUCs. >>>>> The data are cross sectional and the outcomes are binary. I am testing >>>>> which of several models provide the best discrimination. Would it be >>>>> most >>>>> appropriate to report AUC with 95% CI's? >>>>> >>>>> I have been looking in to the "net reclassification improvement" (see >>>>> below for reference) but thus far I can only find a version in Hmisc >>>>> package which requires survival data. Any idea what the best approach >>>>>is >>>>> for cross-sectional data? >>>> >>>> I believe that the function in Hmisc that does this will also work on >>>> binary data. >>>> >>>>> >>>>> Thanks >>>>> >>>>> Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr, Vasan RS. Evaluating >>>>>the >>>>> added predictive ability of a new marker: from area under the ROC >>>>>curve >>>>> to >>>>> reclassification and beyond. Stat Med 2008;27:157-172 >>>>> >>>> >> >> >>-- >>Kevin E. Thorpe >>Biostatistician/Trialist, Applied Health Research Centre (AHRC) >>Li Ka Shing Knowledge Institute of St. Michael's >>Assistant Professor, Dalla Lana School of Public Health >>University of Toronto >>email: kevin.tho...@utoronto.ca Tel: 416.864.5776 Fax: 416.864.3016 > ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
