It is not possible to have one cutoff point unless you have a very strange utility function. Nor is there a need for a cutoff when using a probability model.
It is not advisable to compare models based on ROC area as this loses power. A likelihood-based approach is recommended. Frank Andra Isan wrote: > > Hello All, > I have used logistic regression glm in R and I am evaluating two models > both learned with glm but with different predictors. model1 <- glm (Y ~ > x4+ x5+ x6+ x7, data = dat, family = binomial(link=logit))model2 <- glm > (Y~ x1 + x2 +x3 , data = dat, family = binomial(link=logit)) > and I would like to compare these two models based on the prediction that > I get from each model: > pred1 = predict(model1, test.data, type = "response")pred2 = > predict(model2, test.data, type = "response") > I have used ROCR package to compare them:pr1 = prediction(pred1,test.y)pf1 > = performance(pr1, measure = "prec", x.measure = "rec") plot(pf1) which > cutoff this plot is based on? > pr2 = prediction(pred2,test.y)pf2 = performance(pr2, measure = "prec", > x.measure = "rec")pf2_roc = performance(pr2,measure="err")plot(pf2) > First of all, I would like to use cutoff = 0.5 and plot the ROC, > precision-recall curves based on that cutoff value. In other words, how to > define a cut off value in performance function?For example, in pf2_roc = > performance(pr2,measure="err"), when I do plot(pf2_roc), it plots for > every single cutoff point. I only want to have one cut off point, is there > any way to do that?Second, I would like to see the performance of the two > models based on the above measures on the same plot so the comparison > would be easier. In other words, how can I plot (pf1, pf2) and compare > them together?plot(pf1, pf2) would give me an error as follows:Error in > as.double(x) : cannot coerce type 'S4' to vector of type 'double' > Could you please help me with that? > Thanks a lot,Andra > > > > [[alternative HTML version deleted]] > > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > ----- Frank Harrell Department of Biostatistics, Vanderbilt University -- View this message in context: http://r.789695.n4.nabble.com/ROCR-package-question-for-evaluating-two-regression-models-tp3787301p3787855.html Sent from the R help mailing list archive at Nabble.com. ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
