Dear Prof. Harrell, Thank you very much for your prompt and very helpful response. I guess that since a global statistic such as a chi-sq test is not applicable in this case, the calibration curve itself from the calibration() is the most informative alternative (most graphical methods reveal more information than a single statistic anyway!). I will try the updated version in the rms package to compare.
Best Wishes Eleni Rapsomaniki Research Associate Strangeways Research Laboratory Department of Public Health and Primary Care University of Cambridge -----Original Message----- From: Frank E Harrell Jr [mailto:f.harr...@vanderbilt.edu] Sent: 23 November 2009 13:01 To: Eleni Rapsomaniki Cc: r-help@r-project.org Subject: Re: [R] Calibration score for survival probability Eleni Rapsomaniki wrote: > Good afternoon! > > I need to evaluate the goodness-of-fit (aka calibration) for survival probability estimates from a Cox model. > I tried to use 'calibrate' in the Design package but I'm not sure if it should/would produce what I need (ie a chi-sq type statistic with a table of expected vs observed probabilities). Any other functions I should be aware of? > > Also, has anybody come across an implementation of the statistic described in: > "A global goodness of fit statistic for Cox regression models" by Parzen & Lpisitz, Biometrics 55, 1999 > > Many thanks in advance > > Eleni Rapsomaniki Eleni, The Design package, and its replacement, the rms package, produces calibration curves but no chi-square test because we do not have a corresponding method for that. Formal tests are overused in this context anyway. An index such as the maximum or 90th percentile of absolute calibration error are often more useful. I have learned however that any statistical method that categorizes continuous variables (in this case, the predictions or the covariate space) is arbitrary and has many other problems. The calibrate functions in the rms package have a new option to obtain smooth calibration curves without grouping, by fitting spline hazard models during validation. Note that if you have done any model/variable selection you have to re-run such model building from scratch for each resample of the data, or the calibration plot will be over optimistic. calibrate() makes this automatic if doing backward stepdown variable selection. Many statisticians make the mistake of only "validating" the final selected model, which can only be done by one-time data splitting (which requires tens of thousands of observations to perform adequately). Frank > > Research Associate > Strangeways Research Laboratory > Department of Public Health and Primary Care > > University of Cambridge > > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > -- Frank E Harrell Jr Professor and Chair School of Medicine Department of Biostatistics Vanderbilt University ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
