On Fri, Feb 28, 2014 at 1:13 AM, zhenjiang zech xu <zhenjiang...@gmail.com> wrote: > Dear all, > > I did a 5-repeat of 10-fold cross validation using partial least square > regression model provided by caret package. Can anyone tell me how are the > values in plsTune$resample calculated? Is that predicted on each hold-out > set using the model which is trained on the rest data with the optimized > parameter tuned from previous cross validation?
Yes, those values are the performance estimates across each hold-out using the final model. There is an option in trainControl() that will have it return the resamples from all models too. > So in the following > example, firstly, 5-repeat of 10-fold cross validation gives 2 for ncomp as > the best, and then using ncomp of 2 and the training data to build a model > and then predict the hold-out data with the model to give a RMSE and > RSQUARE - is what I am thinking true? It is. Max > > >> plsTune > 524 samples > 615 predictors > > Pre-processing: centered, scaled > Resampling: Cross-Validation (10 fold, repeated 5 times) > > Summary of sample sizes: 472, 472, 471, 471, 471, 471, ... > > Resampling results across tuning parameters: > > ncomp RMSE Rsquared RMSE SD Rsquared SD > 1 16.8 0.434 1.47 0.0616 > 2 14.3 0.612 2.21 0.0768 > 3 13.5 0.704 6.33 0.145 > 4 14.6 0.706 9.29 0.163 > 5 15.2 0.703 10.9 0.172 > 6 16.5 0.69 13.4 0.181 > 7 18.4 0.672 17.8 0.194 > 8 20 0.651 20.4 0.199 > 9 20.9 0.634 20.9 0.199 > 10 22.1 0.613 22.1 0.197 > 11 23.3 0.599 23.8 0.198 > 12 24 0.588 24.7 0.198 > 13 24.9 0.572 25.2 0.197 > 14 25.8 0.557 26.2 0.194 > 15 26.2 0.544 25.8 0.191 > 16 26.6 0.532 25.5 0.187 > > RMSE was used to select the optimal model using the one SE rule. > The final value used for the model was ncomp = 2. >> >> plsTune$resample > ncomp RMSE Rsquared Resample > 1 2 13.61569 0.6349700 Fold06.Rep4 > 2 2 16.02091 0.5808985 Fold05.Rep1 > 3 2 12.59985 0.6008357 Fold03.Rep5 > 4 2 13.20069 0.6296245 Fold02.Rep3 > 5 2 12.43419 0.6560434 Fold04.Rep2 > 6 2 15.36510 0.5954177 Fold04.Rep5 > 7 2 12.70028 0.6894489 Fold03.Rep2 > 8 2 13.34882 0.6468300 Fold09.Rep3 > 9 2 14.80217 0.5575010 Fold08.Rep3 > 10 2 19.03705 0.4907630 Fold05.Rep4 > 11 2 14.26704 0.6579390 Fold10.Rep2 > 12 2 13.79060 0.5806663 Fold05.Rep3 > 13 2 14.83641 0.5918039 Fold05.Rep2 > 14 2 12.48721 0.7011439 Fold01.Rep3 > 15 2 14.98765 0.5866102 Fold07.Rep4 > 16 2 10.88100 0.7597167 Fold06.Rep1 > 17 2 13.60705 0.6321377 Fold08.Rep5 > 18 2 13.42618 0.6136031 Fold08.Rep4 > 19 2 13.26066 0.6784586 Fold07.Rep1 > 20 2 13.20623 0.6812341 Fold03.Rep3 > 21 2 18.54275 0.4404729 Fold08.Rep2 > 22 2 11.80312 0.7177681 Fold05.Rep5 > 23 2 18.56271 0.4661072 Fold03.Rep1 > 24 2 13.54879 0.5850439 Fold10.Rep3 > 25 2 14.10859 0.5994811 Fold06.Rep5 > 26 2 13.68329 0.6701091 Fold01.Rep5 > 27 2 16.12123 0.5401200 Fold10.Rep1 > 28 2 12.92250 0.6917220 Fold06.Rep3 > 29 2 12.94366 0.6400066 Fold06.Rep2 > 30 2 12.39889 0.6790578 Fold01.Rep2 > 31 2 13.48499 0.6759649 Fold01.Rep1 > 32 2 12.52938 0.6728476 Fold03.Rep4 > 33 2 16.43352 0.5795160 Fold09.Rep5 > 34 2 12.53991 0.6550694 Fold09.Rep4 > 35 2 12.78708 0.6304606 Fold08.Rep1 > 36 2 13.97559 0.6655688 Fold04.Rep3 > 37 2 15.31642 0.5124997 Fold09.Rep2 > 38 2 15.24194 0.5324943 Fold09.Rep1 > 39 2 12.90107 0.6318960 Fold04.Rep1 > 40 2 13.59574 0.6277869 Fold01.Rep4 > 41 2 19.73633 0.4154821 Fold07.Rep5 > 42 2 12.03759 0.6537381 Fold02.Rep5 > 43 2 15.47139 0.5597097 Fold02.Rep4 > 44 2 22.55060 0.3816672 Fold07.Rep3 > 45 2 14.57875 0.6269560 Fold07.Rep2 > 46 2 13.02385 0.6395148 Fold02.Rep2 > 47 2 13.81020 0.6116137 Fold02.Rep1 > 48 2 13.46100 0.6200828 Fold04.Rep4 > 49 2 13.95487 0.6709253 Fold10.Rep5 > 50 2 12.65981 0.6606435 Fold10.Rep4 > > Best, > Zhenjiang > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
