Hi Max,
Thanks for your help. In the case of randomForest, the keyword
keep.inbag=TRUE in the train function provokes the return the information
about which data rows were in-bag in each tree. That should provide the
required info to re-compute the OOB error for any given alternative error
definition.
It is somewhat convoluted and likely to differ between bagging algorithm
implementations, so it doesn't surprise me that this isn't supported, I
just wanted to check I hadn't missed anything before re-inventing the wheel.
Thanks again,
Matt
On Sat, Apr 14, 2012 at 2:53 AM, Max Kuhn <mxk...@gmail.com> wrote:
> Matt,
>
> > I've been using a custom summary function to optimise regression model
> > methods using the caret package. This has worked smoothly. I've been
> using
> > the default bootstrapping resampling method. For bagging models
> > (specifically randomForest in this case) caret can, in theory, uses the
> > out-of-bag (oob) error estimate from the model instead of resampling,
> which
> > (in theory) is largely redundant for such models. Since they take a while
> > to build in the first place, it really slows things down when estimating
> > performance using boostrap.
> >
> > I can successfully run either using the oob 'resampling method' with the
> > default RMSE optimisation, or run using bootstrap and my custom
> > summaryFunction as the thing to optimise, but they don't work together.
> If
> > I try and use oob and supply a summaryFunction caret throws an error
> saying
> > it can't find the relevant metric.
> >
> > Now, if caret is simply polling the randomForest object for the stored
> oob
> > error I can understand this limitation
>
> That is exactly what it does. See caret:::rfStats (not a public function)
>
> train() was written to be fairly general and this level of control
> would be very difficult to implement, especially since each model that
> does some type of bagging uses different internal structures etc.
>
> > but in the case of randomForest
> > (and probably other bagging methods?) the training function can be asked
> to
> > return information about the individual tree predictions and whether data
> > points were oob in each case. With this information you can reconstruct
> an
> > oob 'error' using whatever function you choose to target for
> optimisation.
> > As far as I can tell, caret is not doing this and I can't see anywhere
> that
> > it can be coerced to do so.
>
> It will not be able to do this. I'm not sure that you can either.
> randomForest() will return the individual forests and
> predict.randomForest() can return the per-tree results but I don't
> know if it saves the indices that tell you which bootstrap samples
> contained which training set points. Perhaps Andy would know.
>
> > Have I missed something? Can anyone suggest how this could be achieved?
> It
> > wouldn't be *that* hard to code up something that essentially operates in
> > the same way as caret.train but can handle this feature for bagging
> models,
> > but if it is already there and I've missed something please let me know.
>
> Well, everything is easy for the person not doing it =]
>
> If you save the proximity measures, you might gain the sampling
> indices. WIth these, you would use predict.randomForest(...,
> predict.all=TRUE) to get the individual predictions.
>
> Max
>
[[alternative HTML version deleted]]
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
