Hello,
I'm attempting to evaluate the accuracy of the probability predictions
for my model. As previously discussed here, the AUC is not a good
measure as I'm not concerned with classification accuracy but
probability accurcy.
It was suggested to me that the loess function would be a good measure
to look at.
I can see some libraries (Design) will plot the loess function as a
curve of the resulting model. That's nice to look at, but I need a way
to feed in a test set of data and measure the accuracy of my predicted
probabilities.
I read the help page for loess and it looks like a learner of its own. I
already have a model trained with SVM and can apply the test data for a
result of output that is predicted probabilities and true labels. How
can I feed this to the loess function to both see a curve and get the
mean absolute error among other measures.
Ultimately what I am trying to do is develop a model with high accuracy
of predicted probabilities. I'm testing many different learning
functions, kernels, parameters, etc. I'mm looking of a single
"performance measure" that summarized the probability accuracy for a
given model. That way I can track my experiment's results and pick the
best one.
Any suggestions along these lines would be greatly appreciated.
Thank You,
-Noah
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
