Hi Claudia, I am a novice to statistics and R.The information you have provided is really very helpful.Thank you.
I was using the neural network in SPSS in which we can figure out the maximum accuracy (in %) that is possible with the model for the current dataset.It also gives the estimated accuracy(in %) after executing the model with present parameters.I was trying to figure out something similar using nnet in R. Is there any way to do this currently in R? I will try playing around with ROCR package to figure out the performance measures for classification.I guess your tool would be of great help in achieving this for continous values.(like when we use nnet.predict(type = "raw") for continuous predictions).It will be my pleasure to be your test user. Thanks and Regards, Raji On Thu, Oct 21, 2010 at 4:46 PM, Claudia Beleites [via R] < ml-node+3005405-1812510136-199...@n4.nabble.com<ml-node%2b3005405-1812510136-199...@n4.nabble.com> > wrote: > Raji, > > you first need to tell us what kind of accuracy you mean. > > The term accuracy has different meanings in different areas of science. > However, in classification it usually refers to something along the line > number > of correctly predicted samples / total number of samples (possibly weighted > > according to the number of samples per class). > > Procedures: > You can calculate that for different "types" of test samples: > > - prediction of the training samples gives you a goodness of fit. If you > have > (too) many variates you have in your model, this measure is close to > useless. > Useless, because most people are not interested in the goodness of fit > anyways > but want to know the performance for new samples. > > - prediction of unknown (statistically independent) samples: this is > usually > what is of interest. You may use resampling schemes (out-of-bootstrap & > Co., > (iterated) cross validation). > There's package boot (though I never used it as it does not properly fit my > data) > > - Resampling schemes usually cannot tell you the performance for /future/ > samples: for that you need a test set that is acquired later (and as close > as > possible to the real data to predict). > You need to do this if you want to take into account things like instrument > > drift etc. > > There's tons of literature around, what to do depends somewhat on your > field. I > can point you to chemometric literature. > > Calculating: > - package ROCR calculates all sorts of classifier performance measures for > binary classification > . > - I'm developing a package that gives performance measures directly for > continuous predictions (such as predict.mulitnom with type = "probs"). You > are > welcome to be a test user: just let me know if you want to try it out. > > > Hope that helps, > > Claudia > > > > > On 10/21/2010 05:37 AM, Raji wrote: > > > > Hi R-Helpers , am working on nnet package.Multinom() has an option for > > finding the goodness of fit by giving the AIC value. Does nnet also gives > > > some value to determine the accuracy. If not, can you guide me with some > > procedure to figure out the accuracy/goodness of fit of nnet model? > > > > Thanks in advance. > > > -- > Claudia Beleites > Dipartimento dei Materiali e delle Risorse Naturali > Università degli Studi di Trieste > Via Alfonso Valerio 6/a > I-34127 Trieste > > phone: +39 0 40 5 58-37 68 > email: [hidden email]<http://user/SendEmail.jtp?type=node&node=3005405&i=0> > > ______________________________________________ > [hidden email] <http://user/SendEmail.jtp?type=node&node=3005405&i=1>mailing > list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > > > ------------------------------ > View message @ > http://r.789695.n4.nabble.com/Accuracy-Goodness-of-fit-of-nnet-tp3004978p3005405.html > To unsubscribe from Accuracy/Goodness of fit of nnet, click > here<http://r.789695.n4.nabble.com/template/TplServlet.jtp?tpl=unsubscribe_by_code&node=3004978&code=cmFqaS5zYW5rYXJhbkBnbWFpbC5jb218MzAwNDk3OHw2MTcwMzEyNjY=>. > > > -- View this message in context: http://r.789695.n4.nabble.com/Accuracy-Goodness-of-fit-of-nnet-tp3004978p3005446.html Sent from the R help mailing list archive at Nabble.com. [[alternative HTML version deleted]]
______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
