Although interesting, Dave, this doesn't fit my problem. I want to measure the percentage agreement corrected for chance using an extension of kappa. The example data presented in the paper you linked to is considering an ordinal measure (ranked preference), whereas I'm looking to measure correlation between a nominal measure (agreement between non-ordered categories).
The paper by Kraemer is cited over 100 times in Google Scholar, mostly in the health sciences, so I'm surprised it's not implemented in R. But I suppose this is a niche problem (multi-response version of kappa), or that there is some other extension to kappa, maybe in the social sciences, that I'm not aware of. Cheers, Luk Arbuckle On Wed, Feb 1, 2012 at 17:13, David Winsemius wrote: > Searching on "multiple raters attributes" at the same site brings up > > http://finzi.psych.upenn.edu/R/library/smacof/doc/smacof.pdf (by Jan De > Leeuw) > > .... Which has as one example multiple raters scoring "different breads". > > > > On Feb 1, 2012, at 4:41 PM, Luk Arbuckle wrote: > > Thanks David, but those are not "multi-response" versions of the kappa. > Extensions to multiple raters are common and well known. I am hoping > someone familiar with multiple response extensions of kappa might see my > post and be able to help. > > As I said, my search on cran has failed. I tried all the expected > keywords, and looked through several kappa functions, but I don't see any > that deal with the multi-response case as I've described it. Either it > isn't available in R, or I'm looking in the wrong place. > > I did not intentionally double post, nor try to deceive your efforts to > block double posting. I am not receiving my posts, contrary to my > settings, so I rewrote the first one. I thought maybe it was blocked the > first time because someone thought it wasn't an R question, so I changed > the subject. > > Cheers, > > Luk Arbuckle > > On Wed, Feb 1, 2012 at 16:25, David Winsemius wrote: > >> >> On Feb 1, 2012, at 3:13 PM, Luk Arbuckle wrote: >> >> I'm very sorry for double posting! My r-help setting Receive your own >>> posts to the list? is set to Yes, and Mail delivery is Enabled. Yet I >>> did >>> not get a copy of my post (this message is a reply from my sent mail). I >>> only learned of the double posting when I found it copied in an r-help >>> archive. Again, my apologies. >>> >> >> I actually had a chance to prevent that second posting. It looked >> familiar when viewed in the moderation queue and I took a quick look at >> what was in my inbox but since you used a different subject line my search >> failed. >> >> Speaking of searching ... you are asked in the Posting Guide (that no one >> reads) to post the specifics of your own efforts. My first search with >> "multi-rater kappa" failed. My second search is here: >> >> http://search.r-project.org/**cgi-bin/namazu.cgi?query=** >> multiple+raters+kappa&max=100&**result=normal&sort=score&** >> idxname=functions&idxname=**Rhelp08&idxname=Rhelp10&**idxname=Rhelp02<http://search.r-project.org/cgi-bin/namazu.cgi?query=multiple+raters+kappa&max=100&result=normal&sort=score&idxname=functions&idxname=Rhelp08&idxname=Rhelp10&idxname=Rhelp02> >> >> Having gotten more than one apparently on-target result with relatively >> minor effort, I see no point in my expending even more time. >> >> -- >> David. >> >> >> >> >> >>> Luk Arbuckle >>> >>> On Wed, Feb 1, 2012 at 13:47, Luk Arbuckle wrote: >>> >>> I'm looking for a function in R that extends kappa to multiple raters >>>> when >>>> there is more than one response per subject. For example, say a group >>>> of >>>> doctors have to assign diseases to patients. Each patient will be >>>> assigned >>>> one to many diseases, and the number of doctors assigning diseases to >>>> any >>>> one patient will be two to many. >>>> >>>> Here's an extremely simple example of the type of data I might have (two >>>> patients, three doctors, five diagnoses, and doctor 'x' assigns multiple >>>> diagnoses per patient): >>>> >>>> pat<-c('a','a','a','b','b','b'**,'b') >>>> doc<-c('x','x','y','x','x','y'**,'z') >>>> dx<-c('1','2','2','3','4','4',**'5') >>>> df<-data.frame(pat=pat,doc=**doc,dx=dx) >>>> df >>>> >>>> I found a paper that can address this [1], although I can't find it >>>> referenced on cran. What I have found on cran doesn't address the >>>> multi-response case. I don't want to reinvent the wheel if this has >>>> already been implemented in R, and I'm still new to the language so I'm >>>> not >>>> a fast coder. If anyone can help it would be greatly appreciated. >>>> >>>> Cheers! >>>> >>>> Luk Arbuckle >>>> >>>> [1] Kraemer, Helena. Extension of the Kappa Coefficient. Biometrics >>>> 36, >>>> no. 2 (1980). >>>> http://dx.doi.org/10.2307/**2529972<http://dx.doi.org/10.2307/2529972> >>>> . >>>> >>>> >>> [[alternative HTML version deleted]] >>> >>> ______________________________**________________ >>> R-help@r-project.org mailing list >>> https://stat.ethz.ch/mailman/**listinfo/r-help<https://stat.ethz.ch/mailman/listinfo/r-help> >>> PLEASE do read the posting guide http://www.R-project.org/** >>> posting-guide.html <http://www.R-project.org/posting-guide.html> >>> and provide commented, minimal, self-contained, reproducible code. >>> >> >> David Winsemius, MD >> West Hartford, CT >> >> > > David Winsemius, MD > West Hartford, CT > > [[alternative HTML version deleted]]
______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
