Hello David, Thank you for your patience with the amount of ignorance I am revealing of myself in my questions, and for your willingness to help. My replies to your e-mail are bellow, prefaced with "T:" for ease of navigation.
On Mon, Jul 20, 2009 at 9:02 PM, David Winsemius <dwinsem...@comcast.net>wrote: > > On Jul 20, 2009, at 5:22 AM, Tal Galili wrote: > > Hello David Winsemius and the rest of the R help group, >> >> David, I tried to answer your question to the best of my abilities, If I >> was unclear or still am leaving some things out, please help me in focusing >> my situation even further. here are my answers to the questions you posed: >> 1) Please define "better" - >> "better" is the one that is able to handle the questions at hand (marginal >> homogenity and symmetry) while giving meaningful results although the data >> is sometimes sparse (with Zeros in it) and the sample size is somewhat small >> (around 25 kids) >> > > Frankly, the phrases "marginal homogeneity" and "symmetry" are, for me > anyway, not particularly evocative of an interpretable sort of difference. I > try to express my findings in terms I think my audience may have some chance > of understanding: odds ratios or risk ratios or difference in mean effects > ... T: Understood and agreed, yet to the audience I am working with, OR and RR and the sorts are not of the field (as is the case in epidemiology), so I am not using them at the moment. > > 2) And now ... define "right" >> What I meant with "right" is "what test did each of these procedures just >> perform" and also "what can I learn from each of the P's if they where to >> pass the significance bar (of let's say .05)" >> > > 3) "Perhaps from the perspective of a statistically naive reviewer." >> Thank you for pointing to this being superficial, I would love for any >> help you could give in deepening my understanding. >> > > It appeared from context (which was snipped) that you thought one was > "better" because its p-value was lower. If the criterion by which you > choose one statistical test over another is whether or not it happens to > produce a signal p <0.05, then I think you are dredging rather than > analyzing. I think the question should be instead whether the test is the > most powerful for the particular hypothesis and data situation. T: I don't consider one test better then the other just because of having a P value that is smaller. I was meaning better in the same sense of a fisher test being "better" to a chi square test in cases of very few observations. The P's will be different, but the "better" is in the sense of "a better suited/valid test to the type of data we are handling". I was not referring to the power factor, but rather to the validity. > > > >> 4) "The problem I am trying to solve" is for the following situation: >> >> The data set: >> I am analyzing a data set with subjects (kids) listening to the same music >> two times (randomized, and on different times and so on), the condition of >> the experiment is a bit different the first time the kids listens (X=1) then >> the second time (X=2). >> And the response (Y) the kid is making for the experiment is recorded as >> an ordered number of three levels: -1, 0, 1 >> > > So you would certainly want a test that properly handles ordinal effects. I > am not sure that was clear at all from your earlier questions. Tests of > hypotheses regarding ordered alternatives are often more powerful than ones > that evaluate less specific alternatives. T: Thank you David, I agree with that. But it will require me a learning curve that I won't be able to climb before my upcoming deadline (but for my future work, it is definitely on my to-study list!) > > > >> The (statistical) question: did the difference in the experiment >> conditions yielded different rankings from the kids? and if so, was there a >> specific direction? >> e.g: did kids who by now (in part one of the experiment) answered mostly >> -1 and 0, now (in part two of the experiment) started answering more 0 and >> 1? Or, did kids who by now mostly answered 0 now started answering -1 and 1 >> ? and so on. >> >> Analyses approach: >> There are two basic ways to do this. >> 1) The first one is a Willcox test, to see if there was change in answers >> (Y) between the two situations (X=1, X=2) >> > > I am here puzzled. Is the Willcox test a well known one in your academic > domain? If it is I apologize for my lack of breadth in named tests. Or > could you be referring to what is invoked in R with wilcox.test()? I am > guessing from context that you might be asking about the Wilcoxon > signed-rank test for paired data situations. It would in fact address the > ordering of your paired outcomes, but all of the Wilcoxon tests are based on > the measures being from a continuous distribution and statistical validity > for your situation would be questionable. T: I was indeed referring to the "wilcox.test", sorry for not referring to it in that way. Regarding the statistical validity of using the wilcox.test with non continuous data (as is warned by the function itself) - I was thinking that we are talking about a less powerful test, but not a less valid one. Here is an example of what I had in mind: seed(1564) N <- 50 x1 <- runif(N) x2 <- runif(N) - runif(N) mean(x1) mean(x2) wilcox.test(x1,x2) rx1 <- round(x1) rx2 <- round(x2) wilcox.test(rx1,rx2) If there is a flaw in my argument, I would be happy to know of it. > > > I would think that a proportional odds model for ordinal repeated responses > would fit the data situation and the hypothesis of interest. > You may want to search out Laura Thompson's R/S companion to Agresti's > text. She has some worked examples. > > 2) The second one is to produce a 3 by 3 table, with the rows indicating >> what the kids answered to setting 1 of the experiment, and the columns >> indicating the kids answers to setting 2. >> Now the question is: >> was there marginal homogenity? if not, then that is an indicator that the >> general response to the experimental settings was different for the kids. >> > > Can you put into natural language what you will explain to your audience > once you determine the presence or absence of "marginal homogeneity"? T: "The answers the kids gave in the two experiments was different beyond the noise of the data" With regards, Tal > >> Challenges: >> 1) what about symmetry ? >> As Peter pointed out - you can easily check that the following two >> matrices have the same homogeneous margins, but only one is symmetric: >> 3 2 1 >> 2 3 2 >> 1 2 3 >> >> 3 1 2 >> 3 3 1 >> 0 3 3 >> >> And running the two tests we have yields very interesting results (and if >> someone has an explanation for them, they would be greatly appreciated): >> >> > tt <- as.table(t(matrix(c(30,10,20, >> + 30,30 ,10, >> + 0 ,30 ,30) >> + , ncol .... [TRUNCATED] >> > > The truncation is most unfortunate since it results our not seeing what > made these two calls different. T: The call is not relevant since the rest of the commands/outputs is bellow. > > >> > print(tt) >> A B C >> A 30 10 20 >> B 30 30 10 >> C 0 30 30 >> >> > mcnemar.test(tt) >> >> McNemar's Chi-squared test >> >> data: tt >> McNemar's chi-squared = 40, df = 3, p-value = 1.066e-08 >> >> >> > mh_test(tt) >> >> Asymptotic Marginal-Homogeneity Test >> >> data: response by groups (Var1, Var2) >> stratified by block >> chi-squared = 0, df = 2, p-value = 1 >> >> >> > tt <- as.table(t(matrix(c(30,10,20, >> + 30,30 ,10, >> + 1 ,30 ,30) >> + , ncol .... [TRUNCATED] >> >> > print(tt) >> A B C >> A 30 10 20 >> B 30 30 10 >> C 1 30 30 >> >> > mcnemar.test(tt) >> >> McNemar's Chi-squared test >> >> data: tt >> McNemar's chi-squared = 37.1905, df = 3, p-value = 4.194e-08 >> > > The truncation snipped off the likely sources of the differences. > > >> >> > mh_test(tt) >> >> Asymptotic Marginal-Homogeneity Test >> >> data: response by groups (Var1, Var2) >> stratified by block >> chi-squared = 0.0244, df = 2, p-value = 0.9879 >> >> >> 2) what about sparsity ? >> What is the correct way to handle a sparse tables that includes some Zeros >> in them? >> (is filling them with 1's, in cases where the mcnemar is resulting with >> NA's a legitimate strategy ?) >> >> >> >> David, thank you for the queries and the good intentions, >> I would be very happy for any help, directions, clerifications from you >> and from the other members of this wonderful discussion group. >> >> >> With much gratitude, >> Tal >> >> >> >> I hopes this helped clarify >> >> >> On Mon, Jul 20, 2009 at 3:20 AM, David Winsemius <dwinsem...@comcast.net> >> wrote: >> >> On Jul 19, 2009, at 6:09 PM, Tal Galili wrote: >> >> Hello Charles, >> Thank you for the detail reply. >> >> I am still left with the leading question which is: which test should I >> use >> when analyzing the 3 by 3 matrix I have? The mcnemar.test or the mh_test? >> Is the one necessarily better then the other? >> >> Please define "better". >> >> >> (for example for >> sparser matrices ?) >> >> That does not help. >> >> >> >> What about: >> mh_test(as.table(matrix(1:16,4))) >> It returns a very significant result: >> chi-squared = 11.4098, df = 3, p-value = 0.009704 >> >> Where as "mcnemar.test(matrix(1:16,4))", didn't: >> McNemar's chi-squared = 11.5495, df = 6, p-value = 0.0728 >> >> So which one is "right" ? >> >> And now ... define "right". >> >> >> (from the looks of it, the mh_test is doing much better) >> >> Perhaps from the perspective of a statistically naive reviewer. >> >> >> >> Should the strategy be to try and use both methods, and start digging when >> one doesn't sit well with the other? >> >> I am reminded of Jim Holtam's tag line: "What problem are you trying to >> solve?" >> >> >> >> >> Thanks, >> Tal >> >> >> >> On Sun, Jul 19, 2009 at 10:26 PM, Charles C. Berry <cbe...@tajo.ucsd.edu >> >wrote: >> >> On Sun, 19 Jul 2009, Tal Galili wrote: >> >> Hello David,Thank you for your answer. >> >> Do you know then what does the "mcnemar.test" do in the case of a 3*3 >> table >> ? >> >> >> print(mcnemar.test) >> >> will show you what it does. >> >> Because the results for the simple example I gave are rather different (P >> value of 0.053 VS 0.73) >> >> >> The test mcnemar.test() constructs is one of symmetry, which is equivalent >> to marginal homogenity in hierarchical log-linear models as I recall from >> Bishop, Fienberg, and Holland's 1975 opus on count data. >> >> Stuart-Maxwell uses the dispersion matrix of marginal difference. >> >> These are two different tests. I suspect that Stuart-Maxwell is less >> susceptible to continuity issues in very sparse tables, which may account >> for the difference you see here. >> >> >> >> In case the mcnemar can't really handle a 3*3 matrix (or more), shouldn't >> there be an error massage for this case? (if so, who should I turn to, in >> order to report this?) >> >> >> Well, the code is pretty straightforward and >> >> mcnemar.test(matrix(1:16,4)) >> >> returns 11.5495 which is correct. >> >> It looks like there is nothing to report. 3,1,5), ncol = 3)))) >> >> >> Chuck >> >> >> Thanks again, >> Tal >> >> >> >> >> >> On Sun, Jul 19, 2009 at 3:47 PM, David Freedman <3.14da...@gmail.com> >> wrote: >> >> >> There is a function mh_test in the coin package. >> >> library(coin) >> mh_test(tt) >> >> The documentation states, "The null hypothesis of independence of row and >> column totals is tested. The corresponding test for binary factors x and >> y >> is known as McNemar test. For larger tables, StuartÂ’s W0 statistic >> (Stuart, >> 1955, Agresti, 2002, page 422, also known as Stuart-Maxwell test) is >> computed." >> >> hth, david freedman >> >> >> Tal Galili wrote: >> >> >> Hello all, >> >> I wish to perform a mcnemar test for a 3 by 3 matrix. >> By running the slandered R command I am getting a result but I am not >> >> sure >> >> I >> am getting the correct one. >> Here is an example code: >> >> (tt <- as.table(t(matrix(c(1,4,1 , >> 0,5,5, >> 3,1,5), ncol = 3)))) >> mcnemar.test(tt, correct=T) >> #And I get: >> McNemar's Chi-squared test >> data: tt >> McNemar's chi-squared = 7.6667, df = 3, p-value = *0.05343* >> >> >> Now I was wondering if the test I just performed is the correct one. >> >> From looking at the Wikipedia article on mcnemar ( >> >> http://en.wikipedia.org/wiki/McNemar's_test), it is said that: >> "The Stuart-Maxwell >> test<http://ourworld.compuserve.com/homepages/jsuebersax/mcnemar.htm> >> is >> different generalization of the McNemar test, used for testing marginal >> homogeneity in a square table with more than two rows/columns" >> >> From searching for a Stuart-Maxwell >> >> test<http://ourworld.compuserve.com/homepages/jsuebersax/mcnemar.htm> >> in >> google, I found an algorithm here: >> >> >> >> http://www.m-hikari.com/ams/ams-password-2009/ams-password9-12-2009/abbasiAMS9-12-2009.pdf >> >> >> From running this algorithm I am getting a different P value, here is >> the >> >> (somewhat ugly) code I produced for this: >> get.d <- function(xx) >> { >> length1 <- dim(xx)[1] >> ret1 <- margin.table(xx,1) - margin.table(xx,2) >> return(ret1) >> } >> >> get.s <- function(xx) >> { >> the.s <- xx >> for( i in 1:dim(xx)[1]) >> { >> for(j in 1:dim(xx)[2]) >> { >> if(i == j) >> { >> the.s[i,j] <- margin.table(xx,1)[i] + margin.table(xx,2)[i] - >> 2*xx[i,i] >> } else { >> the.s[i,j] <- -(xx[i,j] + xx[j,i]) >> } >> } >> } >> return(the.s) >> } >> >> chi.statistic <- t(get.d(tt)[-3]) %*% solve(get.s(tt)[-3,-3]) %*% >> get.d(tt)[-3] >> paste("the P value:", pchisq(chi.statistic, 2)) >> >> #and the result was: >> "the P value: 0.268384371053358" >> >> >> >> So to summarize my questions: >> 1) can I use "mcnemar.test" for 3*3 (or more) tables ? >> 2) if so, what test is being performed ( >> Stuart-Maxwell< >> >> http://ourworld.compuserve.com/homepages/jsuebersax/mcnemar.htm>) >> >> ? >> 3) Do you have a recommended link to an explanation of the algorithm >> employed? >> >> >> Thanks, >> Tal >> >> >> > snipped various sigs >> > > >> >> My contact information: >> Tal Galili >> Phone number: 972-50-3373767 >> FaceBook: Tal Galili >> My Blogs: >> http://www.r-statistics.com/ >> http://www.talgalili.com >> http://www.biostatistics.co.il >> >> >> > David Winsemius, MD > Heritage Laboratories > West Hartford, CT > > -- ---------------------------------------------- My contact information: Tal Galili Phone number: 972-50-3373767 FaceBook: Tal Galili My Blogs: http://www.r-statistics.com/ http://www.talgalili.com http://www.biostatistics.co.il [[alternative HTML version deleted]]
______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
