Hi Rolf, Thank you for clarifying. After reading the help page over more carefully, I realize that I misunderstood what chisq.test. was doing. I thought it was using simulation whenever the expected frequencies were given using the "p" argument, whereas it used the asymptotic chi-squared distribution of the test statistic when "p"was not given. I now see that, based on the help page, the asymptotic distribution is always used, unless simulate.p.value is TRUE.
Mike On Dec 25, 2011, at 3:06 AM, Rolf Turner wrote: > > When the p-value is calculated via simulation the test is not, strictly > speaking a ``chi-squared test''. That is, it is *not* assumed that the > distribution of the test statistic, under the null hypothesis, is a > chi-squared > distribution. Instead an empirical distribution derived from simulating > samples under the null hypothesis is used. The *same statistic* is > calculated, > but the chi-squared distribution is never invoked. The simulated p-value > facility is provided so that tests can be conducted even when the requirements > for the validity of the chi-squared (null) distribution are not met. > > The test, whether you want to call it a chi-squared test or not, is a > perfectly > valid goodness of fit test. (Modulo the caveat that I previously mentioned, > i.e. that it is required that there be no ties amongst the values of the test > statistics. Which is a mild worry, but has nothing to do with continuity > corrections > which are meaningless in this context.) > > cheers, > > Rolf Turner > > On 25/12/11 08:26, Michael Fuller wrote: >> On Dec 22, 2011, at 8:56 PM, Rolf Turner wrote: >> >>> On 20/12/11 10:24, Michael Fuller wrote: >>>> TOPIC >>>> My question regards the philosophy behind how R implements corrections to >>>> chi-square statistical tests. At least in recent versions (I'm using >>>> 2.13.1 (2011-07-08) on OSX 10.6.8.), the chisq.test function applies the >>>> Yates continuity correction for 2 by 2 contingency tables. But when used >>>> as a goodness of fit test (GoF, aka likelihood ratio test), chisq.test >>>> does not appear to implement any corrections for widely recognized >>>> problems, such as small sample size, non-uniform expected frequencies, and >>>> one D.F. >>>> >>>>> From the help page: >>>> "In the goodness-of-fit case simulation is done by random sampling from >>>> the discrete distribution specified by p, each sample being of size n = >>>> sum(x)." >>>> >>>> Is the thinking that random sampling completely obviates the need for >>>> corrections? >>> Yes. >>>> Wouldn't the same statistical issues still apply >>> No. >>>> (e.g. poor continuity approximation with one D.F., >>> There are no degrees of freedom involved. There is no continuity >>> involved. >>> The observed test statistics (say "Stat") is compared with a number of >>> test statistics, Stat_1, ..., Stat_N, calculated from data sets >>> simulated under >>> the null hypothesis. If the null is true, then Stat and Stat_1, ...., >>> Stat_N are >>> all of ``equal status''. If there are m values of the Stat_i which are >>> greater >>> than Stat, then the ``probability of observing, under the null >>> hypothesis, >>> data as extreme as, or more extreme than, what you actually observed'' >>> is the probability of randomly selecting one of a specified set of m+1 >>> ``slots'' >>> out of a total of N+1 slots (where each slot has probability 1/(N+1)). >> But if the test is truly a chi-square Goodness of Fit (GoF) test, then: >> >> (1) the test compares the Stat to a chi-square distribution with (k-1) >> degrees of >> freedom, for k frequency categories. >> >> (2) the shape of the distribution depends on the degrees of freedom >> >> (3) continuity is an issue, because the values for Stat are discrete, whereas >> the chi-square distribution is continuous. Therefore the p-value is not >> exact. >> >> If I understand your description, the function generates a probability >> distribution by sampling a set of values with equal probability. The >> probability of >> any value in the final distribution depends on its frequency in the sample >> set. This >> is a standard Monte Carlo method, but the PDF it generates is not chi-square >> (the >> resulting distribution depends upon the distribution of the sample set). >> Clearly, >> using a permutation method such as this to determine the statistical >> significance of >> a given test statistic is non-parametric and in particular, not chi-square. >> >> If what you say is true, then the chisq.test function in R does not >> implement a true GoF >> test. I don't understand why the GoF test is not computed the standard way. >> It should >> compute the chi-square statistic, using the values of p to generate expected >> frequencies. And it should use a continuity correction when degrees of >> freedom = 1. >> >> Thank you for taking the time to respond to my message. >> >> Cheers, >> Mike >> >>> Thus the p-value is (exactly) equal to (m+1)/(N+1). >>> >>> The only restriction is that there be no ties amongst the values of Stat >>> and Stat_1, ..., Stat_N. There being ties is of fairly low probability, >>> but is >>> not of zero probability --- since there is a finite number of possible >>> samples >>> and hence of statistic values. So this restriction is a mild worry. >>> >>> However a ``continuity correction'' would be of no help whatsoever. >>>> problems with non-uniform expected frequencies, etc) with random sampling? >>> Don't understand what you mean by this. >>> >>> cheers, >>> >>> Rolf Turner >> ==================== >> Michael M. Fuller, Ph.D. >> Department of Biology >> University of New Mexico >> Albuquerque, NM >> EMAIL: mmful...@unm.edu >> WEB: biology.unm.edu/mmfuller >> >> >> >> >> > ==================== Michael M. Fuller, Ph.D. Department of Biology University of New Mexico Albuquerque, NM EMAIL: mmful...@unm.edu WEB: biology.unm.edu/mmfuller ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
