On Thu, 23 Mar 2000 16:01:13 -0500, Rich Ulrich <[EMAIL PROTECTED]>
wrote:
>Now let me jump on Andy!
>
>
>On Thu, 23 Mar 2000 17:51:04 GMT, [EMAIL PROTECTED] (Andy Gilpin)
>wrote:
> < snip, problem; comment >
>>
>> Still, it seems to me that, other things equal, (a) measuring data
>> costs a researcher something, and (b) there are clear diminishing
>> returns in terms of increased power. Consider the following estimated
>> sample sizes for an independent-groups t-test with 2-tailed alpha=.05
>> and a moderate effect size (in Cohen's terms) of d=.5.
>...
>
>Oh, Andy, this is such a naive *scaling* conclusion. How can you
>regard "power" as a metric that ought to be equal-interval?
Wait a minute, Rich! I'll admit I'm often naive, but here I thought I
was saying "Don't forget that power is NOT so simply related to N."
I do think that my students sometimes fail to appreciate this,
however, perhaps since many texts represent increased N as the chief
mechanism by which to increase power. I think that a quick
examination of tables of critical values of the t distribution will
help them see things more accurately. But my impression is that many
low-level texts seem to discuss increased N and increased alpha as the
best ways to increase power--ignoring factors such as increased
reliability in measuring the dependent variable.
> By my way
>of thinking, extra power gets cheaper and cheaper, as the magnitude of
>N increases. In your table, I am looking at successive doublings of
>N, and Odds and Odds-ratios for power:
>
>The power at .80, in terms of an "Odds",
>is 0,80/0.20 or 4:1, with N=128;
>it is 39:1 when the N is doubled, to N=248;
>it is 9999:1 if N is doubled again, to N=518.
>
>The first doubling corresponds to an Odds ratio, for your increase in
>power, of 10:1, which might seem sizable, but the second doubling
>provides an OR of 250:1. That is *one* way to say that I disagree.
>
One way to look at it. But I'd be buying lottery tickets if I could
get odds as 'low' as 4:1, and in research I think many researchers
would be satisfied with power of .8 (which, like it or not, seems to
be emerging as a convention at least in the field of psychology).
Maybe not a really defensible choice, but that's a different issue.
>Andy>
>>...
>The ground gained by quadrupling the number of cases is always,
>basically for a t-test, the reduction of the width of the Confidence
>Interval by half.
>
>Do you want a smaller CI? Do you need a smaller CI?
>
>"Barely not-overlapping zero" is what you have for the usual rejection
>of the null hypothesis in psychology. That's not too bad with tiny N,
>because it works out neatly: the 5% rejection when d=1.0, implying
>"d>0.0" may be equivent to a 15% test (say) that "d>0.5". But to
>set up a CI > 0.5 with a large sample is going to assume that there
>is a *huge* power for detecting an effect that is merely " > 0.0"
>
>TO put it another way, you can't assume that the only goal is to
>detect an effect as being non-zero. In fact, I think it is pretty
>useless to cite 95% CI's as an "effect" when the test is barely at 5%;
>the range is just LARGE.
>
Here I think Rich makes an important point, but it supposes the need
to shift paradigms in experimentation. If researchers typically
conceptualized two-group experiments in terms of attempts to estimate
a confidence interval--as they probably should do, but sadly mostly
don't--the interval would continue to shrink with additional N. But
most actual researchers appear to want to test the null hypothesis
that the means are identical, and for that purpose, once we have
adequate power to reject that hypothesis most of the time (when it's
false), adding more cases is only going to inflate the economic cost
of the research without being likely to change the decision.
It's not that really large sample sizes don't improve estimation of
error--but that they don't necessarily change the decision regarding
the hypothesis most often entertained in practice, viz., equal means.
>--
>Rich Ulrich, [EMAIL PROTECTED]
>http://www.pitt.edu/~wpilib/index.html
----------------
Andy Gilpin, Dept. of Psychology Internet: [EMAIL PROTECTED]
Univ. of Northern Iowa, Cedar Falls, Phone: (319) 273-6104
IA (US) 50614-0505 Fax: (319) 273-6188
===========================================================================
This list is open to everyone. Occasionally, less thoughtful
people send inappropriate messages. Please DO NOT COMPLAIN TO
THE POSTMASTER about these messages because the postmaster has no
way of controlling them, and excessive complaints will result in
termination of the list.
For information about this list, including information about the
problem of inappropriate messages and information about how to
unsubscribe, please see the web page at
http://jse.stat.ncsu.edu/
===========================================================================
