Power and sample size-Alternative hypothesis of a Individual signficance test

[Jorge Aguayo]

I am trying to conduct a power test in order to obtain the power given a sample of 5000 obs on a regression of life satisfaction over lottery. The problem i have is when i try to define my alternative hypothesis. Given the fact that I am making a individual signficance test, my null hypothesis must be =0 and the alternative one should be <>0. The issue is that the command power does not let me use <>0 as a alternative hypothesis.
Here is my code:

reg lifesat_overall lottery
test lottery
power oneslope 0 <>0, n(5000)

Any suggestions to solve this problem with the specification of the alternative hypothesis?
Thank you in advance for the help.

[Clyde Schechter]

You must pick a specific numerical value for the alternate hypothesis in a power calculation; compound hypotheses are not allowed. So you will have to determine what degree of difference in expected values between the lottery and non-lottery groups will be one that is large enough that it is important to detect in your study. That is a matter of judgment and knowledge of the area you are working in.

That's not some arbitrary restriction of Stata's power command. It's the mathematics of it. If you think about it, if you really wanted to have some pre-specified power to detect any non-zero difference, no matter how small, no finite sample would be large enough. You would need an infinitely large sample. Or, from the perspective of starting with a fixed sample size of 5000, you will have higher power to detect larger effect sizes, but as the effect size gets close to zero, the power will get smaller and smaller, ultimately settling to its theoretical minimum at the specified Type I error rate (which, since you don't specify one in your command, defaults to 0.05). The only way you could come up with an overall power to detect any non-zero effect size would be to then somehow average over all the possible effect sizes. But that means you're imposing a prior distribution on the effect size; which means you're doing Bayesian statistics; which means that power is really not relevant.

[Jorge Aguayo]

Thanks for your answer Clyde, in that sense do you have any clue of how to conduct a power test for a individual significance test? Probably a code example or so, this due to the fact that i can't formulate the alternative hypothesis as i did before.

[Clyde Schechter]

Showing you coded examples won't help you with your basic problem. Your problem is that you need to formulate your alternative hypothesis conceptually. Once you've done that, writing out the correct command for -power- will be very easy.

As I don't know what you are studying here, and even if I did, I probably wouldn't know enough about that subject to give you any useful advice, let me just walk you through how I think about selecting an alternate hypothesis. Let's imagine we are dealing with a population of people who have disease that limits how far they can walk before they have to stop and rest due to pain, fatigue, or shortness of breath. Let's say that on average, they can walk about 20 meters before they have to stop. Suppose we are going to do a randomized trial of a new treatment that might enable them to walk farther.

So to determine the alternate hypothesis here we have to come up with some additional walking distance that people would consider to be a meaningful improvement in their ability to function. At one extreme, if the drug increased their walking distance to 21 meters, I think most people would agree that this is not large enough to matter. Nobody would care about that kind of improvement. Nobody would take any risks to get it, and nobody would pay much for a treatment if that were all it accomplished. In fact, people might even laugh at the very idea. At another extreme, say that the drug improved walking distance all the way to 1000 meters. That's a huge difference. That makes it possible perhaps to walk from home to a nearby grocery store. It means getting around within your own home without stopping at all. It means going out to visit a nearby friend is a possibility. You could walk to a convenient bus stop or train station. Your world really opens up.

So if you designed your study with such low power that you couldn't detect a 1,000 meter difference, you'd be doing yourself and the study participants a disservice, because it could easily come out with negative results even if the drug were really life-changing. You certainly need to power your study for that effect.

Now, those are extreme cases. Somewhere in between, there is some distance that is large enough that "it matters," even though it might not be "a game changer." In my mind, increasing the walking distance to something like 100 meters begins to feel like it matters. It doesn't give you a whole new world, but it makes it possible to walk a long block in a city. That's meaningful because in a city you probably have to stop and wait for a traffic light at the end of each block anyway--which also gives you time to rest up. So that makes some things possible that weren't when only could only go 20 meters. So I might choose my alternate hypothesis to be a difference of 80 (= 100 - 20) meters.

Now other reasonable people might disagree with that judgment. And in reality, this is the kind of decision that I usually make in collaboration with other people who might have a different perspective on what is a meaningful distance to be able to walk. After discussing it with others, I might move that number up or down a bit--but I'm pretty confident it would end up not too far from what I'm suggesting here.

So that's the thought process. Now you have to apply that to your outcome, and what change or difference in outcome would be meaningful in a practical real-world sense. There is no code for that. There are no formulas for that. It's a matter of judgment about real-world importance.