Probability of type 2 error

[UA Munira]

Hi all,

I had run a fixed effect regression for my panel data and found the coefficients of my IVs insignificant. I want to determine the probability of type 2 error. Can anyone tell me how to do that in Stata?

Thank you in advance.

Umme

[John Mullahy]

I would suggest looking at this classic paper by Don Andrews. He suggests several empirical strategies to consider when an analyst fails to reject a null hypothesis. Moreover, the statistics that Andrews proposes are quite simple to compute.
https://www.jstor.org/stable/1913623

[UA Munira]

Thank you Sir

[Steve Samuels]

See this blog, which argues that such post-hoc calculations are not helpful. (I agree.)

[Roman Mostazir]

Further to Steve's suggestion here is another resource on why post-hoc power is fundamentally flawed:

The Abuse of Power, John M Hoenig & Dennis M Heisey, The American Statistician, 55:1, 19-24, DOI: 10.1198/000313001300339897

[Bruce Weaver]

Here are a couple of my favorite resources for explaining why post hoc power (as it is typically computed) is really nothing more than a re-scaling of the p-value. (I.e., if p < .05, post hoc power is adequate; if p > .05, post hoc power is inadequate.

• http://homepage.stat.uiowa.edu/~rlen...2badHabits.pdf
• https://onlinelibrary.wiley.com/doi/...9.1997.96102.x

Steve, I took issue with this statement in the blog you linked to (emphasis added):

Another common albeit less perilous misunderstanding is that calculating post-hoc (or ‘retrospective )’ power can explain why an analysis did not achieve significance. Besides proving a severe bias of the researcher towards rejecting the Null hypothesis (‘There must be another reason for not obtaining a significant result than that the hypothesis is incorrect!), this is the equivalent of a statistical tautology. Of course the study was not powerful enough, this is why the result was not significant!

The flip side of that statement would seem to be if one is able to reject H₀, one obviously has sufficient power. I tackled that claim in this conference presentation, and attempted to show via simulation that it is wrong. Likewise, one could fail to reject H₀ in a particular study, despite having power = 80%, 90%, or even 95%. That would not prove power was inadequate.

Cheers,
Bruce