Puzzling Post Hoc Power Calculation

[Dimitriy V. Masterov]

I was trying to show a colleague that post hoc power calculations are nonsensical with the following example:

. /* Congruent Results */
.
. power twoproportions 0.146, n1(254426) n2(255237) alpha(.05) power(0.8) test(chi2) effect(ratio)

Performing iteration ...

Estimated experimental-group proportion for a two-sample proportions test
Pearson's chi-squared test
H0: p2 = p1  versus  Ha: p2 != p1; p2 > p1

Study parameters:

        alpha =    0.0500
        power =    0.8000
            N =   509,663
           N1 =   254,426
           N2 =   255,237
        N2/N1 =    1.0032
           p1 =    0.1460

Estimated effect size and experimental-group proportion:

        delta =    1.0191  (ratio)
           p2 =    0.1488

. display "MDE (in %)) " 100*(r(p2) - r(p1))/r(p1)
MDE (in %)) 1.9056807

.
. prtesti 254426 0.146 255237 .1488, level(95) // use MDE from above

Two-sample test of proportions                     x: Number of obs =   254426
                                                   y: Number of obs =   255237
------------------------------------------------------------------------------
             |       Mean   Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
           x |       .146      .0007                      .1446279    .1473721
           y |      .1488   .0007044                      .1474193    .1501807
-------------+----------------------------------------------------------------
        diff |     -.0028   .0009931                     -.0047465   -.0008535
             |  under H0:   .0009931    -2.82   0.005
------------------------------------------------------------------------------
        diff = prop(x) - prop(y)                                  z =  -2.8193
    H0: diff = 0

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(Z < z) = 0.0024         Pr(|Z| > |z|) = 0.0048          Pr(Z > z) = 0.9976

. display "Estimated ratio = "  100*(r(P2) - r(P1))/r(P1)
Estimated ratio = 1.9178082

This worked exactly as I anticipated.

But then he gave me a counterexample, where a statistically significant difference is smaller than the MDE:

. /* Incongruent Results From Experiments */

. prtesti 254426 0.146 255237 .148, level(95)

Two-sample test of proportions                     x: Number of obs =   254426
                                                   y: Number of obs =   255237
------------------------------------------------------------------------------
             |       Mean   Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
           x |       .146      .0007                      .1446279    .1473721
           y |       .148   .0007029                      .1466224    .1493776
-------------+----------------------------------------------------------------
        diff |      -.002    .000992                     -.0039443   -.0000557
             |  under H0:    .000992    -2.02   0.044
------------------------------------------------------------------------------
        diff = prop(x) - prop(y)                                  z =  -2.0161
    H0: diff = 0

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(Z < z) = 0.0219         Pr(|Z| > |z|) = 0.0438          Pr(Z > z) = 0.9781

. display "Estimated ratio = "  100*(r(P2) - r(P1))/r(P1) // SS diff < MDE
Estimated ratio = 1.369863

I am having a hard time wrapping my brain around this. Why can I reject the null when it is smaller than the MDE from the power calculation?

[Dimitriy V. Masterov]

Here's a slightly tweaked version of the code. I would love to understand why the MDE from -power twoproportions- is a bit conservative here.

cls
clear

/* need to swap T and C for prtesti */

/* Congruent Results */
power twoproportions 0.146, n1(254426) n2(255237) alpha(.05) power(0.8) test(chi2) effect(diff)
display "MDE (in %)) " 100*(r(p2) - r(p1))/r(p1)
prtesti 255237 .1488 254426 0.146 , level(95) // use MDE from above
display "Estimated ratio = "  100*(r(P1) - r(P2))/r(P2)

/* Incongruent Results From Experiments */
prtesti 255237 0.148 254426 0.146 , level(95) 
display "Estimated ratio = "  100*(r(P1) - r(P2))/r(P2) // SS diff < MDE

[Dimitriy V. Masterov]

Posted and answered by myself here. Rookie mistake on my end thinking of power as fixed.