Is the percentage significantly higher than another?

[Shan Zhao]

when I want to compare the difference of two percentages, I input the value in the code:
prtesti 1196 273 1549 313, count

the result shows as below:
Two-sample test of proportions x: Number of obs = 1196
y: Number of obs = 1549
------------------------------------------------------------------------------
| Mean Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
x | .2282609 .0121363 .2044742 .2520475
y | .2020658 .0102024 .1820694 .2220623
-------------+----------------------------------------------------------------
diff | .026195 .0158549 -.0048801 .0572701
| under Ho: .0157729 1.66 0.097
------------------------------------------------------------------------------
diff = prop(x) - prop(y) z = 1.6608
Ho: diff = 0

Ha: diff < 0 Ha: diff != 0 Ha: diff > 0
Pr(Z < z) = 0.9516 Pr(|Z| > |z|) = 0.0968 Pr(Z > z) = 0.0484

P value is 0.097, >0.05, which means we don't reject Ho. That means there's no difference between two percentages.
But Pr(Z > z) = 0.0484<0.05, which seems that we reject Ho. That means percentage of x is higher than percentage of y.


Please help correct it! Thanks

[Maarten Buis]

Two comments:

First, you first decide your null-hypothesis, and than do the test. it is rather fishy if you do lots of tests on different null-hypotheses, and only choose the "significant" ones. What you did is thus rather fishy.

Second, do you really believe that there is a meaningfull difference between 0.048 and 0.052? Remember, p-values are also estimates and thus uncertain. What is so special about 5 percent? Why not put the cut-off at 4, or 6 or \(1-\frac{e}{\pi}\)? These are all equally arbitrary choices.

[Shan Zhao]

Thanks. Maybe it's a good idea to put a new cut-off instead.

[Maarten Buis]

No, no, no, no. You do not change the cut-off such that you get the conclusions you desired. You choose it before the analysis, and afterward you can never, ever, ever change it again. You have chosen the cut-off value .05 (and you have made this choice public on this list), so that is the value you will have to live with. So before your analysis you decide on your null-hypothesis and the significance level. At that point they are set in stone. You cannot, under any circumstance, change it. If that means that your results are not significant, then that is your finding. Changing either to get "significant" results is academic fraud.

My point, and that of a lot of statisticians, is that the idea of a cut-off and significance is wrong. See https://www.tandfonline.com/toc/utas20/73/sup1