Fisher Exact Test: p-value discrepency stata vs formula

[Lennart Rehm]

Hey everyone,
this forum has been a big help with prior stata problems I had. This is the first time I could not find an answer to my question on a different post. I have two datasets and conducted a Fisher Exact Test using stata 18 with the following code:

cci 59 55 52 62, exact

cci 13 8 13 17, exact

This results in the following output tables:

                                                         Proportion
                 |   Exposed   Unexposed  |      Total      exposed
-----------------+------------------------+------------------------
           Cases |        59          55  |        114       0.5175
        Controls |        52          62  |        114       0.4561
-----------------+------------------------+------------------------
           Total |       111         117  |        228       0.4868
                 |                        |
                 |      Point estimate    |    [95% conf. interval]
                 |------------------------+------------------------
      Odds ratio |         1.279021       |    .7354733    2.225323 (exact)
 Attr. frac. ex. |          .218152       |   -.3596687    .5506271 (exact)
 Attr. frac. pop |         .1129032       |
                 +-------------------------------------------------
                                  1-sided Fisher's exact P = 0.2133
                                  2-sided Fisher's exact P = 0.4267

                                                         Proportion
                 |   Exposed   Unexposed  |      Total      exposed
-----------------+------------------------+------------------------
           Cases |        13           8  |         21       0.6190
        Controls |        13          17  |         30       0.4333
-----------------+------------------------+------------------------
           Total |        26          25  |         51       0.5098
                 |                        |
                 |      Point estimate    |    [95% conf. interval]
                 |------------------------+------------------------
      Odds ratio |            2.125       |    .5926572    7.795925 (exact)
 Attr. frac. ex. |         .5294118       |    -.687316    .8717279 (exact)
 Attr. frac. pop |         .3277311       |
                 +-------------------------------------------------
                                  1-sided Fisher's exact P = 0.1536
                                  2-sided Fisher's exact P = 0.2581

Afterwards I used the data to calculate the p-values by hand using the formula from https://en.wikipedia.org/wiki/Fisher%27s_exact_test. The resulting p-value for the first dataset was p=0.0688 and for the second example I calculated p=0.983. Overall I cannot report significant results at the 5% level either way but I would like to understand where the discrepency comes from.

I have already looked at https://www.stata.com/support/faqs/s...rs-exact-test/ but think it does not explain my problem but it could be that I misunderstood the post.

Thank you for taking the time to read my post (:

[Andrew Musau]

Have you checked the Methods and Formulas section of the command's PDF manual entry for details on Stata's calculation? The Stata FAQ that you link admits to different ways to calculate the p-value, so there is no global "correct" way of doing this.

[Lennart Rehm]

Thank you for the reply. I checked the section and it is really short but I compared it a lecture slides on the Fisher Exact Test from the university of chicago and am pretty sure that the concept is the same. Following the Stata FAQ I assumed that even if there are multiple ways to calculate the p-value for the two sided test they should be resulting in similar values. In this example the difference is rather extreme resulting in a difference large enough for one method resulting in a p-value that is significant at the 10% level and one that is not significant at any level. Further I tried using the formula to calculate the p-values of some other stata example i found in the forum and got very similar or even the exact same p-values with stata. Which still leaves the question of why my two datasets result in such large differences

[Andrew Musau]

Following the Stata FAQ I assumed that even if there are multiple ways to calculate the p-value for the two sided test they should be resulting in similar values.

No, that's not correct. Consider the following statements:

The test naturally gives a one-sided p-value, and there are at least four different ways to convert it to a two-sided p-value (Agresti 2002, 93). One way, not implemented in Stata, is to double the one-sided p-value; doubling is simple but can result in p-values larger than one.

Stata instead adds the probabilities of all the tables at least as unlikely as the observed table. (For a rigorous statement, see Methods and Formulas of [R] tabulate twoway). In our example, all the “unlikelier” tables are in the same tail as the observed table. The other tail does not contribute to the p-value, so the one-sided and two-sided p-values are equal.

So Stata in this instance reports the same one-sided and two-sided p-value (see highlighted), whereas an alternative calculation reports the doubled one-sided p-value.

[Lennart Rehm]

Thanks again. I think i found the mistake in my manual approach. I will check it and might ask more questions if it still doesn't work

[Tiago Pereira]

Check this paper:

https://academic.oup.com/ije/article/37/6/1430/731900