Multiple McNemar Test

[Jason Chui]

Hello all,
I am doing 10 McNemar test for comparing matched data. Example: mcci 192 0 1 0
1. Just like Tukey's method adjusts for all comparisons after ANOVA, I want to adjust for the familywise error rate with the 10 comparisons (McNemar test). I also want to report the adjusted P-value and adjusted 95% confidence interval for the odds ratio. I don't know how to do it in STATA. Can anyone help? What STATA command I can use?
2. How can I specify the exact confidence interval option in Stata
Many thanks,
Jason

[Leonardo Guizzetti]

You can make adjustments for FWER (controlling overall alpha / type 1 error rates) in several ways. To name a few, you have Bonferoni, Holm, Hochberg and stepwise. However, you will need to decide which one you want. The good news is that those procedures are easy to implement manually and can be performed by simple manipulation of the raw (original) p-values (or equivalently, with the alpha).

Adjusted 95% CIs would be calculable with the multiple-comparisons adjusted alpha levels. You would need to compute a 100*(1 - adjusted alpha)% interval instead. That said, most will typically choose to report the adjusted p-value and make inference with respect to this adjustment, and provide the raw 95% CI.

Following -mcc-, it is reporting to you the "exact" Fisher confidence intreval. It is exact in the sense that it assumes the Binomial distribution as a model and computes the confidence interval based on binomial permutations. That said, your OR is undefined in your example.

[Jason Chui]

Hello Leonardo,
Thank you very much for your reply. Really appreciated.
To ensure my understandings is correct, let me discuss with example. I am doing 10 McNemar test for comparing matched data. One of 10 McNemar tests is mcci 55 7 6 126 (pls see the attached STATA output).
1. The raw p-value is 0.7815. By using Bonferoni correction for 10 comparison, the adjusted p-value = 0.7815*10=7.815.
2. The odds ratio in this example is 1.17 [0.34-4.2] (exact) from the STATA output. I don't know how to adjust the odds ratio and CI. Do I just multiple by 10 (the number of comparsion group)?
3. I assume Bonferroni is good my McNemar test. I read many examples in the internet. Most sources use ANOVA as an example. I am uncertain whether Bonferroni is an reasonable option for non-parametric test?
4. Is there any STATA command that I can use to do the Bonferroni correction? Because I do it all manually now.
5. For my previous example, my OR is undefined (i.e. OR = 0 [0-39]). Did you mean the OR is indeterminate in my previous example?
Many thanks.
Jason

[Leonardo Guizzetti]

I suggest you try to find and read the following paper by Dmitrienko & D'Agostino Sr entitled "Traditional multiplicity adjustment methods in clinical trials". While the paper has a regulatory focus, the explanations are quite relevant and useful more broadly. The paper discusses how to make adjustments for confidence intervals, but I have not done so in some time and so I won't give that advice here as I don't have the time to work it out. While it is possible to adjust the CI, it's generally more complicated than working out the adjusted p-value/alpha value.

Within Stata, I am only aware of -margins- implementing some forms of multiplicity adjustment (including Bonferoni) but it's honestly the most straightforward method to implement, so manually doing this is not very much effort. Either multiply your p-values by k, the number fo comparisons, or adjust alpha by dividing by k.

In your previous example - your OR is not equal to 0, it is undefined. This is the case whenever you divide by zero, so a confidence interval is meaningless, unless you introduce some ad hoc adjustment to your cell counts.

[Bruce Weaver]

Thanks for sharing the link to that Dmitrienko & D'Agostino Sr (2013) article, Leonardo Guizzetti. I look forward to reading it. Meanwhile, here a couple 2005 Lancet articles by Schulz & Grimes that I quite like. Perhaps Jason will find them helpful too.

• https://pubmed.ncbi.nlm.nih.gov/15866314/
• https://pubmed.ncbi.nlm.nih.gov/15885299/

Cheers,
Bruce