pairwise comparison of margins for logit models

[Jingyi Wang]

Dear Stata forum,

I have questions about the practices of comparing subgroups in my logit models. I would like to compare the probabilities of more frequent contact between sisters and brothers, taking into account respondents' partnership status and the gender composition of the respondent and their specified sibling.

I have realized that to assess the existence of differences between groups, I may have to perform pairwise comparisons of margins, rather than relying solely on confidence intervals for each group in the marginsplot for the results of the main model. This is because some confidence intervals overlap, and the lengths of bars have significant variations.







I have three questions as I am not sure if it makes sense and I am doing it right.

First, am I right that I should conduct pairwise comparisons to see which groups have significant differences? Second, given that the observations and confidence intervals vary for each group, should I apply any adjustments, such as Bonferroni's, Sidak's, or Sheffe's methods?

Third, there are 496 comparisons to make, due to the variables having 8 categories and 4 categories, respectively. Is there a way to visualize the results for comparisons between groups whose confidence intervals do not overlap? Another question related to these two questions is, since there are so many groups, would the differences make sense? (I am thinking that with a larger number of groups, the likelihood of having two groups with significant differences would also increase. But maybe that's why adjustments are needed?)

Below are the codes I plan to run for the pairwise comparison:

logit contact_sib_bi close_3_options i.ego_rom_relation_v4##ib8.genderpair

margins ego_rom_relation_v4#genderpair, pwcompare(effects) mcompare(bonferroni)

Thank you very much in advance for the help!!

[Clyde Schechter]

This is one of those games where the only winning move is not to play.

That is, I recommend against even trying to look for "significant" differences in 496 pairwise comparisons. Assuming you take statistical significance seriously (which, full disclosure, I don't, but here I'm taking the perspective of those who do), you do have to do a multiple comparison adjustment--for precisely the reason you cite in your post. Which one you use makes little difference. Bonferroni is the most commonly used due to its simplicity. The Scheffe comparison is the most conservative. But in the best case, your adjusted comparison will involve the equivalent of looking for an unadjusted p-value of something like 0.0001. To find that kind of difference requires one of two circumstances. Either the difference is so large that it would already be folklore and there is no point researching the question, or, your data set is so gigantic that almost any statistical test you could do would turn out to be statistically significant. (The only circumstance where this kind of thing actually works well, is particle physics where they use pvalues like 10^-6 in data sets with astronomically large numbers of collisions to analyze.)

Moreover, consider a given level of difference in probability of sibling contact. For sake of discussion let's say that a difference of 4 percentage points is large enough to be interesting and meaningful. You might have several pairwise comparisons with a difference this large. But if one of them involves relation-gender pairs with lots of instances in your data, and the other involves relation-gender pairs that are rare in your data, then their p-values will be wildly different: the same level of difference might be highly "significant" in one situation, and nowhere near it in the other. But what does that mean?

Finally, put aside all the issues of statistical significance. How will you draw any comprehensible conclusions, or make a useful display of results, from 496 comparisons. It's just too large for anybody to wrap his or her head around. Is it really the goal of your research, to try to make sense of 496 results? You need a more focused research question. (Or if this is not actually your research question, then, truly, there is no reason at all to even think about doing this.)

Assuming that this is actually your research question, here are some ways you could focus it better. Reduce the number of categories. Can you combine some of the relation levels so that you have only 2 or 3, or maybe 4 of those instead of 8? Would it be meaningful to reduce the four genderpair categories to just two:concordant or discordant? Is the relationship variable actually ordinal? If so, maybe instead of having all 8 levels compared with each other it would be sensible to just compare each level with the next highest level.