Interpreting contrasts in margins with interaction terms

[Julien Dagenais]

Dear all,
I am investigating the effects of a continuous#categorical interaction on the binary outcome "aki2". What I am looking to show is how the probability of aki2 changes across different values in the continuous variable (c.log_avl), as a function of the interaction term (i.it_type). Running margins and marginsplot defines this relationship nicely.

logistic aki2 c.log_avl#i.it_type i.agecat male race i.bmicat i.cci_cat Auto_CKD_Preop i.renal i.clavien_cat
quietly margins it_type, at(log_avl=(-2(0.1)6))
marginsplot

So far so good.

Visually, when I inspect the curves 2.it_type and 1.it_type, the confidence intervals separate at the x-axis (log_avl) value of ~1.7. However, my goal is attach statistical proof to that visual observation. When I use the contrast term with margins, I don't get the results I expect. The graph of the difference in curves follow the expected shape; however, the confidence intervals are almost non-existent, and the curves develop statistically significant separation from one another at very small values of x. Why is that? I would expect the red curve below begin to have confidence intervals different from zero at roughly an x-axis value of 1.7. However, that's not what's shown. This happens irrespective of how I specify the contrast option, and irrespective of any correction for multiple comparisons. Any ideas? Many thanks for any help

margins rb1.it_type, at(log_avl=(-2(0.1)6)) mcompare(bonferroni)
marginsplot, yline(0)

Best,
Julien

[Clyde Schechter]

Your code looks correct. I think there is nothing wrong with these results; it is your expectations that are incorrect.

In the first graph, you are getting confidence intervals around the predicted probabilities themselves. But in the second graph you are getting confidence intervals around the differences in predicted probabilities. While it would be going too far to say that there is no necessary connection at all between those, it is certainly fair to say that they can be very, very different. It is entirely possible for a set of data to identify two parameters only rather approximately (wide confidence intervals for each) yet identify their difference with a high degree of precision (narrow confidence interval for the difference.)

[Julien Dagenais]

Appreciate your response, Clyde. That leads me to wonder how to best summarize the effect of the categorical variable on the marginal effect at representative values (MPR) curves shown? For the purposes of the study, it would support the hypothesis if I could claim that the interaction term exerts a significant effect on the outcome (aki2) above some given value of x.

If I use the first graph, and the associated margins output, I would note that the confidence intervals of curves 1 and 2 separate at x (log_avl) = 1.6.
However, if I use the margins contrast option and resulting estimates and graph, the difference in the curves are different from zero at just about every value of x. Which of the two is a more accurate reflection of the hypothesis?

[Clyde Schechter]

For the purposes of the study, it would support the hypothesis if I could claim that the interaction term exerts a significant effect on the outcome (aki2) above some given value of x.

To achieve that goal, you need to use the second graph, not the first.

Let's also clarify your terminology. Interaction terms to "exert an effect" (significant or otherwise) on an outcome. What you mean to say, I believe, is that above some given threshold value of x, the marginal effect of log_avi on the probability of aki2 differs significantly between the it_type groups.

That said, as is so often the case with p-values and significance testing, that sounds like the answer to a question nobody would or should care about. Presumably there is some practical threshold of difference in marginal effect on aki2 probability that matters in the real world. It would make more sense, to me at least, to talk about a threshold of x beyond which the difference marginal effect of x on the probablity of aki2 exceeds that threshold, i.e. it is large enough to be of practical importance. After all, any statement about when the difference becomes statistically significant is at least as much a statement about your sample size as it is about any relationships among the variables in your data set. If you could arbitrarily analyze as much or as little data as you please, you could get that threshold for statistical significance to be anything you wanted to. It's unlikely to be a useful number.

[Zeenat Murtaza]

Hi,

In continuation to above query, I am having few queries regarding marginal plots and contrast command. First, I ran marginal plots command for an interaction term of two factor and 1 continuous variable which leads to following result:

margins fcr, at (ca = (-2 (0.5) 2) crisis=(0 1))
marginsplot, recast(line) x(ca) by(crisis) yline(0)

1ST CRISIS EFFECTS.gph

Because the confidence intervals of fcr are overlapping, does that mean that there is no significant difference between the two categories of fcr in terms of the effect of ca on rnd(dependent variable)?? Also, am I right to interpret that for crisis=0, three lines of fcr=1 above the red line are statistically significant and different from zero and the same is true for crisis =1 while for fcr= 1, 6 lines out of 9 show statiscally significant results and same goes for crisis=1. So, there is no effect of crisis?? and there is no statical difference between fcr=0 & 1 as confidence intervals are overlapping?

I also run the contrast command to compare the effects and I took fcr=1 as the reference group: Command is as follows:

margins rb1.fcr, at (ca = (-2 (0.5) 2) crisis=(0 1))
marginsplot, recast(line) x(ca) by(crisis) yline(0)

Results show the following effects:
fcr 1 CRISIS EFFECTS.gph

Further, I also run contrast command with fcr= 0 as the reference category
margins r.fcr, at (ca = (-2 (0.5) 2) crisis=(0 1))
marginsplot, recast(line) x(ca) by(crisis) yline(0)
fcr 0 CRISIS EFFECTS.gph

Can you please advise if I am doing it right and whether the interpretation is correct?

Second, when I ran system GMM, the results show a negative relationship between ca and rnd but when an interaction term is run between ca#fcr#crisis, the sign is positive. So, given the graphs ( fcr 1 CRISIS EFFECTS.gph fcr 0 CRISIS EFFECTS.gph) how do I interpret these 2 opposite direction graphs?

(fcr & crisis are dummy/factor variables while ca is continuous variables).

Results of GMM are attached for reference: Generalized method of moments estimation.pdf

Please advise on interpretation of results and overlapping CIs .