how to interpret interaction through confidence interval

[Rosemary Li]

Hi Experts,

I graphed difference in difference and its corresponding confidence interval to interpret interaction in nonlinear models. I know the rule is that we conclude the interaction is not significant if the confidence interval includes 0. However, I am not sure how strict this rule is. Sometimes, only a small proportion of the difference's confidence interval includes zero. In that case, should we conclude the interaction is significant or not? Thanks.

Below, I give three examples for your reference. What would you conclude for each example? Thanks.

To calculate difference in difference: calculate margins when x2 is high with x1 on x axis, calculate margins when x2 is low with x1 on x axis, then calculate the difference between the two and confidence interval of the difference.

In my case, x2 is a categorical variable. I use level 1 as the reference group, so the difference would be between a higher level and level 1. And x3 is a continuous variable, so the difference I calculated is between maximum x3 and minimum x3. In both cases, x1 which I put on x axis is a continuous variable.

Example 1

Example 2

Example 3

[Phil Bromiley]

I'd report exactly what you have. For example, the difference between max x3 and min x3 is statistically significant at a% for x1>0 (use margins to find the precise point) which is z% of the z values. Since interactions can even change the sign of coefficients, this kind of "significant for these values of x" results are quite common. You can even have a significant positive effect for some values and a significant negative effect for others (but you do not appear to have these in your graphs). However, you also need to be careful not to mistake significant for effect size. In the last graph, the predicted y increases with x1 throughout - it only become insignificant because the confidence interval gets larger.

[Rosemary Li]

My only concern is that reviewers usually get used to a one line conclusion of whether the interaction is significant or not--in other words, whether the hypothesis is supported or not--on the basis of a p value (at 5% level of significance for example). If I report the interaction is significant--in other words, the hypothesis is supported--for these values not those values, reviewers may not feel comfortable if you know what I mean. This is my first time to interpret significance through confidence interval, rather than a straightforward p value, so I don't know what is the usual practice to make a conclusion under the circumstances.

If I report what I have and conclude that the hypothesis is generally supported, I am not sure whether the reviewers will say I make the wrong conclusion and say the hypothesis is not supported because some proportion of the confidence interval includes zero (which is true). This is actually a response from one of my coauthors. I could understand this kind of response. In the mean while, as you also mentioned, it is common that significance changes as the variable on x axis changes. If we conclude the hypothesis is not supported once confidence interval includes zero, no matter how small the proportion is (very strictly stick to the rule) (like example 2), we may rarely find support for hypothesis.

This is a real dilemma for a novice like me. I wish I could also hear some experiences of journal submissions.

By the way, thank you for commenting on effect size, though I wish you could elaborate a little bit. It seems to be an important point, while I am not sure I completely get it. If you could give an illustration of mistaking significance for effect size, I would appreciate it. Thanks.

[Rosemary Li]

By the way, I actually use the full range (from minimum to maximum) of x1 (on x axis). Is this the usual practice?

For example, the minimum of real x1 (not centered) is 0. In this case, x1=0, the interaction between x1 and x3 is 0, and the actual model seems to be reduced from y=beta0+beta1*x1+beta2*x3+beta3*x1*x3+control variables to y=beta0+beta2*x3+control variables. Then difference in example 3 when x1=-20 (x1 in example 3 is centered, x1=-20 corresponding to real x1=0) seems to be a difference in main effect of x3 (high x3 vs. low x3). In other words, the first bar of confidence interval in example 3 does not represent interaction effect. I am wondering whether I should drop it from the graph. For instance, should the range of x1 be (>0, maximum or some other value), instead of (0, maximum or some other value). Also feel free to let me know if my understanding is wrong.

If any of you experts could shed light upon it, I would really appreciate it.

[Rosemary Li]

Correction: the three examples I gave are not difference in difference; they are just difference in predicted y (y may be probability or count for example).

[Rosemary Li]

I now close this post, because I confused interpreting interaction through confidence interval in the graphs I shown with (traditional) interpreting interaction through p value. Observing different patterns of difference in predicted y on the basis of different x1 values on the x axis is actually suggesting an interaction effect.

Even though I was confused, Phil's comments are still correct and helpful.

[Carlo Lazzaro]

Tyrah:

If I report the interaction is significant--in other words, the hypothesis is supported--...

if you find a statistical significant result, you do not support the alternate hypothesis, but reject the null.

[Doug Hemken]

I think you need to clarify what you mean by "interaction".
Suppose you have an model y = b0 + b1*x1 + b2*x2 + b3*x1*x2 (+e)
If you are reporting on the "interaction term", you are interested in just the significance of b3, the curvature term.
Your graphs suggest you are interested in the combination of b1 and b3 (or b2 and b3), that is, the marginal effect of x1 or of x2. But the effect of x1 depends upon the value of x2, and vice versa. In fact, there will be some numerical value of x2 where the effect of x1 is 0!

[Rosemary Li]

Carlo, I understand what you mean and I adore your rigorousness

[Rosemary Li]

Hi Doug, I am dealing with a nonlinear model, so it is no longer appropriate to report b3. The reasons are discussed in Ai and Norton (2003). For the same reasons, I am not reporting the exact marginal effect we often refer to. I report the difference in expected y instead.

But you are right that what I report is a combination of b1 (or b2) and b3.

By interaction effect, I learned that it means the effect of one variable depends on the other variable. It is discussed in Brambor, Clark, and Golder (2006).

Feel free to comment. I welcome all comments and like communications.

Ai, C., & Norton, E. C. (2003). Interaction terms in logit and probit models. Economics letters, 80(1), 123-129.
Brambor, T., Clark, W. R., & Golder, M. (2006). Understanding interaction models: Improving empirical analyses. Political analysis, 14(1), 63-82.

[Doug Hemken]

But you are right that what I report is a combination of b1 (or b2) and b3.

I think Ai & Norton's use of the phrase "interaction effect" is confusing. They would be better off referring to "interaction effects". I think most of us would agree that the interaction term, by itself, is seldom all that interesting - it is a rate of change of some other effects.

[Rosemary Li]

Now I reflect and realize a difference between best practice and commonly accepted practice. For the latter, it becomes a norm. Under that norm, people make simplified statements during communication, with certain assumptions taken for granted. For instance, when we make conclusions on hypothesis testing, we implicitly refer to 5% level of significance unless another value explicitly stated. And we often see in journal articles that (alternative) hypotheses are supported or not, even though we know we do not accept the hypotheses. As for interaction effect, it is a bit different. A few review articles show that it is not well understood in many publications. (I myself spend a lot of time on figuring out what is right and what is wrong and I am still on the way.) So the same term "interaction effect" is used in varied ways.

I agree that it is a better practice to be more rigorous and spend a few more minutes and a few more sentences to follow the best practice and make things clearer. But it is also understandable to make simplified statements to save some time when communicating with people from the same or similar area I guess.

[Doug Hemken]

You said it! This is why we so often go back to the mathematical representation - although with different notational assumptions, even the math can be a source of confusion sometimes.