How to interpret the insignificant, independent main effects, which became significant in the full model along with an interaction effect?

[PH SOH]

Hi, I use Stata 13, my dataset consists of the following, and I used the xtabond2 command to account for lagged dep var and endogenous regressors. My collaborators and I found that xtdpdml is not suitable for our dynamic panel data analysis because of the large N and T (25 years). The question is posted below the data. We have an image of the full model. The log file in notepad couldn't be uploaded (it says invalid file). We appreciate your response to our question. Thank you.

[CODE]
* Example generated by -dataex-. To install: ssc install dataex
clear
input float(explore_subgroup Ftech_mean Fmkt_mean rdintensity size) int firmage_revised float(InvAvgAge InvAvgFreq)
80 -.12253585 -.009261696 . . 3 2 1
25 -.13529032 -.009261696 . . 4 3 1
8 -.13492349 -.009261696 . . 5 4 1
102 .40838856 -.009261696 .1084 11125.602 108 2 3
96 .26603687 -.009261696 .1792 12061.073 109 2 3
111 .1564204 -.09259503 .10913333 13216.213 110 3 3.333333
146 .2426098 -.009261696 .137175 14471.038 111 3.75 3.25


We have results from xtabond2 estimations, which include the models with the respective main effects of Ftech_mean (x1) and Fmkt_mean (x2), and the full model with the main and interaction effects of the two variables. In the models with main effects only, individually (X1, X2) and combined (X1+X2), the coefficients are NOT statistically significant but have the correct signs. However, in the full model, all coefficients become statistically significant, i.e. X1 + X2 + X1X2.

[Clyde Schechter]

The coefficients of X1 and X2 in a model that also contains X1#X2 have a different meaning from the coefficients of X1 and X2 in the model without that interaction, and they should not be compared to each other. There is no reason to expect them to be similar in any way.

The non-interaction model implicitly assumes that the marginal effect of X1 on outcome is independent of X2. In the model without interaction, the coefficient of X1 is the unique marginal effect of X1 on the outcome variable.

By contrast, in the model with the interaction included, there is no such thing as the effect of X1 on the outcome. Rather there are infinitely many marginal effects of X1 on the outcome, one corresponding to each value of X2. In this model, the coefficient of X1 is the marginal effect of X1 on the outcome conditional on X2 = 0.

Similar considerations apply to interpreting the coefficient of X2 in each model.

As an aside, even when you are comparing coefficients across models in circumstances where they can be compared, you should never do so in terms of whether they are statistically significant. The difference between a statistically significant result and a not statistically significant result is, itself, not statistically significant. In fact, it's not even meaningful in any useful sense.

[PH SOH]

Hi Clyde, thank you for your reply, and your explanations are very clear and convincing. In management journal publications, reviewers often question and compare the coefficient estimates across the models, and they insist that we should show all the models with each entry of the main factor. This is the case especially when we develop hypotheses for X1 and X2, respectively, and then a third, which includes the interaction effect.

I have two remaining questions, whether we are correct about these interpretations.
1) In combining all the models, we could say that our first two hypotheses for the main effects are rejected, meaning that they don't have predictive power independently on the outcome. However, the hypothesis for the interaction effect is the only one supported.
2) In the full model, we could conclude the same for the third hypothesis so long the coefficient of the interaction effect is statistically significant, but the coefficients of the main effects need not be so (I recall reading a similar post for the interaction effect model).

Thank you again.

[Clyde Schechter]

If a reviewer raises a question about the coefficient of X1 in a non-interaction model being different from the coefficient of X1 in an interaction model, your response should be to paraphrase my response in #1 and gently remind the reviewer of these basic facts about interaction models.

Personally, I do not see any merit in showing both the model with and without the interaction. If the interaction term is meaningfully large (which isn't the same thing as statistically significant, but let's put that aside for now), then that is evidence that the non-interaction model is mis-specified and invalid. So why show the latter? If the interaction term is too small to matter, then I suppose there is no harm in showing both models, but what is the point?

Assuming, nevertheless, that you do show both models in your situation just interpret the coefficients for what they are. In the non-interaction model, the coefficients of X1 and X2 are estimates of the marginal effects of these variables on your outcome. But then point out that the model with interaction supports the view that the first model is incorrect. In the new model, the coefficients of X1 and X2 are marginal effects of X1 and X2, each conditional on the other being 0. If 0 is not a sensible value for either X1 or X2, then it is also worth pointing out that that coefficient is, therefore, unrelated to anything in the real world and should basically just be ignored: it's only purpose is to help in computing the marginal effects of that variable at other values.

Frankly, I think that discussing the coefficients of variables in interaction models is usually just a recipe for confusion anyway. I think it is much better, instead to graphically display the outcome as a function of X1 and X2, or to display the marginal effects of X1 as a function of X2 and vice versa.

If you re-run your model using factor variable notation (-help fvvarlist-) you will be able to use the -margins- and -marginsplot- commands after that to create these interaction graphs quite simply. The pictures they provide are worth thousands of words. The coefficients of an interaction model are themselves abstract entities that most audiences have considerable difficulty grasping, and, unfortunately, they don't usually correspond to anything of actual real world interest: they are just ingredients in equations. (The exception is the coefficient of the interaction term itself, which in many contexts is interpetable as a difference-in-differences.)

[PH SOH]

Hi Clyde, thanks for your great advice. We will certainly refer to and acknowledge your points in our paper. Here is the graph generated. We could see that the main effect of X1 has the correct -ve sign, and at a high level of X2, the slope of the top line is steeper than the middle and bottom lines, indicating the presence of an interaction effect. The conditional value of X1 or X2 = 0 respectively in the interaction model indeed is not meaningful for firms, so we will not interpret X1 and X2 individually. Very helpful explanations, thank you again!