GEE interpretation - significance disappears when interaction term is added

[Amber van Dijk]

Dear Statalist users,

I have some difficulties with interpreting the results of a GEE analysis. I hope someone is able to help me with this. I have no profound background in statistics, therefore I was hope someone could explain it to me in relatively easy language.

Question 1: Why does the significance of CEO Career Variety disappears when the variable CEO political liberalism is added to the model? (model 3a)

Question 2: Why does the significance of CEO Political Liberalism disappears when I add the interaction effect (Career Variety x Political Liberalism). (model 3b)

Question 3: How could it be the case that the significance disappears for CEO Career Variety in model 3a and 3b?

Thanks in advance.

[Clyde Schechter]

The simplest answer to your question is that there is no reason to expect that changes like this would not occur. But let's unpack that a little bit.

Models 2b and 3a are different models. In each model, the estimated effects of career variety are conditional on the other variables that are included in that model. When you add or remove variables, from a model, the results for any variables common to both models can change. The extent of change can be large or small, and in either direction. More specifically, in this case, it is likely that political liberalism and career variety are correlated with each other, so that some of the association of career variety with the outcome seen in model 2b gets re-attributed to career liberalism when the latter is added into model 3a, and the remaining part of he career variety : outcome association is small enough that it no longer reaches statistical significance.

Models 3a and 3b are even more different from each other than 2b and 3a, because the inclusion of an interaction term changes the very meaning of the variables themselves. That is, the variable called political liberalism in model 3a is a different thing from the variable called political liberalism in model 3b, so it is not even meaningful to compare them in the first place. Specifically, in model 3a, the variable called political liberalism gives an estimate of the average effect of political liberalism on the outcome. In model 3b, the variable called political liberalism gives only an estimate of the effect of political liberalism on the outcome when career variety = 0. (I do not know how your career variety variable is measured: it may be that 0 isn't even a possible value for that variable; in which case the variable political liberalism, by itself, is just a real-world-meaingless mathematical abstraction in that model.) Models with interaction terms are complicated and require care in interpretation. I highly recommend you read the excellent Richard Williams' https://www3.nd.edu/~rwilliam/stats2/l53.pdf, which is the clearest explanation I know of how interaction models work.

Finally, although it isn't really an issue in this particular data, where large and meaningful changes in the coefficients are being observed, I strongly caution you in the future not to focus on, nay, never to even ask, whether the "statistical significance" of a variable changes from one model to another. It is a misleading question. Apart from the differences between models outlined above, changing models sometimes entails changes in sample size or effective sample size, so that even with no change in the coefficient itself, the statistical significance can change. Also, even when, as in your table, the sample size remains the same, the standard errors can change when variables are added or removed, and the same, or nearly the same coefficient, can, as a result, change its "statistical significance." It is also important to remember that difference between statistically significant and not statistically significant is not, itself, statistically significant.

These remarks may seem paradoxical, because you may have been taught that "statistically significant" means "there is an effect" and not "statistcally significant" means "there is no effect." This notion, though utterly false and misguided, is widely taught and has caused brain damage to thousands of budding data analysts. The very notion of statistical significance is so widely misunderstood that many of us believe that the very notion should no longer be used at all (although I do believe that when properly used in narrow circumstances, it does have some usefulness). If you have time and are interested in understanding this better, see https://www.tandfonline.com/doi/full...5.2019.1583913 for the "executive summary" and https://www.tandfonline.com/toc/utas20/73/sup1 for all 43 supporting articles. Or https://www.nature.com/articles/d41586-019-00857-9 for the tl;dr. In any case, if you continue to use the concept of statistically significance in your work (it isn't going away any time soon), you should train yourself not to confuse it with presence vs absence of an effect--that thought pattern will get you in trouble much of the time.

[Amber van Dijk]

Thank you so much for your help Clyde. The article about the interaction effect explains a lot.

However, I am still struggling with the interpretation of the interaction effect. The interaction term is not significant. However, let's assume it is.

So for either Career Variety x MBA (model 2b) or Career Variety x Political Liberalism (model 3b), what conclusions can I draw here?

[Clyde Schechter]

Let's look at Model 2b. The coefficient for the Career Variety X MBA interaction is, to 2 decimal places, -3.46. This means that there is an interference between the effect of career variety on the outcome variable and the effect of MBA on it. The whole is less than the sum of the parts, as it were, and specifically it is less by 3.46 units. To interpret it more specifically, I'm going to assume that career variety is a 0/1 dichotomous variable, and the MBA is also. (The same principles apply with continuous variables but the calculations get more complicated.) The coefficient of career variety, 7.66, is the expected outcome difference between a CEO with and without career variety, provided that he or she has no MBA. The coefficient of MBA, 1.45, is the expected outcome difference between a CEO with and without an MBA, provided that he or she has no career variety. What if a CEO has both career variety and an MBA. In that case, relative to a CEO with neither, that CEO's expected outcome is higher, but not by 7.66 + 1.45. Instead, that CEO's expected outcome is higher by 7.66 + 1.45 - 3.46.

The situation in Model 3b can be interpreted in an analogous way.