Failing to confirm effect size of interaction term using margins

[Klaus Klausen]

Hi,

I would like to determine the effect size of the interaction term and plot the interaction accordingly. However, I am struggling to confirm the effect size using the margins command.

Negative binomial regression                            Number of obs =  1,285
                                                        Wald chi2(37) =      .
Dispersion: mean                                        Prob > chi2   =      .
Log pseudolikelihood = -824.95521                       Pseudo R2     = 0.0709

--------------------------------------------------------------------------------------------------------------------------------------------
                                                                           |               Robust
                                                             IM_Offsetting | Coefficient  std. err.      z    P>|z|     [95% conf. interval]
---------------------------------------------------------------------------+----------------------------------------------------------------
                                                                CEO_tenure |   .0105915    .011397     0.93   0.353    -.0117462    .0329291
                                                                   CEO_Age |   .0257327   .0100673     2.56   0.011     .0060011    .0454643
                                                                CEO_Gender |   -.454186    .335237    -1.35   0.175    -1.111238    .2028664
                                                                Acq_MA_exp |   .0303799    .013696     2.22   0.027     .0035363    .0572235
                                                                Deal_Value |   1.13e-11   1.95e-11     0.58   0.561    -2.69e-11    4.95e-11
                                                              Deal_AllCash |  -.4187356   .2615526    -1.60   0.109    -.9313693    .0938981
                                                                Deal_Stock |  -.3219534   .2822003    -1.14   0.254    -.8750559    .2311491
                                                              Targ_Listing |   .0394426   .0495802     0.80   0.426    -.0577328     .136618
                                                                  FF12_Div |    .254928   .1234985     2.06   0.039     .0128754    .4969807
                                                         Acq_Size_MB_ratio |   1.024231   .6737465     1.52   0.128    -.2962874     2.34475
                                                               Acq_Lev_WWU |   .0102851   .0233773     0.44   0.660    -.0355336    .0561038
                                                           Acq_TobinsQ_WWU |   .0537747   .0613892     0.88   0.381     -.066546    .1740953
                                                                   Acq_FCF |   2.038891   1.559126     1.31   0.191    -1.016939    5.094722
                                                             Acq_Cash_hold |   .4754442   .4200388     1.13   0.258    -.3478167    1.298705
                                                                   Acq_ROA |  -1.830164   1.528498    -1.20   0.231    -4.825965    1.165638
                                                              1.CEO_celeb3 |  -.1467234   .2383542    -0.62   0.538     -.613889    .3204422
                                                                       CNS |   .0360034   .1035405     0.35   0.728    -.1669321     .238939
                                                                           |
                                                          CEO_celeb3#c.CNS |
                                                                        1  |   .9279872   .3289432     2.82   0.005     .2832704    1.572704

. gen sample = 1 if e(sample) == 1
. sum CNS if sample == 1

    Variable |        Obs        Mean    Std. dev.       Min        Max
-------------+---------------------------------------------------------
         CNS |      1,285    .0006369    .6455761  -1.837005   2.953353

. sum IM_Offsetting if sample == 1

    Variable |        Obs        Mean    Std. dev.       Min        Max
-------------+---------------------------------------------------------
IM_Offsett~g |      1,285     .292607    .6375941          0          4

CEO_Celeb3 is a binary variable indicating the celebrity status of a CEO. 1 = CEO is considered to be a celebrity and 0 otherwise.

CNS is a CEO narcissism indicator and my dependent variable is the number of positive tone press releases (PR) around a specific firm event. The variable CNS is standardized.
From my interpretation, it appears that the influence of the CEO narcissism tendency on the number of issued PR is stronger for celebrity CEOs.

When it comes to interpreting the effect size I would go with: Ceteris paribus, increasing CNS by one SD (0.6455761) is associated with an 0.928 SD increase in published press releases when the CEO is a celebrity. Given that the SD of my y variable is 0.637 the effect size would be calculated as 0.928*0.637 = 0.60. In other words, increasing CNS by one SD is associated with an ~ 0.60 increase in published press releases when the CEO is a celebrity.
I am well aware of the fact that such interpretations are highly debatable in the sense of their meaningfulness but would appreciate it if someone could clarify whether the interpretation is correct in the first place.

I am struggling to confirm the effect size using margins:

margins, at(CNS = 0.0006369 CEO_celeb3 = 1) atmeans noatlegend // Effect of Interaction term if CNS takes on its mean value and the CEO is a celebrity

Adjusted predictions                                     Number of obs = 1,285
Model VCE: Robust

Expression: Predicted number of events, predict()

------------------------------------------------------------------------------
             |            Delta-method
             |     Margin   std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
       _cons |   .2020967   .0462454     4.37   0.000     .1114573    .2927362
------------------------------------------------------------------------------


margins, at(CNS = 0.646213 CEO_celeb3 = 1) atmeans noatlegend // Effect of Interaction term if CNS takes on its mean value + 1 SD and the CEO is a celebrity

Adjusted predictions                                     Number of obs = 1,285
Model VCE: Robust

Expression: Predicted number of events, predict()

------------------------------------------------------------------------------
             |            Delta-method
             |     Margin   std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
       _cons |   .3765594   .0994308     3.79   0.000     .1816786    .5714403
------------------------------------------------------------------------------

According to margins, the effect would be 0.3765594-0.2020967 = 0.17
I am wondering what contributes to the different results. My guess would be that these two results aren't really comparable due to different values of CNS in the regression analysis. That is, CNS might not even take on it's mean value of 0.0006369 for the regression coefficients.
I am happy about any comments on my problem.

[Clyde Schechter]

Ceteris paribus, increasing CNS by one SD (0.6455761) is associated with an 0.928 SD increase in published press releases when the CEO is a celebrity.

That is incorrect. The 0.928 coefficient in your regression output is the difference between the effect of CNS when the CEO is a celebrity and the effect of CNS n IM_offsetting when the CEO is not a celebrity. The effect of CNS when CEO is a celebrity is therefore .036 + .928.

Moreover, given that CNS is already standardized before you put it in the regression, I don't think it makes any sense to then use .645 (the SD of CNS in the estimation sample) as the magnitude of a 1 SD change in CNS. Just use 1. (Or, first ascertain what the regression sample's SD of CNS will be and then standardize CNS with that as the denominator.)

Finally, your calculation using the regression coefficients does not give you effects on the dependent variable. It gives you effects on the linear combination of your variables defined by your coefficients--and in the non-linear -nbreg- model you are using, these are quite different. By contrast, the -margins- command is working entirely in the dependent variable metric. In fact, given that the -nbreg- model has a log-link, additive differences in the independent variables correspond to multiplicative changes in the dependent variable.

[Klaus Klausen]

Thanks for the prompt answer Clyde.
I will have to dive deeper into the margins and nbreg command to fully understand the implications of your last paragraph but that certainly helps a lot.
Regarding the interpretation of the interaction would it be correct (in methodology terms) to say that for a given level of CNS, a celebrity CEO is associated with a .92 increase of published PR?

[Clyde Schechter]

Regarding the interpretation of the interaction would it be correct (in methodology terms) to say that for a given level of CNS, a celebrity CEO is associated with a .92 increase of published PR?

No. That is only true of celebrity CEO's who have CNS = 0. You are running an interaction model here. That means that the effect of being a celebrity CEO depends on the specific level of CNS. There is no one, overall effect of being a celebrity. I recommend you read the excellent Richard Williams' https://www3.nd.edu/~rwilliam/stats2/l53.pdf for a very lucid explanation of how interaction models work.

Moreover, on top of that, it would also be wrong to speak of a 0.92 additive increase in the number of published PR. Rather, if the correct level of CNS were stipulated, one might speak of a 2.5-fold multiplicative increase (2.5 = exp(.92)) in the number of published PR.

[Klaus Klausen]

Thanks, I haven't worked with interactions in a while and spent the day diving deeper into calculating the effects of coefficients in negative binomial regression models and (hopefully) got it right now. For those who are facing similar problems, I recommend the book Negative Binomial Regression by Joseph M. Hilbe.

In order to calculate the effect of the interaction for a given level of CNS I would have to apply the following formula: exp(β_{CEO_celeb3}+β_{CEO_celeb3xCNS}*CNS)

That is, according to the significant interaction term the effect for the mean level of CNS (.0006369) given the CEO is a celebrity would be: exp(-.147+.928*.000637) = .864
If one would increase CNS by 1, the effect would be: exp(-.147+.928*1.000637) = 2.185 -> Following the advice not to use the SD of CNS as a unit of increase but 1 instead.

This would mean that increasing the mean CNS by 1 would be associated with a 2.185-0.864 = 1.3 multiplicative increase in the number of published PR, given the CEO is a celebrity.

I would appreciate your thoughts on this.

[Clyde Schechter]

Your attempt to replicate what -margins- is telling you by simple calculations on the coefficients is futile because this is a non linear model. One of the consequences of that is that in the outcome metric (as opposed to the xb metric) there is automatically also an "interaction" among all variables. In other words, because -margins- takes into account all of the other variables and how, at the outcome level they influence the effects of CNS and CEO celebrity, but your coefficient analysis does not, you will not get them to match up. Sometimes when, as you have done, you restrict the other variables to their means in the -margins- command, the results come out close, but that is by no means guaranteed.

In a linear model, what you are trying to do would be workable. With a non-linear model, it is not. To replicate the results that -margins- gives you, you would need to, in effect, code a stripped-down version of the -margins- command. While that is possible, it is complicated, and not, in my opinion, worth the time and effort.

Added: Also, I'll just note that you don't compare two multiplicative effects by subtracting them. You do that by taking their ratio.

[Klaus Klausen]

At this point, my main goal is to get the correct interpretation of my interaction term and its corresponding effect. So right now I am just confused about whether the interpretation I provided in #5 is correct in terms of that is what the effect of my interaction term is according to the model (except that the change would be calculated as 2.185/0.864).
Sorry for any confusion I might have caused because I wasn't very precise with my previous post.

[Clyde Schechter]

Well, it is a legitimate calculation to do, but it could not be fairly described as "the effect of the interaction term." An accurate description is rather wordy: it is the ratio of the relative effect of a unit increase in CNS on PR posts in a celebrity CEO to the relative effect of a unit increase in CNS on PR posts in a non-celebrity CEO, all other variables equal. This wordiness is unavoidable because in an interaction model, the interaction term is never representative of an "effect," it is always representative of a difference (or ratio) of effects. So the description must always rerer to the two effects being contrasted (differenced, or ratioed) and it must explain what each of them is.[INDENT=2] [/INDENT]

[Klaus Klausen]

it is the ratio of the relative effect of a unit increase in CNS on PR posts in a celebrity CEO to the relative effect of a unit increase in CNS on PR posts in a non-celebrity CEO, all other variables equal.

Thanks for pointing that out.
Can you tell me how one would calculate the difference in effects for a celebrity CEO with a mean CNS (.0006369) compared to a celebrity CEO with a mean CNS + 1? In other words, I would like to calculate the effect difference of two CNS levels for celebrity CEOs. I thought that is what I calculated in #5 but according to your interpretation, it's rather the effect difference of being a CEO to the case where the CEO is no celebrity.
If I understand you correctly, the difference between the two effects can also be contrasted by their difference instead of their ratios. But isn't this contradictory to what you wrote in #6, i.e., that the effect differences should be compared in ratio form?

[Clyde Schechter]

Can you tell me how one would calculate the difference in effects for a celebrity CEO with a mean CNS (.0006369) compared to a celebrity CEO with a mean CNS + 1?

Leave out the -.147 and work only with the interaction coefficient.

If I understand you correctly, the difference between the two effects can also be contrasted by their difference instead of their ratios. But isn't this contradictory to what you wrote in #6, i.e., that the effect differences should be compared in ratio form?

Sure, you can contrast them by their difference, but when the effects are, themselves, ratios, doing that is, at best awkward, and at worst confusing.

Suppose an analysis tells me that A is 5 times as big as C, and B is 2 times as big as C. So the A-effect is a ratio of 5 relative to baseline C, and the B-effect is a ratio of 2. Yes, you can then say that the difference between the A and B effects is 5-2 = 3 times C. But I think it is clearer and more to the point to leave C out of it and just say that the A effect is 2.5 times larger than the B effect, unless you need to specifically re-emphasize that the baseline is C.

If you want to work with differences rather than ratios and communicate clearly, the -margins- results lend themselves more easily to that.

[Klaus Klausen]

After reconsidering all the helpful input you gave me I made another attempt to interpret the interaction term which is slightly different from the one I did before.

I am applying the following formula to calculate the Incidence Rate Ratio for my interaction: IRR = exp(β_{CEO_celeb3}+β_{CEO_celeb3xCNS}*CNS) resulting in: exp(0.1274982 + 1.026106*0.068) = 1.218 (Note that the coefficients do not match the ones in #1 as I made some adjustments to my model).
My interpretation would be: Compared to a non-celebrity peer, a celebrity CEO who is scoring average the narcissism score (0.068) is associated with a 22% increase of published press releases around a specific firm event, everything else equal.

[Clyde Schechter]

That looks right to me.