Very different Results for Marginal effects vs. average marginal effects (continuous interaction)

[Gesine Hoeltmann]

Dear all,

this is my first time posting on StataList so I hope Im not in breach of most rules and requirements.

My problem is the following:

I am running a logistic regression with a continuous by continuous interaction term, however, I get very different results for my predicted probabilities of the interaction term when using A) the margins command which defines different levels for each variable versus B) the dydx command, which, if I understand correctly, is simply meant to provide me with the average of the marginal effects observed in the marginsplot obtained from A.

While using the code in Option A provides me with relatively realistic predicted probabilities (ranging from 0 to 0.4), the code in Option B deviates starkly from these results (ranging from 0.01 to 0.03), which seems odd given the strongly positive and significant effect these variables have in my prior analysis without the interaction term.

The interaction term of interest is the effect of c.Dissatisfaction_Democracy on the Populist right-wing vote (binary) at different levels of c.Limit_Refugees. (c.Dissatisfaction_Democracy##c.Limit_Refugees).

• Dissatisfaction_Democracy has a scale of 1-10
• Limit_Refugees has a scale of 1-5

CODE

Option A:
logistic dep c.Dissatisfaction_Democracy##c.Limit_Refugees i.Age_Cat i.Sex ib1.Education i.Income i.employment_status Personal_economicposition i.East LeftRight [pweight=w_ipfges]
margins, at (Dissatisfaction_Democracy=(1(1)10) Limit_Refugees=(1(1)5)) vsquish
marginsplot, name(A1, replace)

Option B:
logistic dep c.Dissatisfaction_Democracy##c.Limit_Refugees i.Age_Cat i.Sex ib1.Education i.Income i.employment_status Personal_economicposition i.East LeftRight [pweight=w_ipfges]
margins, dydx(Dissatisfaction_Democracy) at(Limit_Refugees=(1(1)5)) vsquish post saving(file1, replace)
marginsplot, yline(0) name(A1, replace)


Is this stark difference in predicted probabilities a natural outcome of these operations? Or is there something wrong with my code?

Very grateful for your insight!
Gesine

[Clyde Schechter]

Well, the two -margins- commands ask Stata for very, very different things. You seem to be under the impression that they should be the same--but they should not. So let's review what they each are and then you can decide which one, if either, is what you want. (Or maybe you want them both for different purposes.)

margins, at (Dissatisfaction_Democracy=(1(1)10) Limit_Refugees=(1(1)5)) vsquish

This one asks Stata to calculate the predicted probabilities of dep at every combination of Dissatisfcation_Democracy between 1 and 10 with every value of Limit_Refugees between 1 and 5, averaged over the joint sample distribution of all the other variables in the model. These are predicted probabilities, not marginal effects.

margins, dydx(Dissatisfaction_Democracy) at(Limit_Refugees=(1(1)5)) vsquish post saving(file1, replace)

This one asks Stat to calculate the marginal effectof Dissatisfaction_Democracy at each value of Limit_Refugees from 1 through 5, averaged over the joint sample distribution of all the other variables in the model. These are marginal effects, not predicted probabilities. Remember that the marginal effect is, as the Stata syntax suggests, a first derivative. It is the rate of change in the predicted probability of dep per unit change in Dissatisfaction_Democracy.

There is no reason to expect the rate at which the predicted probability changes will equal the predicted probability itself (unless the relationship is a pure exponential.)

[Gesine Hoeltmann]

Dear Clyde,

please excuse my late response - your explanation has been extremely helpful!

I definitely confused the two concepts.

Thank you for your help!!
Gesine

[Nadine OShea]

Dear all,

I am new here and I hope that I am following the guidelines of the forum.

I am using nbreg (as well as Poisson) in Stata 16.1 with an interaction term (both variables dummy variables).
This would be the equation (roughly, as I leave out the control variables here): Y-hat(nr of civilians killed) = b0 + b1(mediation) + b2(intell) + b3(mediation*intell)

* Example generated by -dataex-. For more info, type help dataex
clear
input double OSVgvt_best float year strL location float(Lmediation_dummy Lgvt_rec_supp_intell ln_brd Lag_ln_UN_troop ln_LagOSVreb_best rwanda94 country_id)
   0 1993 "Algeria"                  . .  6.716595         0         0 0 615
   0 1994 "Algeria"                  1 0  7.571989         0 3.2580965 0 615
   0 1995 "Algeria"                  1 0  8.025844         0  2.484907 0 615
   0 1996 "Algeria"                  1 0   6.99485         0         0 0 615
   0 1997 "Algeria"                  1 0  7.604895         0   1.94591 0 615
   0 1998 "Algeria"                  1 0  8.096817         0  5.872118 0 615
   0 1999 "Algeria"                  1 0  7.050123         0  4.691348 0 615
   0 2000 "Algeria"                  1 0  7.071573         0  4.828314 0 615
   0 2001 "Algeria"                  1 0   6.51323         0  5.505332 0 615
   0 2002 "Algeria"                  1 0  6.964136         0  3.465736 0 615
   0 2003 "Algeria"                  1 0  6.570883         0  6.251904 0 615
   0 2004 "Algeria"                  1 0  6.214608         0   3.78419 0 615
   0 2005 "Algeria"                  1 1  5.446737         0 4.4998097 0 615
   0 2006 "Algeria"                  0 0  5.676754         0   2.70805 0 615
   0 2007 "Algeria"                  0 0  6.204558         0  3.610918 0 615
   1 1993 "Angola"                   . .   9.42529         0         0 0 540
   0 1994 "Angola"                   1 1  8.296297  4.406719   5.83773 0 540
   0 1995 "Angola"                   1 0   5.47227  4.890349  3.555348 0 540
   1 1996 "Angola"                   1 0  3.465736 10.378603 3.8501475 0 540
  37 1997 "Angola"                   1 0   4.65396 11.260688         0 0 540
  41 1998 "Angola"                   1 0  7.082549 10.690125  2.833213 0 540
   1 1999 "Angola"                   1 0  7.888334 8.8679905 4.1431346 0 540
   1 2000 "Angola"                   1 0  7.095064  7.732369   5.97381 0 540
   0 2001 "Angola"                   0 0  7.675082  1.609438  5.549076 0 540
  12 2002 "Angola"                   1 0  6.361302         0  6.936343 0 540
  59 2003 "Angola"                   1 0 4.4308167         0 3.8286414 0 540
  13 2004 "Angola"                   1 0  3.637586         0         0 0 540
   1 2007 "Angola"                   . . 3.2580965         0         0 0 540
   0 1994 "Burundi"                  . . 4.0253515         0         0 0 516
1118 1995 "Burundi"                  1 0  7.344073         0         0 0 516
1542 1996 "Burundi"                  1 0  7.890957         0  5.062595 0 516
 199 1997 "Burundi"                  1 0   7.36074         0   6.39693 0 516
 314 1998 "Burundi"                  1 0  7.242798         0   6.25575 0 516
 225 1999 "Burundi"                  1 0  6.919684         0  6.137727 0 516
 389 2000 "Burundi"                  1 0  7.635787         0  5.572154 0 516
   1 2001 "Burundi"                  1 0  7.133296         0  5.455321 0 516
 243 2002 "Burundi"                  1 0  7.362645         0  4.248495 0 516
 129 2003 "Burundi"                  1 0  7.126891         0   5.09375 0 516
  33 2004 "Burundi"                  1 0  6.520621         0 4.6151204 0 516
  26 2005 "Burundi"                  1 0  5.934894    10.248  5.313206 0 516
   8 2006 "Burundi"                  1 0  5.220356 11.052936  4.189655 0 516
  21 2001 "Central African Republic" . .  5.375278         0         0 0 482
  21 2002 "Central African Republic" 0 0   5.02388         0         0 0 482
 133 2006 "Central African Republic" . .  5.247024         0         0 0 482
 193 1993 "Chad"                     . .  5.746203         0         0 0 483
 230 1994 "Chad"                     1 0   6.25575         0         0 0 483
   4 1997 "Chad"                     . .   4.89784         0  4.820282 0 483
 145 1998 "Chad"                     1 0  5.288267         0  3.496508 0 483
   0 1999 "Chad"                     1 0  5.976351         0  2.944439 0 483
   5 2000 "Chad"                     0 0  6.975414         0         0 0 483
   4 2001 "Chad"                     1 0  5.438079         0         0 0 483
   0 2002 "Chad"                     1 0  5.693732         0         0 0 483
   0 2003 "Chad"                     1 0  3.610918         0 1.0986123 0 483
  45 2005 "Chad"                     . .  5.587249         0         0 0 483
  19 2006 "Chad"                     1 0  7.578657         0  4.804021 0 483
   0 2007 "Chad"                     1 1  6.066108         0  6.159095 0 483
   0 1997 "Comoros"                  . . 4.0253515         0         0 0 581
  82 1993 "Congo"                    . .  4.905275         0         0 0 484
 373 1997 "Congo"                    . .  9.250714         0         0 0 484
 433 1998 "Congo"                    1 0  8.232972         0 3.6635616 0 484
 322 1999 "Congo"                    1 0  6.854354         0  4.060443 0 484
   9 2002 "Congo"                    . .  5.455321         0         0 0 484
1353 1996 "DR Congo (Zaire)"         . .   10.4497         0         0 0 490
4120 1997 "DR Congo (Zaire)"         1 0  9.539285         0  10.36284 0 490
1591 1998 "DR Congo (Zaire)"         1 0  8.989569         0  8.521783 0 490
 622 1999 "DR Congo (Zaire)"         1 0   8.92864         0  7.914983 0 490
 108 2000 "DR Congo (Zaire)"         1 0  8.023552         0  7.575585 0 490
  27 2001 "DR Congo (Zaire)"         1 0  7.201171  4.276666  6.045005 0 490
  16 2005 "DR Congo (Zaire)"         . .  5.743003  9.882315  5.568345 0 490
 153 2006 "DR Congo (Zaire)"         1 0  5.963579 12.115678  5.568345 0 490
  92 2007 "DR Congo (Zaire)"         1 0  6.783325  12.16165 3.9318256 0 490
   0 1993 "Djibouti"                 . .  3.496508         0         0 0 522
   0 1994 "Djibouti"                 0 1 3.6635616         0         0 0 522
   0 1999 "Djibouti"                 . .  3.218876         0         0 0 522
   0 1997 "Eritrea"                  . .  3.433987         0         0 0 531
   0 1999 "Eritrea"                  . .  9.752839         0         0 0 531
   0 2003 "Eritrea"                  . . 4.0430512         0         0 0 531
  26 1993 "Ethiopia"                 . .  4.890349         0         0 0 530
   0 1994 "Ethiopia"                 0 0 4.3307333         0         0 0 530
   0 1995 "Ethiopia"                 0 0  6.949856         0         0 0 530
   0 1996 "Ethiopia"                 0 0 4.1743875         0         0 0 530
   3 1998 "Ethiopia"                 . .  6.934397         0         0 0 530
   0 1999 "Ethiopia"                 1 0 10.334815         0         0 0 530
   0 2000 "Ethiopia"                 0 0 10.792736         0         0 0 530
   0 2001 "Ethiopia"                 0 0  5.961005  7.561121         0 0 530
  35 2002 "Ethiopia"                 0 1  6.695799 10.591346         0 0 530
 509 2003 "Ethiopia"                 0 1  6.996682 10.746197         0 0 530
 244 2004 "Ethiopia"                 1 1  6.163315 10.740865  .6931472 0 530
  24 2005 "Ethiopia"                 1 0  5.669881 10.712237         0 0 530
  20 2006 "Ethiopia"                 0 0  6.063785 10.526802         0 0 530
 153 2007 "Ethiopia"                 0 1  6.214608  10.36505         0 0 530
   0 2000 "Guinea"                   . .  5.545177         0         0 0 438
   0 2001 "Guinea"                   1 0  6.063785         0 2.0794415 0 438
   0 1998 "Guinea-Bissau"            . .  6.224558         0         0 0 404
   0 1999 "Guinea-Bissau"            1 0  5.293305         0         0 0 404
 227 2002 "Ivory Coast"              . .   6.93537         0         0 0 437
  73 2003 "Ivory Coast"              1 0  6.198479         0 4.7095304 0 437
  59 2004 "Ivory Coast"              1 0  5.717028         0  4.744932 0 437
   0 1998 "Lesotho"                  . .  4.356709         0         0 0 570
  44 2000 "Liberia"                  . . 4.6151204         0         0 0 450
end

I firstly aim at finding out whether the interaction term is significant (bearing in mind that there is a debate on significance). This is the code I use:

nbreg OSVgvt_best i.Lmediation##i.Lgvt_rec_supp_intell ln_brd Lag_ln_UN_troop ln_LagOSVreb_best if rwanda94==0, cl(country_id)

Fitting Poisson model:

Iteration 0:   log pseudolikelihood = -24321.468  
Iteration 1:   log pseudolikelihood = -23652.845  
Iteration 2:   log pseudolikelihood = -23601.887  
Iteration 3:   log pseudolikelihood = -23600.726  
Iteration 4:   log pseudolikelihood = -23600.724  

Fitting constant-only model:

Iteration 0:   log pseudolikelihood = -809.98182  
Iteration 1:   log pseudolikelihood = -583.62777  
Iteration 2:   log pseudolikelihood = -583.27134  
Iteration 3:   log pseudolikelihood = -583.27079  
Iteration 4:   log pseudolikelihood = -583.27079  

Fitting full model:

Iteration 0:   log pseudolikelihood =  -574.2187  
Iteration 1:   log pseudolikelihood = -560.91356  
Iteration 2:   log pseudolikelihood = -560.22844  
Iteration 3:   log pseudolikelihood =  -560.2201  
Iteration 4:   log pseudolikelihood =  -560.2201  

Negative binomial regression                    Number of obs     =        125
                                                Wald chi2(6)      =     156.50
Dispersion           = mean                     Prob > chi2       =     0.0000
Log pseudolikelihood =  -560.2201               Pseudo R2         =     0.0395

                                                     (Std. Err. adjusted for 20 clusters in country_id)
-------------------------------------------------------------------------------------------------------
                                      |               Robust
                          OSVgvt_best |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
--------------------------------------+----------------------------------------------------------------
                   1.Lmediation_dummy |    2.02481   .5402429     3.75   0.000     .9659534    3.083667
               1.Lgvt_rec_supp_intell |   2.347461   .8495254     2.76   0.006     .6824214      4.0125
                                      |
Lmediation_dummy#Lgvt_rec_supp_intell |
                                 1 1  |  -1.920368   .8699342    -2.21   0.027    -3.625407   -.2153279
                                      |
                               ln_brd |   1.043398   .1639055     6.37   0.000     .7221491    1.364647
                      Lag_ln_UN_troop |   .0007073   .0433578     0.02   0.987    -.0842724     .085687
                    ln_LagOSVreb_best |  -.0524842   .0913355    -0.57   0.566    -.2314984    .1265301
                                _cons |  -4.229268   .9649702    -4.38   0.000    -6.120575   -2.337961
--------------------------------------+----------------------------------------------------------------
                             /lnalpha |   1.614225   .3389045                      .9499845    2.278466
--------------------------------------+----------------------------------------------------------------
                                alpha |   5.023994   1.702654                       2.58567    9.761693

The interaction term is significant and negative, which some scholars in my field would interpret as 'governments that receive intell are less likely to kill civilians during mediation processes' while others might even go as far to say 'governments that receive intell are associated with a decrease in killings during mediation processes'. However, I doubt that this is what the table is saying, particularly since I am dealing with an interaction term in a non-linear model. Furthermore, I also doubt that I can even make these statements when I use the margins command.

And this is where I have some trouble now because after numerous hours of reading and with the help of the wonderful explanations here, I am still struggling to understand the difference between the dydx and the the predicted probabilities. I will illustrate my problem.

For my hypothesis, I follow Busenbark, et al. (2022). A Marginal Effects Approach to Interpreting Main Effects and Moderation. Organizational Research Methods, 25(1), 147–169
Hypothesis: The presence of external intelligence moderates the positive relationship between mediation and government violence against civilians such that the relationship is weakened when the government receives external intelligence but is strengthened when the government does not receive external intelligence.

Busenbark et al. (2022) also suggest using the dydx command (as various scholars have done on Statalist as well) to get the first derivative. But I still struggle to interpret the results:

 margins, dydx(Lmediation_dummy) at(Lgvt_rec_supp_intell=(0 1))
 
Average marginal effects                        Number of obs     =        125
Model VCE    : Robust
 
Expression   : Predicted number of events, predict()
dy/dx w.r.t. : 1.Lmediation_dummy
 
1._at        : Lgvt_rec_supp_intell=           0
 
2._at        : Lgvt_rec_supp_intell=           1
 
-------------------------------------------------------------------------------------
                    |            Delta-method
                    |      dy/dx   Std. Err.      z    P>|z|     [95% Conf. Interval]
--------------------+----------------------------------------------------------------
0.Lmediation_dummy  |  (base outcome)
--------------------+----------------------------------------------------------------
1.Lmediation_dummy  |
                _at |
                 1  |   258.6501    129.105     2.00   0.045     5.608927    511.6912
                 2  |   45.29828   252.3147     0.18   0.858    -449.2295    539.8261
-------------------------------------------------------------------------------------
Note: dy/dx for factor levels is the discrete change from the base level.
 
. marginsplot, recast(line)
 
  Variables that uniquely identify margins: Lgvt_rec_supp_intell

I then look at the predicted probabilities, which is commonly done in my field. And here it appears that the presence of mediation and intel is actually increasing the number of killings. This is surprising as the interaction term was negative and significant.

 . margins, at(Lmediation_dummy=(0 1) Lgvt_rec_supp_intell=(0 1))
 
Predictive margins                              Number of obs     =        125
Model VCE    : Robust
 
Expression   : Predicted number of events, predict()
 
1._at        : Lmediation_dummy=           0
               Lgvt_rec_supp_intell=           0
 
2._at        : Lmediation_dummy=           0
               Lgvt_rec_supp_intell=           1
 
3._at        : Lmediation_dummy=           1
               Lgvt_rec_supp_intell=           0
 
4._at        : Lmediation_dummy=           1
               Lgvt_rec_supp_intell=           1
 
------------------------------------------------------------------------------
             |            Delta-method
             |     Margin   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         _at |
          1  |   39.34037   28.46128     1.38   0.167     -16.4427    95.12345
          2  |     411.46   190.0587     2.16   0.030     38.95172    783.9683
          3  |   297.9904   146.2778     2.04   0.042     11.29125    584.6896
          4  |   456.7583   275.1427     1.66   0.097    -82.51139     996.028
------------------------------------------------------------------------------
 
. marginsplot, recast(line)

While I know that one cannot meaningfully interpret the interaction term in the regression table, I thought that the direction indicates that the presence of intell decreases the marginal effect of mediation on killings.

I am basically still trying to understand the dydx command (what do the results mean in regard to the rate of change on the predicted probabilities?), why the coefficient term and the predicted probabilities appear to contradict each other and what these different results mean for my hypothesis.
I could just follow what other scholars do in my field but I have the feeling that interaction terms are often misunderstood and therefore, I would like to really grasp how Stata is dealing with them in non-linear models.

Any help is much appreciated. Thank you.

[Clyde Schechter]

You have to focus on the difference between "intel increases/decreases the frequency of killings" and "intel increases/decreases the strength of the relationship between mediation and killings." Those things can be in opposite directions. Your hypothesis is explicitly about the latter, not the former. So the directly relevant -margins- command is the one with dydx(). That command calculates the expected difference in killings between mediated and non-mediated entities when intel is absent and the expected difference in killings between mediate and non-mediated entities when intel is present. Those are what the two output rows in the table you get from that command show you. When intel is absent, on average, the presence of mediation is associated with an increase of about 259 killings, whereas when intel is present, on average, the presence of mediation is associated with only an increase of about 45 killings. (I'm rounding everything throughout to the nearest integer.)

The results of the other -margins- command are not directly relevant to this hypothesis. They instead present the expected number of killings in each of the four combinations of intel and mediation. However, even here, you can reproduce the results in the -dydx()- table by noticing that comparing the number of killings with and without mediation when intel is absent (at_3 vs at_1 in the table)you see 298 killings vs 39, which is an increase of 259, and when intel is present there are 457 killings with mediation and 411 without, a difference of 45.

So the tables actually provide the same answer, although it is more directly presented and easier to see in the -dydx()- table.

[Nadine OShea]

Dear Clyde,

thank you so much for your input, I appreciate your help!

I understand what you are writing but I am still wondering about one thing: As you wrote, the dydx results show that when intel is present, the presence of mediation is associated with an increase of 'only' 45 killings instead of the 259 when intel is absent. However, I do not see this decrease in the predicted probabilities. I actually see the opposite because with mediation (and intel is absent), there are about 298 killings (_at 3) while when both mediation and intel are present, we can observe about 456 killings (_at 4), so this is an actual increase.

I would like to compare what happens to the effect of mediation with and without the presence of intel and therefore, I focus on _at 3 and _at 4, which show the increase. I am wondering how the presence of intel can have a moderating and decreasing effect on the positive correlation between mediation and killings (the dydx command, from 259 to 45 killings) but at the same time, the predicted probabilities increase when intel is present (from 298 to 456). I think that I do understand the difference now between the commands but logically, I cannot comprehend how intel can decrease the positive relation between mediation and killings while at the same time, the predicted numbers show that the actual number of killings increase even further.

Initially, I wanted to generate the following hypothesis: Intel decreases the number of killings during mediation processes (prior research finds that mediation increases killings and I have the same results but I would like to show that with the presence of intel the number of killings decreases ). Here, the predicted probabilities would be suitable. But then, I got reminded that the interaction term does not show an actual decrease because b1(2.02) + b3(-1.92)=0.1 (it is still positive) and that one cannot really talk about the decrease/increase of the frequency with interaction terms anyways. That confused me and I chose to be more modest by generating this hypothesis: Intel decreases the positive relationship between mediation and killings"

So, I am not focusing on cases with and without mediation but on mediation with and without intel.

Thank you for your help!

[Clyde Schechter]

I would like to compare what happens to the effect of mediation with and without the presence of intel and therefore, I focus on _at 3 and _at 4, which show the increase. I am wondering how the presence of intel can have a moderating and decreasing effect on the positive correlation between mediation and killings (the dydx command, from 259 to 45 killings) but at the same time, the predicted probabilities increase when intel is present (from 298 to 456).

Yes, they can, and that is precisely what is happening in your data. By definition, the effect of mediation, or relationship of mediation to killings (these are the same thing) is the difference between killings with and killings without mediation. That effect of mediation is positive both with and without intel. But it is much smaller with intel than without it. Therefore intel reduces, or weakens, the effect of mediation; it is a negative moderation.

The comparison of 3_at with 4_at is irrelevant to your stated hypothesis: "The presence of external intelligence moderates the positive relationship between mediation and government violence against civilians such that the relationship is weakened when the government receives external intelligence but is strengthened when the government does not receive external intelligence." The comparison of 3_at with 4_at is something entirely different: it is the actual effect of intel itself under the condition that there is mediation. And yes, that effect is positive. There is no inconsistency between this finding and the one in the previous paragraph. They are both correct answers to two entirely different questions.

[Nadine OShea]

Dear Clyde,

thank you again for your answer. I think this example really shows that merely looking at the interaction term in the regression table can be very misleading. Just from the coefficient, I would have expected that the presence of intel during mediation is associated with a decrease of the actual number of killings (if I wanted to answer my initial hypothesis) but the predicted probabilities show the opposite.

Since scholars in my field usually report the predicted probabilities, I might to stick to my initial hypothesis and reject it but I would still report the results for the rate of the change as I think that this result is also valuable (and this is the actual marginal effect).

Thank you for taking the time!