Predictive Margins Interpretation Between Interaction Terms

[Cat Santos]

Hello,

I am trying to interpret the results of my predictive margins between the interaction of two binary variables: treatment and subsidy. My goal is to examine the effects on the dependent variable when both variables are present or absent, and when each is present/absent and the other one is not.

Question 1: If I understood the command correctly, the predicted margins should give me the predicted value of the dependent variable (dv2), when treatment is 1 and subsidy is 1, when treatment is 0 and subsidy is 0, and so on. Is this interpretation correct?

Question 2: How can I interpret the results of the predicted margins I got, considering that in the logit model treatment is not significant and subsidy is? And why are the pvalues all significant?

Thank you very much!

. logit dv2 i.treatment##i.subsidy age male i.country_co i.identity, r

Iteration 0:  Log pseudolikelihood = -1462.3576  
Iteration 1:  Log pseudolikelihood = -1404.8565  
Iteration 2:  Log pseudolikelihood = -1404.5085  
Iteration 3:  Log pseudolikelihood = -1404.5084  

Logistic regression                                     Number of obs =  2,204
                                                        Wald chi2(10) = 107.09
                                                        Prob > chi2   = 0.0000
Log pseudolikelihood = -1404.5084                       Pseudo R2     = 0.0396

-------------------------------------------------------------------------------------
                    |               Robust
                dv2 | Coefficient  std. err.      z    P>|z|     [95% conf. interval]
--------------------+----------------------------------------------------------------
        1.treatment |  -.1163288   .4771637    -0.24   0.807    -1.051552    .8188949
        1.subsidy |  -.2296406   .0945648    -2.43   0.015    -.4149842   -.0442969
                    |
treatment#subsidy |
               1 1  |  -.2295296   .5884806    -0.39   0.697     -1.38293    .9238711
                    |
           age |   .0103576   .0029747     3.48   0.000     .0045273    .0161878
               male |   .0621989   .0906214     0.69   0.492    -.1154158    .2398136
                    |
         country_co |
                 2  |   .2326212   .2389167     0.97   0.330     -.235647    .7008894
                 3  |   .0429415     .27989     0.15   0.878    -.5056328    .5915158
                    |
           identity |
                 2  |   .0954899   .2697841     0.35   0.723    -.4332773    .6242571
                 3  |    .954601    .305423     3.13   0.002     .3559829    1.553219
                 4  |  -.3116946   .2088277    -1.49   0.136    -.7209894    .0976002
                    |
              _cons |  -20.21188   5.871116    -3.44   0.001    -31.71905   -8.704701
-------------------------------------------------------------------------------------

. margins i.subsidy#i.treatment

Predictive margins                                       Number of obs = 2,204
Model VCE: Robust

Expression: Pr(dv2), predict()

-------------------------------------------------------------------------------------
                    |            Delta-method
                    |     Margin   std. err.      z    P>|z|     [95% conf. interval]
--------------------+----------------------------------------------------------------
subsidy#treatment |
               0 0  |   .6486665   .0146136    44.39   0.000     .6200243    .6773086
               0 1  |   .6230715   .1049369     5.94   0.000      .417399     .828744
               1 0  |   .5975045   .0146136    40.89   0.000     .5688624    .6261467
               1 1  |   .5169178   .0892162     5.79   0.000     .3420572    .6917784
-------------------------------------------------------------------------------------

[Clyde Schechter]

Question 1: The output of the -margins- command shows you the model-predicted probability of dv2 in each of the four combinations of treatment and subsidy. The statistical significance of these values is meaningless, and it would have been a better design for the command if Stata had simply suppressed that part of the output. Remember what it means to say that a parameter estimate is statistically significant. It means that you reject the null hypothesis that the parameter's value is zero (or 1 in case of a ratio). The very concept of statistical significance, in general, is subject to the criticism that the null hypothesis is almost always a straw man. In this particular context, it is clearly a straw man. There is no situation in which you would seriously entertain the hypothesis that the probability of dv2 is 0. You can reject that hypothesis by simply observing once, ever, a situation where dv2 is true. So all those p-values are telling you is that the data are not compatible with the "true" probabilities of dv2 in these situations is zero. But you knew that before you ever started this research project. It is simply applying the framework of null hypothesis significance testing in a context where nobody should take it seriously.

Question 2: The lack of statistical significance of the treatment variable in the logistic regression output needs careful interpretation. The first thing is to understand that the coefficient of treatment in this interaction model is not the effect of treatment. It is the effect (in the log-odds metric) of treatment in the circumstance where subsidy = 0. Similarly, the coefficient of subsidy is not the effect of subsidy. It is the effect of subsidy in the circumstance where treatment = 0. In a model containing an X#Y interaction, there is no such thing as "the effect of X," nor "the effect of Y." The very use of an interaction model is a stipulation that X has different effects depending on the value of Y, and vice versa. To see those, I recommend that you run different -margins- commands:

margins treatment, dydx(subsidy)
margins subsidy, dydx(treatment)

The output will give you the marginal effects of subsidy and treatment conditional on each of the values of the other variable. One caveat: the output of -margins- will be in the probability metric, not the log odds metric, so it is not directly comparable to the logistic regression output you have. Note that the idea that a treatment or subsidy effect might be 0 is not (or, at least, not necessarily) a straw man, so that if you take null hypothesis significance testing seriously, it is appropriate to apply it to these outputs.

[Cat Santos]

Dear Clyde,

Thank you so much for your incredibly helpful and comprehensive answer to both of my questions. I learnt a lot from what you wrote.

I did run the code you provided but I am finding the interpretation a little changelling as this is my first time using this command.

. margins subsidy, dydx(treatment)

Average marginal effects                                 Number of obs = 2,204
Model VCE: Robust

Expression: Pr(dv2), predict()
dy/dx wrt:  1.treatment

------------------------------------------------------------------------------
             |            Delta-method
             |      dy/dx   std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
0.treatment  |  (base outcome)
-------------+----------------------------------------------------------------
1.treatment  |
   subsidy |
          0  |   -.025595   .1063407    -0.24   0.810    -.2340189     .182829
          1  |  -.0805868   .0902358    -0.89   0.372    -.2574457    .0962722
------------------------------------------------------------------------------
Note: dy/dx for factor levels is the discrete change from the base level.

. margins treatment, dydx(subsidy)

Average marginal effects                                 Number of obs = 2,204
Model VCE: Robust

Expression: Pr(dv2), predict()
dy/dx wrt:  1.subsidy

------------------------------------------------------------------------------
             |            Delta-method
             |      dy/dx   std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
0.subsidy  |  (base outcome)
-------------+----------------------------------------------------------------
1.subsidy  |
   treatment |
          0  |  -.0511619    .020985    -2.44   0.015    -.0922917   -.0100321
          1  |  -.1061537   .1325001    -0.80   0.423    -.3658491    .1535416
------------------------------------------------------------------------------
Note: dy/dx for factor levels is the discrete change from the base level.

If I understood correctly, in the second margins model (margins treatment, dydx(subsidy)), when subsidies are distributed (=1), the absence of the treatment (=0) lowers the dependent variable when compared to when subsidies are not distributed, and this is statistically significant. Can broady say that when subsides are distributed, they only impact the DV if the treatment doesn't happen?

And in the first model, that when the treatment happens, the effects of subsidies are irrelevant to change the DV? Apologies in advance, I'm aware I'm probably misinterpreting part or all of it.

Once again, thank you so much for taking the time to be so helpful!

Best regards,
Cat

[Clyde Schechter]

Well, you have it backwards. For -margins treatment, dydx(subsidy)- the interpretation is that when treatment is 0, distribution of subsidy (subsidy = 1) is associated with a decrease in expected value of dv2 by 0.05 (95% CI 0.01 to 0.09), whereas if treatment is 1, the effect of distribution of subsidy is -.11 (95% CI -0.37 to +0.15).

With regard to the first output, -margins subsidy, dydx(treatment)-, here the numbers shown are the marginal effects of treatment in the presence or absence of subsidy distribution--the reverse of the second model. Here, for both values of subsidy the confidence intervals contain 0, so the data does not provide precise enough estimates to determine whether the effect of treatment on dv2 is positive, negative, or 0. The statement that treatment is irrelevant takes this too far. Unless you have a prior power analysis that supports that way of interpreting things, the conclusion is that this part of your study is inconclusive.

[Cat Santos]

Dear Clyde,

I got it now thanks to your explanation.

Once again thank you so much for your precious help!

Best wishes,
Cat