Interpretation of triple interaction

[Nicolas Orgeira]

Hi,

I am looking at the heterogeneous effects of an intervention (three groups - 0/1/2) on gender (female=0/1) and would need some help with interpreting the results:

• My understanding is that the coefficients 1.intervention#1.time and 2.intervention#1.time are the DID coefficients for the two interventions (.7104208 and 1.464891 respectively), with their p-values indicating whether they statistically differ with the coefficient 0.intervention#1.time.
• And that the coefficient 1.intervention#1.gender#1.time (-2.308946) is the difference in effect between male and female who received the first intervention, and so the actual coefficient for 1.intervention#1.gender#1.time is actually 1.intervention#1.time + 1.intervention#1.gender#1.time (=-1.598526, computed below using lincom).

I was wondering whether the p-value in the regression table for coefficient 1.intervention#1.gender#1.time indicates a statistical difference between 1.intervention#1.gender#1.time and 1.intervention#1.time (as would lincom also compute), or the statistical difference between 1.intervention#1.gender#1.time and 0.intervention#1.time. Is there a way to determine whether 1.intervention#1.gender#1.time and 0.intervention#1.time differ statistically, without using margins? Thank you

. svy: ologit righttotax i.intervention##i.female##i.time
(running ologit on estimation sample)

Survey: Ordered logistic regression

Number of strata   =         7                  Number of obs     =      2,340
Number of PSUs     =     1,207                  Population size   = 7,395.2248
                                                Design df         =      1,200
                                                F(  11,   1190)   =       4.02
                                                Prob > F          =     0.0000

-------------------------------------------------------------------------------------
                    |             Linearized
         righttotax |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
--------------------+----------------------------------------------------------------
       intervention |
                 1  |   .2217146   .5188795     0.43   0.669    -.7962973    1.239727
                 2  |  -.7342184   .3209357    -2.29   0.022    -1.363876    -.104561
                    |
             female |
            Female  |  -.5882525   .3920571    -1.50   0.134    -1.357446    .1809411
                    |
intervention#female |
          1#Female  |   .1602964   .8438967     0.19   0.849    -1.495381    1.815974
          2#Female  |   .6699956    .509731     1.31   0.189    -.3300674    1.670059
                    |
               time |
              2017  |   .0938864   .3025162     0.31   0.756    -.4996331    .6874059
                    |
  intervention#time |
            1#2017  |   .7104208   .9145775     0.78   0.437    -1.083928     2.50477
            2#2017  |   1.464891   .5294928     2.77   0.006     .4260561    2.503725
                    |
        female#time |
       Female#2017  |   2.722754   1.106064     2.46   0.014     .5527191    4.892789
                    |
       intervention#|
        female#time |
     1#Female#2017  |  -2.308946   1.837614    -1.26   0.209    -5.914239    1.296346
     2#Female#2017  |  -1.609661   1.350826    -1.19   0.234    -4.259905    1.040583
--------------------+----------------------------------------------------------------
              /cut1 |  -3.520324   .2437719                     -3.998591   -3.042057
              /cut2 |  -2.720061   .2184403                     -3.148629   -2.291494
-------------------------------------------------------------------------------------

. lincom 1.intervention#1.time+1.intervention#1.female#1.time

 ( 1)  [righttotax]1.intervention#1.time + [righttotax]1.intervention#1.female#1.time
       = 0

------------------------------------------------------------------------------
  righttotax |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |  -1.598526   1.586276    -1.01   0.314    -4.710708    1.513657
------------------------------------------------------------------------------

. lincom 2.intervention#1.time+2.intervention#1.female#1.time

 ( 1)  [righttotax]2.intervention#1.time + [righttotax]2.intervention#1.female#1.time
       = 0

------------------------------------------------------------------------------
  righttotax |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |  -.1447704   1.238054    -0.12   0.907    -2.573762    2.284222
------------------------------------------------------------------------------

[Clyde Schechter]

My understanding is that the coefficients 1.intervention#1.time and 2.intervention#1.time are the DID coefficients for the two interventions (.7104208 and 1.464891 respectively), with their p-values indicating whether they statistically differ with the coefficient 0.intervention#1.time.

No. In this model with a three way interaction, there is no such thing as the coefficient for either of the interventions. Each intervention has two different coefficients, one for males and one for females. The coefficients 1.intervention#1.time and 2.intervention#1.time are the coefficients (relative to intervention 0 ) among males (female == 0) only.

And that the coefficient 1.intervention#1.gender#1.time (-2.308946) is the difference in effect between male and female who received the first intervention, and so the actual coefficient for 1.intervention#1.gender#1.time is actually 1.intervention#1.time + 1.intervention#1.gender#1.time (=-1.598526, computed below using lincom).

Your model does not have a variable called gender. I assume you meant to refer to the model variable female. The coefficient of 1.intervention#1.female#1.time is, indeed the difference between male and female coefficients of the first intervention, so that the "actual coefficient" for females is, indeed, 1.intervention#1.time + 1.intervention#1.female#1.time.

I was wondering whether the p-value in the regression table for coefficient 1.intervention#1.gender#1.time indicates a statistical difference between 1.intervention#1.gender#1.time and 1.intervention#1.time (as would lincom also compute), or the statistical difference between 1.intervention#1.gender#1.time and 0.intervention#1.time.

No. If you have any three way interaction x##y##z, then the coefficient of 1.x#1.y#1.z is not, itself the difference between xb for any subsets defined by values of x, y, and z. The reason is that, for example, if you wanted to compare it with x = 0, y = 0, z = 1, in addition to considering 1.z#1.y#1.z, you also have to add in 1.x, 1.y, and 1.x#1.y coefficients.

Trying to do all of these interpretations tends to make one's head spin. And the probability of getting it all right using a bunch of -lincom- commands is low, even for people who are pretty experienced at this. I strongly recommend you use -margins, dydx()- for this. It'll save you a lot of time, and more importantly, it will save you from a lot of mistakes.

And whatever you do, always remember that with a 3-way interaction in the model (x##y##z) there are no such things as effects of particular values of x, y, z, x&y, y&z, or x&z in any particular combinations of values. There are only separate effects for every combination of x, y, and z values. (From -margins- you can get average effects for combinations of values not specifying all three variables, but that is a different animal.)

[Nicolas Orgeira]

Hi Clyde,

Thank you for your help and your detailed responses. For publication purposes, would you then recommend to report the main results from the DID (that is, 1.intervention#1.time, 2.intervention#1.time, 1.intervention#1.female#1.time and 1.intervention#1.female#1.time, with the latter two computed using lincom to get the actual coefficients) as per the regression? Or report the margins results? The main results I'd like to present are:

• the treatment effects of intervention 1 (1.intervention#1.time) and intervention 2 (2.intervention#1.time) on male respondents
• the treatment effects of each intervention on female respondents (1.intervention#1.time+1.intervention#1.female#1.t ime) and (2.intervention#1.time+2.intervention#1.female#1.t ime)
• Determine whether there is a statistical difference between 0.intervention#1.time and 1.intervention#1.time, 0.intervention#1.time and 2.intervention#1.time, and 1.intervention#1.time and 2.intervention#1.time
• Determine whether there is a statistical difference between (2.intervention#1.time+2.intervention#1.female#1.t ime) and (1.intervention#1.time+1.intervention#1.female#1.t ime)

However, using the average marginal effects, I can only determine the predicted probability for specific values of righttotax, by intervention, and not the average effect (I'm reluctant to use atmeans, based on Richard William's file), while using pwcompare does not seem to provide information on statistical differences between group (see below).

Thank you again for your help

. margins i.intervention#i.female#i.time

Adjusted predictions                            Number of obs     =      2,340
Model VCE    : Linearized

1._predict   : Pr(righttotax==1), predict(pr outcome(1))
2._predict   : Pr(righttotax==2), predict(pr outcome(2))
3._predict   : Pr(righttotax==3), predict(pr outcome(3))

----------------------------------------------------------------------------------
                 |            Delta-method
                 |     Margin   Std. Err.      t    P>|t|     [95% Conf. Interval]
-----------------+----------------------------------------------------------------
        _predict#|
    intervention#|
     female#time |
  1#0#Male#2012  |   .0287394   .0068045     4.22   0.000     .0153894    .0420895
  1#0#Male#2017  |   .0262316   .0068248     3.84   0.000     .0128418    .0396213
1#0#Female#2012  |   .0505907   .0156952     3.22   0.001     .0197977    .0813838
1#0#Female#2017  |   .0031768   .0032342     0.98   0.326    -.0031686    .0095221
  1#1#Male#2012  |   .0231568   .0115736     2.00   0.046     .0004501    .0458635
  1#1#Male#2017  |   .0104946   .0071389     1.47   0.142    -.0035115    .0245007
1#1#Female#2012  |   .0350912   .0190462     1.84   0.066    -.0022764    .0724588
1#1#Female#2017  |   .0106425   .0104129     1.02   0.307    -.0097869     .031072
  1#2#Male#2012  |   .0580796   .0148202     3.92   0.000     .0290033    .0871559
  1#2#Male#2017  |   .0128069   .0046882     2.73   0.006     .0036089    .0220049
1#2#Female#2012  |    .053766   .0128805     4.17   0.000     .0284953    .0790367
1#2#Female#2017  |   .0039123   .0023618     1.66   0.098    -.0007214     .008546
  2#0#Male#2012  |   .0330605   .0072013     4.59   0.000     .0189319    .0471891
  2#0#Male#2017  |   .0303435   .0066613     4.56   0.000     .0172744    .0434127
2#0#Female#2012  |   .0554527   .0165607     3.35   0.001     .0229615    .0879438
2#0#Female#2017  |   .0038677   .0038714     1.00   0.318    -.0037278    .0114631
  2#1#Male#2012  |   .0269699   .0112643     2.39   0.017     .0048699    .0490698
  2#1#Male#2017  |   .0125709   .0090735     1.39   0.166    -.0052308    .0303726
2#1#Female#2012  |   .0398038   .0218538     1.82   0.069    -.0030722    .0826798
2#1#Female#2017  |   .0127439   .0125956     1.01   0.312     -.011968    .0374558
  2#2#Male#2012  |   .0626177   .0126634     4.94   0.000     .0377728    .0874626
  2#2#Male#2017  |   .0152621   .0053245     2.87   0.004     .0048158    .0257084
2#2#Female#2012  |   .0585214   .0119318     4.90   0.000     .0351118     .081931
2#2#Female#2017  |   .0047554   .0028318     1.68   0.093    -.0008005    .0103112
  3#0#Male#2012  |   .9382001   .0126653    74.08   0.000     .9133515    .9630487
  3#0#Male#2017  |   .9434249   .0123474    76.41   0.000        .9192    .9676498
3#0#Female#2012  |   .8939566   .0305997    29.21   0.000     .8339218    .9539915
3#0#Female#2017  |   .9929556    .007073   140.39   0.000     .9790788    1.006832
  3#1#Male#2012  |   .9498734   .0223641    42.47   0.000     .9059962    .9937505
  3#1#Male#2017  |   .9769345   .0160721    60.78   0.000      .945402    1.008467
3#1#Female#2012  |    .925105   .0402961    22.96   0.000     .8460463    1.004164
3#1#Female#2017  |   .9766136   .0229004    42.65   0.000     .9316843    1.021543
  3#2#Male#2012  |   .8793027   .0248942    35.32   0.000     .8304616    .9281437
  3#2#Male#2017  |    .971931   .0096383   100.84   0.000     .9530213    .9908407
3#2#Female#2012  |   .8877126   .0222964    39.81   0.000     .8439683    .9314569
3#2#Female#2017  |   .9913324   .0051256   193.41   0.000     .9812763    1.001388
----------------------------------------------------------------------------------

.
. margins intervention, dydx(time)

Average marginal effects                        Number of obs     =      2,340
Model VCE    : Linearized

dy/dx w.r.t. : 1.time
1._predict   : Pr(righttotax==1), predict(pr outcome(1))
2._predict   : Pr(righttotax==2), predict(pr outcome(2))
3._predict   : Pr(righttotax==3), predict(pr outcome(3))

------------------------------------------------------------------------------
             |            Delta-method
             |      dy/dx   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
0.time       |  (base outcome)
-------------+----------------------------------------------------------------
1.time       |
    _predict#|
intervention |
        1 0  |  -.0181933   .0074945    -2.43   0.015    -.0328971   -.0034895
        1 1  |  -.0167791     .01207    -1.39   0.165    -.0404599    .0069016
        1 2  |  -.0468729     .01145    -4.09   0.000     -.069337   -.0244087
        2 0  |  -.0197862   .0083083    -2.38   0.017    -.0360866   -.0034858
        2 1  |  -.0188214   .0125245    -1.50   0.133    -.0433937     .005751
        2 2  |  -.0495948   .0100664    -4.93   0.000    -.0693444   -.0298451
        3 0  |   .0379795   .0153716     2.47   0.014     .0078212    .0681377
        3 1  |   .0356005   .0243908     1.46   0.145    -.0122528    .0834538
        3 2  |   .0964676   .0193627     4.98   0.000      .058479    .1344562
------------------------------------------------------------------------------
Note: dy/dx for factor levels is the discrete change from the base level.

.
. margins time, dydx(intervention)

Average marginal effects                        Number of obs     =      2,340
Model VCE    : Linearized

dy/dx w.r.t. : 1.intervention 2.intervention
1._predict   : Pr(righttotax==1), predict(pr outcome(1))
2._predict   : Pr(righttotax==2), predict(pr outcome(2))
3._predict   : Pr(righttotax==3), predict(pr outcome(3))

---------------------------------------------------------------------------------
                |            Delta-method
                |      dy/dx   Std. Err.      t    P>|t|     [95% Conf. Interval]
----------------+----------------------------------------------------------------
0.intervention  |  (base outcome)
----------------+----------------------------------------------------------------
1.intervention  |
  _predict#time |
        1#2012  |  -.0090466   .0117742    -0.77   0.442    -.0321468    .0140537
        1#2017  |  -.0076324   .0075707    -1.01   0.314    -.0224856    .0072208
        2#2012  |  -.0094292   .0125693    -0.75   0.453    -.0340896    .0152311
        2#2017  |  -.0084644   .0080595    -1.05   0.294    -.0242766    .0073478
        3#2012  |   .0184758   .0242667     0.76   0.447    -.0291341    .0660857
        3#2017  |   .0160968   .0155627     1.03   0.301    -.0144363    .0466299
----------------+----------------------------------------------------------------
2.intervention  |
  _predict#time |
        1#2012  |   .0202009   .0118882     1.70   0.090    -.0031231    .0435249
        1#2017  |  -.0084786   .0051738    -1.64   0.102    -.0186293    .0016721
        2#2012  |    .020305   .0108307     1.87   0.061    -.0009443    .0415543
        2#2017  |  -.0095035   .0056097    -1.69   0.090    -.0205094    .0015023
        3#2012  |   -.040506   .0223728    -1.81   0.070    -.0844001    .0033881
        3#2017  |   .0179822    .010639     1.69   0.091     -.002891    .0388553
---------------------------------------------------------------------------------
Note: dy/dx for factor levels is the discrete change from the base level.


. margins female, dydx(time intervention) pwcompare

Pairwise comparisons of average marginal effects

Number of strata   =         7                  Design df         =      1,200
Model VCE    : Linearized

dy/dx w.r.t. : 1.intervention 2.intervention 1.time
1._predict   : Pr(righttotax==1), predict(pr outcome(1))
2._predict   : Pr(righttotax==2), predict(pr outcome(2))
3._predict   : Pr(righttotax==3), predict(pr outcome(3))


Contrast Delta-method         Unadjusted
dy/dx   Std. Err.     [95% Conf. Interval]

0.intervention              (base outcome)

1.intervention            
_predict#female
(1#Female) vs (1#Male)     .0085658   .0145121     -.0199061    .0370376
(2#Male) vs (1#Male)    -.0013605   .0023846     -.0060389    .0033179
(2#Female) vs (1#Male)     .0092869   .0150541     -.0202485    .0388222
(3#Male) vs (1#Male)     .0351085   .0238458     -.0116755    .0818926
(3#Female) vs (1#Male)     .0158954   .0269925     -.0370624    .0688532
(2#Male) vs (1#Female)    -.0099263   .0153672     -.0400759    .0202234
(2#Female) vs (1#Female)     .0007211   .0010979     -.0014329    .0028751
(3#Male) vs (1#Female)     .0265428   .0204158     -.0135119    .0665974
(3#Female) vs (1#Female)     .0073296   .0381296     -.0674785    .0821378
(2#Female) vs (2#Male)     .0106474   .0158638     -.0204765    .0417713
(3#Male) vs (2#Male)      .036469   .0246227     -.0118394    .0847775
(3#Female) vs (2#Male)     .0172559    .026752       -.03523    .0697418
(3#Male) vs (2#Female)     .0258217   .0207717     -.0149312    .0665745
(3#Female) vs (2#Female)     .0066085   .0387306     -.0693788    .0825958
(3#Female) vs (3#Male)    -.0192131    .030301     -.0786619    .0402356

2.intervention            
_predict#female
(1#Female) vs (1#Male)    -.0036612   .0115846     -.0263895     .019067
(2#Male) vs (1#Male)    -.0008286   .0012541     -.0032891    .0016318
(2#Female) vs (1#Male)    -.0036234   .0114824     -.0261512    .0189045
(3#Male) vs (1#Male)    -.0155963   .0227629     -.0602558    .0290633
(3#Female) vs (1#Male)    -.0091403   .0190055      -.046428    .0281474
(2#Male) vs (1#Female)     .0028326   .0117371     -.0201948      .02586
(2#Female) vs (1#Female)     .0000379   .0006693     -.0012753     .001351
(3#Male) vs (1#Female)     -.011935    .017522     -.0463121     .022442
(3#Female) vs (1#Female)    -.0054791   .0261928     -.0568679    .0459098
(2#Female) vs (2#Male)    -.0027947    .011599     -.0255514    .0199619
(3#Male) vs (2#Male)    -.0147676   .0228328     -.0595643     .030029
(3#Female) vs (2#Male)    -.0083117   .0189248      -.045441    .0288176
(3#Male) vs (2#Female)    -.0119729   .0174264     -.0461624    .0222166
(3#Female) vs (2#Female)    -.0055169   .0260221     -.0565707    .0455368
(3#Female) vs (3#Male)      .006456   .0231587       -.03898    .0518919

0.time                      (base outcome)

1.time                    
_predict#female
(1#Female) vs (1#Male)    -.0210275   .0113146     -.0432262    .0011712
(2#Male) vs (1#Male)    -.0013156   .0041428     -.0094436    .0068124
(2#Female) vs (1#Male)    -.0246808   .0149522     -.0540161    .0046546
(3#Male) vs (1#Male)     .0645397   .0214465      .0224628    .1066165
(3#Female) vs (1#Male)     .1089323   .0202581      .0691871    .1486776
(2#Male) vs (1#Female)     .0197119   .0117502     -.0033414    .0427652
(2#Female) vs (1#Female)    -.0036533   .0085311     -.0203908    .0130842
(3#Male) vs (1#Female)     .0855671   .0178751      .0504971    .1206372
(3#Female) vs (1#Female)     .1299598   .0282435      .0745477    .1853719
(2#Female) vs (2#Male)    -.0233652   .0128052     -.0484883     .001758
(3#Male) vs (2#Male)     .0658552   .0205298       .025577    .1061335
(3#Female) vs (2#Male)     .1102479   .0205294      .0699704    .1505254
(3#Male) vs (2#Female)     .0892204    .016825      .0562108      .12223
(3#Female) vs (2#Female)     .1336131   .0297104       .075323    .1919031
(3#Female) vs (3#Male)     .0443927   .0237421      -.002188    .0909733

Note: dy/dx for factor levels is the discrete change from the base level.

[Clyde Schechter]

So the fact that you have an ordinal outcome with 3 levels multiplies the complications still farther. It's going to be very difficult to make sense out of that. Moreover, the relationship between the dydx() in the xb (coefficient) metric does not correspond in any straightforward way to the dydx() in the probability metric. The xb metric is pretty opaque to most people; probabilities are easier to understand.

You cannot condition these marginal effects on the value of righttotax, because righttotax is your outcome variable, and there are many combinations of the explanatory variables that can get you to the same probability of righttotax--consequently those would correspond to different marginal effects. If I am understanding what you want correctly, you would get it with:

margins female#time, dydx(intervention) predict(pr)
margins female#time, dydx(intervention) predict(pr) pwcompare

That's an enormous amount of output, some of which is not relevant to the specific comparisons you are interested in, and I don't know how you will organize it in a way that can be digested by mere mortals, but you are welcome to try. You might consider making liberal use of -marginsplot- to show this graphically--I think that will be easier to understand than tables with zillions of numbers.

[Nicolas Orgeira]

Thank you very much Clyde, that's exactly what I was looking for (I will use the codes for non-ordinal outcome variables). For the righttotax variable, I will follow your advice and use a graphical method to present the results.