Announcement

Collapse
No announcement yet.
X
Collapse
  • Page of 1
  • Filter
  • Time
  • Show
Clear All
new posts
  • #1

    Please help me interpretate the interaction effects of my model or: Why are three-way interactions significant, but two-way are not?

    Hello Stata community,

    currently I examine if firms provide bullshit news disclosures.


    To examine this I used the following codes:
    Code:
    xtset FIRM Year

Three-way model:
reghdfe Bullshit x1 x2 x3 x4 x6 LARGE##POLLUTER##CONTROVERSY, absorb(FIRM Year) cluster(FIRM)


    Code:
    HDFE Linear regression                            Number of obs   =        816
Absorbing 2 HDFE groups                           F(  11,    218) =      21.63
Statistics robust to heteroskedasticity           Prob > F        =     0.0000
                                                  R-squared       =     0.6360
                                                  Adj R-squared   =     0.4894
                                                  Within R-sq.    =     0.0284
Number of clusters (FIRM)    =        219         Root MSE        =     0.0128

                                               (Std. err. adjusted for 219 clusters in FIRM)
--------------------------------------------------------------------------------------------
                           |               Robust
                  Bullshit | Coefficient  std. err.      t    P>|t|     [95% conf. interval]
---------------------------+----------------------------------------------------------------
                        x1 |   2.18e-07   1.15e-07     1.90   0.059    -8.48e-09    4.44e-07
                        x2 |   .0000792   .0000704     1.13   0.261    -.0000595    .0002179
                        x3 |  -.0011349   .0013584    -0.84   0.404    -.0038122    .0015424
                        x4 |          0  (omitted)
                        x6 |   .0001505   .0000955     1.58   0.117    -.0000377    .0003387
                   1.LARGE |   -.002641   .0039933    -0.66   0.509    -.0105115    .0052294
                1.POLLUTER |  -.0003113    .002018    -0.15   0.878    -.0042885     .003666
                           |
            LARGE#POLLUTER |
                      1 1  |   .0054256   .0040318     1.35   0.180    -.0025208    .0133719
                           |
             1.CONTROVERSY |   .0177814   .0024065     7.39   0.000     .0130384    .0225244
                           |
         LARGE#CONTROVERSY |
                      1 1  |  -.0179408   .0043084    -4.16   0.000    -.0264323   -.0094492
                           |
      POLLUTER#CONTROVERSY |
                      1 1  |  -.0160671   .0025222    -6.37   0.000    -.0210381   -.0110962
                           |
LARGE#POLLUTER#CONTROVERSY |
                    1 1 1  |   .0125052   .0059916     2.09   0.038     .0006962    .0243141
                           |
                     _cons |   .0624834   .0059209    10.55   0.000     .0508138     .074153
--------------------------------------------------------------------------------------------
    Code:
    Alternatively, the two way model using POLLUTER CONTROVERSY.

reghdfe Bullshit x1 x2 x3 x4 x6 POLLUTER##CONTROVERSY, absorb(FIRM Year) cluster(FIRM)

HDFE Linear regression                            Number of obs   =        816
Absorbing 2 HDFE groups                           F(   7,    218) =       1.45
Statistics robust to heteroskedasticity           Prob > F        =     0.1876
                                                  R-squared       =     0.6336
                                                  Adj R-squared   =     0.4895
                                                  Within R-sq.    =     0.0218
Number of clusters (FIRM)    =        219         Root MSE        =     0.0128

                                         (Std. err. adjusted for 219 clusters in FIRM)
--------------------------------------------------------------------------------------
                     |               Robust
            Bullshit | Coefficient  std. err.      t    P>|t|     [95% conf. interval]
---------------------+----------------------------------------------------------------
                  x1 |   2.21e-07   1.15e-07     1.93   0.055    -4.47e-09    4.47e-07
                  x2 |   .0000796   .0000713     1.12   0.265    -.0000608    .0002201
                  x3 |  -.0010632   .0013383    -0.79   0.428    -.0037009    .0015745
                  x4 |          0  (omitted)
                  x6 |   .0001539   .0000958     1.61   0.110    -.0000349    .0003427
          1.POLLUTER |   .0011961   .0018756     0.64   0.524    -.0025005    .0048927
       1.CONTROVERSY |   .0003586    .003288     0.11   0.913    -.0061218     .006839
                     |
POLLUTER#CONTROVERSY |
                1 1  |  -.0004579   .0037676    -0.12   0.903    -.0078835    .0069676
                     |
               _cons |   .0610246   .0058705    10.40   0.000     .0494545    .0725947
--------------------------------------------------------------------------------------

Absorbed degrees of freedom:
-----------------------------------------------------+
 Absorbed FE | Categories  - Redundant  = Num. Coefs |
-------------+---------------------------------------|
        FIRM |       219         219           0    *|
        Year |         5           0           5     |
-----------------------------------------------------+
* = FE nested within cluster; treated as redundant for DoF computation

    Interpretation:
    In the three-way model, various effects are siginificant.

    If the firm is large and is facing controversies, it provides less bullshit.
    If the firm is a polluter and faces controversies, it provides less bullshit.
    If the firm large, a polluter and faces controversies, it provides more bullshit.

    Are the threeway effects that simple to interprete? Or am I missing something important from the data?
    What does 1.Large , 1.Polluter and 1.Controversy describe?

    Second Question:
    Why are the interaction effects only siginificant in the three way model and not in the two-way model?


    Best regards
    Luca







    Last edited by Luca Haseney; 04 Jan 2023, 12:26.
    Tags: None
  • #2
    Luca:
    1) the best way to get familiar with interactions is to calculate the predicted values and then comparing their values with the ones calculated by hand based on the coefficients obtained from your regression;
    2)they describe, other things being equal, the main effect of each predictor included in the interaction when the other predictors are set at their zero level;
    3) probably -controversy- leads the dance but needs the boost provided by -large- to magnify its effect on the dependent variable.
    Kind regards,
    Carlo
    (Stata 19.0)

    Comment

    • #3
      No, your interpretations are incorrect. The simplest way to get correct results is to follow the (first) regression with:
      [code]
      margins LARGE#POLLUTER#CONTROVERSY
      [/code
      which will show you the expected value of your outcome variable in each of the 8 combinations of large, polluter, and controversy. If you want to do contrasts among these 8 levels, you can run the -margins- command again, this time adding the -pwcompare- option.

      For a clear and lucid explanation of how to interpret the coefficients in an interaction model, I recommend you read the excellent Richard Williams' ttps://www3.nd.edu/~rwilliam/stats2/l53.pdf (explanation of coefficients in simple interaction model).

      When you have understood the meaning of the different terms in the interaction model, you will also recognize why there is no necessary connection between the coefficient of A#B in a two-way interaction model and that of the (deceptively apparently) same A#B in a model that includes an A#B#C three-way interaaction.

      Added: Crossed with #2.
      • 1 like

      Comment

      • #4
        Dear Mr. Lazzarro and dear Mr. Schechter, thank you both for your response.

        I ran the margins command and I read the article, yet I still have trouble understanding it.

        Code:
        . margins, at(LARGE=(0 1) POLLUTER=(0 1) CONTROVERSY=(0 1)) noestimcheck

Predictive margins                                         Number of obs = 816
Model VCE: Robust

Expression: Linear prediction, predict()
1._at: LARGE       = 0
       POLLUTER    = 0
       CONTROVERSY = 0
2._at: LARGE       = 0
       POLLUTER    = 0
       CONTROVERSY = 1
3._at: LARGE       = 0
       POLLUTER    = 1
       CONTROVERSY = 0
4._at: LARGE       = 0
       POLLUTER    = 1
       CONTROVERSY = 1
5._at: LARGE       = 1
       POLLUTER    = 0
       CONTROVERSY = 0
6._at: LARGE       = 1
       POLLUTER    = 0
       CONTROVERSY = 1
7._at: LARGE       = 1
       POLLUTER    = 1
       CONTROVERSY = 0
8._at: LARGE       = 1
       POLLUTER    = 1
       CONTROVERSY = 1

------------------------------------------------------------------------------
             |            Delta-method
             |     Margin   std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
         _at |
          1  |   .0713448   .0021244    33.58   0.000     .0671811    .0755086
          2  |   .0891262   .0027973    31.86   0.000     .0836437    .0946088
          3  |   .0710336   .0025278    28.10   0.000     .0660792    .0759879
          4  |   .0727478   .0033441    21.75   0.000     .0661935    .0793021
          5  |   .0687038   .0020247    33.93   0.000     .0647354    .0726722
          6  |   .0685444   .0042684    16.06   0.000     .0601785    .0769103
          7  |   .0738181   .0033806    21.84   0.000     .0671923    .0804439
          8  |   .0700967   .0040141    17.46   0.000     .0622292    .0779642
------------------------------------------------------------------------------
        What exactly, in addition to the output from model 1, does the table on the bottom tell me? From this table it seems that everything is relevant.

        Additionally, i performed marginsplot:

        Click image for larger version Name: interactipon.jpg Views: 1 Size: 131.2 KB ID: 1696085



        So for me, it is easier to interprete this graph.
        Model 1 states that except large and polluter, all interactions effects are highly significant. (Right?)
        The marginsplot lign tells me:
        If a firm is not a polluter, runs into a controversy and is big, it provides less "fake" news. And this interaction effect is very large.
        On the other hand, if the comapny is a polluter (green and orange), controversy leads to less fake news, while no controversy leads to more fake news.

        Is this line of thought correct? Please excuse me if it is not.

        Comment

        • #5
          Dear Stata Community,

          I reran some of the regressions and also added some further analyses. This is cross-posted here:
          https://stats.stackexchange.com/ques...ion-but-insign

          currently I examine if firm characteristics have an effect on firm communication. ESG, LARGE and CONTROVERSY are all indicator variables, set to zero or one.

          1. I have big troubles interpreting my results.
          2. I do not understand why my two way effects are insignificant, whereas the threeway models are significant.
          3. I do not know if my subsequent testing is accurate and why the tukey method does not apply to my model/How I can make it work.

          My Stata regression equation is this:

          Code:
          reghdfe QUALITY x1 x2 x3 x4 ESG##CONTROVERSY##LARGE, absorb(FIRM Year) cluster(FIRM)

          The model yields:

          Code:
          ```
HDFE Linear regression                            Number of obs   =        802
Absorbing 2 HDFE groups                           F(  11,    214) =      20.02
Statistics robust to heteroskedasticity           Prob > F        =     0.0000
                                                  R-squared       =     0.6332
                                                  Adj R-squared   =     0.4855
                                                  Within R-sq.    =     0.0211
Number of clusters (FIRM)    =        215         Root MSE        =     0.0140

                                          (Std. err. adjusted for 215 clusters in FIRM)
---------------------------------------------------------------------------------------
                      |               Robust
              QUALITY | Coefficient  std. err.      t    P>|t|     [95% conf. interval]
----------------------+----------------------------------------------------------------
                   x1 |  -.0000446   .0038578    -0.01   0.991    -.0076488    .0075597
                   x2 |   .0001482    .000103     1.44   0.152    -.0000548    .0003512
                   x3 |  -.0000391   .0000564    -0.69   0.489    -.0001502     .000072
                   x4 |   .0000958   .0000768     1.25   0.214    -.0000556    .0002472
                1.ESG |  -.0008591   .0023138    -0.37   0.711    -.0054198    .0037016
        1.CONTROVERSY |   .0190566   .0032089     5.94   0.000     .0127316    .0253816
                      |
      ESG#CONTROVERSY |
                 1 1  |  -.0175036   .0033864    -5.17   0.000    -.0241786   -.0108286
                      |
              1.LARGE |  -.0030138   .0044536    -0.68   0.499    -.0117923    .0057648
                      |
            ESG#LARGE |
                 1 1  |   .0067041   .0044098     1.52   0.130     -.001988    .0153963
                      |
    CONTROVERSY#LARGE |
                 1 1  |  -.0185935    .005275    -3.52   0.001    -.0289912   -.0081958
                      |
ESG#CONTROVERSY#LARGE |
               1 1 1  |   .0137926    .006862     2.01   0.046     .0002669    .0273183
                      |
                _cons |   .0666133   .0128766     5.17   0.000      .041232    .0919946

```
          Question 1:
          Based on this model output it seems that b#c,c#a and b#c#a are significant.
          Is this line of thought correct:

          1.1 In comparison to firms that are b=0 and c=0, firms with b=1 and c=1 combined provide 0.0175 lower quality disclosure, compared to what? Compared to firms with b=0 and c=0 average level of quality?

          1.2 Similarily, firms with c=1 and a=1 provide 0.1859 lower quality disclosure. But if firms have c=1 b=1 and a=1 firms provide 0.01379 better quality disclosures.

          Question 2:
          Subsequently, I split the model on a=0 and a=1 and ran the regression again.
          Code:
          ```
drop if a==0

reghdfe QUALITY x1 x2 x3 x4 b##c, absorb(FIRM Year) cluster(FIRM)
margins b#c
```
          The result:

          Code:
          ```
HDFE Linear regression                            Number of obs   =        399
Absorbing 2 HDFE groups                           F(   7,    110) =       1.48
Statistics robust to heteroskedasticity           Prob > F        =     0.1819
                                                  R-squared       =     0.6710
                                                  Adj R-squared   =     0.5256
                                                  Within R-sq.    =     0.0277
Number of clusters (FIRM)    =        111         Root MSE        =     0.0135

                                 (Std. err. adjusted for 111 clusters in FIRM)
------------------------------------------------------------------------------
             |               Robust
     QUALITY | Coefficient  std. err.      t    P>|t|     [95% conf. interval]
-------------+----------------------------------------------------------------
          x1 |  -.0029202   .0086532    -0.34   0.736    -.0200688    .0142283
          x2 |   .0001866   .0001655     1.13   0.262    -.0001414    .0005147
          x3 |  -.0000534   .0001082    -0.49   0.623    -.0002678     .000161
          x4 |   .0001271   .0001318     0.96   0.337    -.0001341    .0003882
         1.b |   .0062823   .0045987     1.37   0.175    -.0028312    .0153958
         1.c |  -.0019914   .0033063    -0.60   0.548    -.0085437    .0045608
             |
         b#c |
        1 1  |  -.0027965   .0062421    -0.45   0.655    -.0151669    .0095739
             |
       _cons |   .0722146   .0320609     2.25   0.026     .0086775    .1357516
------------------------------------------------------------------------------

Absorbed degrees of freedom:
-----------------------------------------------------+
 Absorbed FE | Categories  - Redundant  = Num. Coefs |
-------------+---------------------------------------|
        FIRM |       111         111           0    *|
        Year |         5           0           5     |
-----------------------------------------------------+
* = FE nested within cluster; treated as redundant for DoF computation

. margins b#c

Predictive margins                                         Number of obs = 399
Model VCE: Robust

Expression: Linear prediction, predict()

------------------------------------------------------------------------------
             |            Delta-method
             |     Margin   std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
         b#c |
        0 0  |   .0731567   .0007759    94.28   0.000     .0716359    .0746775
        0 1  |   .0711653   .0029872    23.82   0.000     .0653105    .0770201
        1 0  |    .079439   .0040798    19.47   0.000     .0714429    .0874352
        1 1  |   .0746511   .0030679    24.33   0.000     .0686382     .080664
------------------------------------------------------------------------------
```
          2.1 Now, the whole model is not significant anymore as indicated by the low F-value.
          The interaction of b and c is also insignificant. Are these interpretations correct?

          2.2 Why does the significance level change so rapidly here?

          In contrast, the results for the other part of the sample when a=1 is dropped.
          Code:
          ```
drop if a==1

reghdfe QUALITY x1 x2 x3 x4 b##c, absorb(FIRM Year) cluster(FIRM)
margins b#c
```
          The output yields:

          Code:
          ```

HDFE Linear regression                            Number of obs   =        384
Absorbing 2 HDFE groups                           F(   7,    108) =       9.61
Statistics robust to heteroskedasticity           Prob > F        =     0.0000
                                                  R-squared       =     0.6294
                                                  Adj R-squared   =     0.4603
                                                  Within R-sq.    =     0.0192
Number of clusters (FIRM)    =        109         Root MSE        =     0.0142

                                 (Std. err. adjusted for 109 clusters in FIRM)
------------------------------------------------------------------------------
             |               Robust
     QUALITY | Coefficient  std. err.      t    P>|t|     [95% conf. interval]
-------------+----------------------------------------------------------------
          x1 |  -.0019486   .0045507    -0.43   0.669    -.0109688    .0070717
          x2 |   .0001128   .0001199     0.94   0.349    -.0001249    .0003505
          x3 |  -.0000952   .0000557    -1.71   0.090    -.0002056    .0000152
          x4 |   .0000728   .0001022     0.71   0.478    -.0001298    .0002754
         1.b |  -.0011176    .002615    -0.43   0.670     -.006301    .0040657
         1.c |   .0189367   .0038611     4.90   0.000     .0112834      .02659
             |
         b#c |
        1 1  |  -.0178547   .0035639    -5.01   0.000    -.0249189   -.0107906
             |
       _cons |   .0727932   .0141331     5.15   0.000      .044779    .1008074
------------------------------------------------------------------------------

Absorbed degrees of freedom:
-----------------------------------------------------+
 Absorbed FE | Categories  - Redundant  = Num. Coefs |
-------------+---------------------------------------|
        FIRM |       109         109           0    *|
        Year |         5           0           5     |
-----------------------------------------------------+
* = FE nested within cluster; treated as redundant for DoF computation

. margins b#c

Predictive margins                                         Number of obs = 384
Model VCE: Robust

Expression: Linear prediction, predict()

------------------------------------------------------------------------------
             |            Delta-method
             |     Margin   std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
         b#c |
        0 0  |   .0725642   .0011141    65.13   0.000     .0703806    .0747478
        0 1  |   .0915009   .0039866    22.95   0.000     .0836873    .0993145
        1 0  |   .0714465   .0015168    47.10   0.000     .0684736    .0744195
        1 1  |   .0725285   .0039184    18.51   0.000     .0648486    .0802085
------------------------------------------------------------------------------
```
          2.3 Now the overall model is significant again, indicated by the high F value.
          Also, the single effect of 1.c is highly significant.
          It means that for this sample, in comparison to firms which have b=0 but c=1, the quality will increase by 0.0189 in comparison to their average level of quality. Right?

          2.4 while 1.b is insignificant, the interaction of b=1 and c=1 is highly significant, indicating that firms with b=1 and c=1 will provide lower quality disclosures of 0.0178 in comparison to firms which have b=0 and c=0.

          Question 3:

          In order to further investigate the effect of a=0 firms, I ran the regression again.

          Code:
          ```
reghdfe QUALITY x1 x2 x3 x4 b##c, absorb(FIRM Year) cluster(FIRM)
```

```

HDFE Linear regression                            Number of obs   =        802
Absorbing 2 HDFE groups                           F(  11,    214) =      20.02
Statistics robust to heteroskedasticity           Prob > F        =     0.0000
                                                  R-squared       =     0.6332
                                                  Adj R-squared   =     0.4855
                                                  Within R-sq.    =     0.0211
Number of clusters (FIRM)    =        215         Root MSE        =     0.0140

                                 (Std. err. adjusted for 215 clusters in FIRM)
------------------------------------------------------------------------------
             |               Robust
     QUALITY | Coefficient  std. err.      t    P>|t|     [95% conf. interval]
-------------+----------------------------------------------------------------
          x1 |  -.0000446   .0038578    -0.01   0.991    -.0076488    .0075597
          x2 |   .0001482    .000103     1.44   0.152    -.0000548    .0003512
          x3 |  -.0000391   .0000564    -0.69   0.489    -.0001502     .000072
          x4 |   .0000958   .0000768     1.25   0.214    -.0000556    .0002472
         1.b |  -.0008591   .0023138    -0.37   0.711    -.0054198    .0037016
         1.c |   .0190566   .0032089     5.94   0.000     .0127316    .0253816
             |
         b#c |
        1 1  |  -.0175036   .0033864    -5.17   0.000    -.0241786   -.0108286
             |
         1.a |  -.0030138   .0044536    -0.68   0.499    -.0117923    .0057648
             |
         b#a |
        1 1  |   .0067041   .0044098     1.52   0.130     -.001988    .0153963
             |
         c#a |
        1 1  |  -.0185935    .005275    -3.52   0.001    -.0289912   -.0081958
             |
       b#c#a |
      1 1 1  |   .0137926    .006862     2.01   0.046     .0002669    .0273183
             |
       _cons |   .0666133   .0128766     5.17   0.000      .041232    .0919946
------------------------------------------------------------------------------
```
          The same output as in the first code.
          Then I went ahead and
          Code:
          ```
margins a#b#c
marginsplot, by(a) x(c) noci
```
          It yields:

          Code:
          ```

             |            Delta-method
             |     Margin   std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
       a#b#c |
      0 0 0  |   .0742696   .0023878    31.10   0.000     .0695896    .0789496
      0 0 1  |   .0933262   .0035282    26.45   0.000     .0864112    .1002413
      0 1 0  |   .0734105   .0028331    25.91   0.000     .0678578    .0789632
      0 1 1  |   .0749635   .0036351    20.62   0.000      .067839    .0820881
      1 0 0  |   .0712558   .0022367    31.86   0.000     .0668719    .0756398
      1 0 1  |   .0717189   .0045715    15.69   0.000      .062759    .0806788
      1 1 0  |   .0771009   .0036927    20.88   0.000     .0698634    .0843384
      1 1 1  |    .073853   .0043652    16.92   0.000     .0652973    .0824087
```
          And:


          Click image for larger version Name: marginsplot.jpg Views: 1 Size: 54.9 KB ID: 1696369


          So, again my interpretation is that b#c, c#a and a#b#c are all significant.

          But then I ran a comparison of the interaction of b and c between both levels of a, and a critical rate test:
          Code:
          ```
contrast b#c@a
. display "critical value per family error rate = " invfprob(1, 214, 0.05/2) 


```
          Yielding:

          Code:
          ```
critical value per family error rate = 5.0953285
------------------------------------------------
             |         df           F        P>F
-------------+----------------------------------
       b#c@a |
          0  |          1       26.72     0.0000
          1  |          1        0.40     0.5265
      Joint  |          2       13.69     0.0000
             |
 Denominator |        214
------------------------------------------------
```
          3.1 So, only at level a=0 the interaction is significant.
          Am I right to assume that this is then simply in line with the results from the split sample analysis?

          3.2 If yes, why does the outout table of the threeway interaction depict the interaction between b#c#a as significant then? Is it significant then or not? The same is for c#a, which is 3.52 in the outout table and significant.

          Anyways, I continued:

          Code:
          ```
margins c@b, at(a=0) contrast
contrast c@b@i(0).a
```
          And it yields:

          Code:
          ```


------------------------------------------------
             |         df           F        P>F
-------------+----------------------------------
       c@b#a |
        0 0  |          1       35.27     0.0000
        1 0  |          1        0.16     0.6888
      Joint  |          2       23.73     0.0000
             |
 Denominator |        214
------------------------------------------------
```
          3.3 Which I interprete as that c is only significant for firms having b=0.

          Continuing to test levels of c:

          Code:
          ```
pwcompare c#i(0).b#i(0).a, mcompare(tukey) effects
```
          It yields:
          method tukey is not allowed with results from reghdfe

          3.4 Why does this happen? Is there a way around it to make the results interpretable?

          I removed mcompare(tukey) and it gave me:
          Code:
          ```
pwcompare c#i(0).b#i(0).a, effects
```
          And it yields:

          Code:
          ```

                    |                            Unadjusted           Unadjusted
                    |   Contrast   Std. err.      t    P>|t|     [95% conf. interval]
--------------------+----------------------------------------------------------------
              c#b#a |
(1 0 0) vs (0 0 0)  |   .0190566   .0032089     5.94   0.000     .0127316    .0253816
```
          Only if c=1 and b=0 and a=0, then there is a significant relationship of c and Quality.

          3.5 Again, is this interpretation correct? And, is this way of computation appropriate here?

          Please excuse me for this long post, but I have really trouble wrapping my head around these results.

          This is cross posted here:
          https://www.statalist.org/forums/for...wo-way-are-not

          Comment

          Previous Next
          Working...
          X