testing the economic significance of an interaction term of two categorical variables

[ruby swinney]

Hello everyone,

I ran the following regression :

xi: areg Y X1 X2 X1_X2 $x i.year, absorb(firm_id) cluster(firm_id)
estimates store reg1



both X1 and X2 are dummy variables (equal either 1 or 0), with year and firm fixed effect and SE clustered on firm level.

my question is how to interpret the results of this regression ?
I ran a wald t test as follow to test whether B1+B3 was statistically different from zero.

test X1+X1_X2==0

my research question would be what's the effect of X1 on Y when X2=1.
am I doing it correctly or is there a better way to interpret the interaction term coefficient?


Thank you

[Carlo Lazzaro]

Ruby:
welcome to this forum:
if your questions are class/howework-related, please see https://www.statalist.org/forums/help#adviceextras, #4.

If thery are not, please see the FAQ on posting more effectively, that includes sharing what you tyoped and what Stata gave you back. Thanks.
As an aside, the -xi:- prefix before -areg- code is redundant.

[ruby swinney]

Thank you for your comment,

its is NOT a homework assignment and I typed the code I used in stata which is

xi: areg Y X1 X2 X1_X2 $x i.year, absorb(firm_id) cluster(firm_id)
estimates store reg1

test X1+X1_X2==0

	Y1	Y2
X1	0.090***	0.033***
	3.24	3.76
X2	-0.001	-0.010***
	-0.29	-7.81
X1_X2	-0.054*	-0.022**
	-1.77	-2.3
control var1	0.405***	0.377***
	42.1	96.58
control var2	-0.004***	-0.001***
	-24.47	-34.75
control var3	0.178***	0.146***
	14.79	41.33
control var4	-0.332***	-0.060***
	-30.91	-29.89
control var5	-0.063***	0.029***
	-36.31	57.54
control var6	0.030***	0.036***
	9.29	26.21
control var9	-26.822***	-3.017***
	-26.9	-7.11
_cons	0.626***	-0.006
	45.33	-1.34
wald t test B1+B3=0	0.0028	0.0124
N	3.76E+05	3.75E+05
r2_a	0.601	0.677

I changed the real name of the variables for simplicity.

X1 and X2 are dummy variable and Y1 and Y2 are continuous variables with year and firm fixed effect and SE clustered on firm level.

I ran a wald t test as follow to test whether B1+B3 was statistically different from zero..

I would like to know what's the proper way to interpret the results of these regressions ( what should I look at ?) and whether there is a mistake in my code

Thank you!

[Carlo Lazzaro]

Ruby:
- -xi:- prefix is redundant before -areg-;
- you can add the -allbeselevels- option at the end of your code to see what's going on with the variables included in the interaction and with the interaction coefficient itself;
- then you can -test- the coefficients according to your (linear or non-linear) needs.
I do hope the following toy-example will be helpful:

. use "https://www.stata-press.com/data/r16/nlswork.dta"
(National Longitudinal Survey.  Young Women 14-26 years of age in 1968)

. areg ln_wage i.not_smsa##i.south tenure, vce(cluster idcode) abs(idcode)  allbaselevels

Linear regression, absorbing indicators         Number of obs     =     28,093
Absorbed variable: idcode                       No. of categories =      4,699
                                                F(   4,   4698)   =     260.85
                                                Prob > F          =     0.0000
                                                R-squared         =     0.6654
                                                Adj R-squared     =     0.5981
                                                Root MSE          =     0.3029

                               (Std. Err. adjusted for 4,699 clusters in idcode)
--------------------------------------------------------------------------------
               |               Robust
       ln_wage |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
---------------+----------------------------------------------------------------
      not_smsa |
            0  |          0  (base)
            1  |  -.1077156   .0227277    -4.74   0.000    -.1522725   -.0631587
               |
         south |
            0  |          0  (base)
            1  |  -.0629998   .0218439    -2.88   0.004     -.105824   -.0201756
               |
not_smsa#south |
          0 0  |          0  (base)
          0 1  |          0  (base)
          1 0  |          0  (base)
          1 1  |   .0492343   .0316996     1.55   0.120    -.0129118    .1113804
               |
        tenure |   .0340884   .0010671    31.95   0.000     .0319964    .0361804
         _cons |    1.61924   .0100615   160.93   0.000     1.599515    1.638965
--------------------------------------------------------------------------------

. mat list e(b)

e(b)[1,10]
              0b.            1.           0b.            1.  0b.not_smsa#  0b.not_smsa#  1o.not_smsa#   1.not_smsa#            
        not_smsa      not_smsa         south         south      0b.south      1o.south      0b.south       1.south        tenure
y1             0    -.10771556             0    -.06299976             0             0             0     .04923429     .03408839

                
           _cons
y1     1.6192399

. test 1.not_smsa#1.south=0

 ( 1)  1.not_smsa#1.south = 0

       F(  1,  4698) =    2.41
            Prob > F =    0.1205

. test 1.south + 1.not_smsa#1.south=0

 ( 1)  1.south + 1.not_smsa#1.south = 0

       F(  1,  4698) =    0.20
            Prob > F =    0.6558

.

[ruby swinney]

Thank you,

what I want to know more precisely is in case of an interaction term, do I have to test for B1+B3 to be able to interpret it? are there other ways?
secondly, I used a wald t test for which p value are <0.05 threshold, does it mean that adding both B1 and B3 improves the fit of my regression?
if the p values are above 0.05, what does it mean? I am not sure how to interpret my results and also other situations when wald t test p values are above 0.05 but B1 and B3 are statistically significant...


I am confused....

[Carlo Lazzaro]

Ruby:
- if the coefficient of the interaction term (that you mentioned as B3) reaches(does not reach statistical significance, you can simply look at the 95% CI and p-value to have an idea of what's going on;
- to check whether adding one or more predictors increase the fit of your regression model, you should look at adjusted R-sq (if available) or other measures of fit (AIC BIC);
- if the p-values are above the arbitrary 0.05 threshold, your coefficients are considered not statistical significant. This is not the end of the world (as many, unfortunately, think), because significant and non-significant coefficients are both informative. That said, if you have this kind of concerns, you should read any decent textbook on regression.;
- as far as your last question is concerned, it's usually the other way round: when assessing their joint significance, unsignificant coefficients give a significant result.

[ruby swinney]

thank you for your comments,

So you mean that if I want to interpret the interaction term B3, I cannot just look at its own statistical significance, I need to test whether the sum of B1+B3 is statistically significant or not?
Am I correct?

[Carlo Lazzaro]

Ruby:
not quite.
You should look at the i.X1##i.X2 (when both X1=1 and X2=1) coefficient.
Please note that Stata:
- gives back no coefficient for X1=0 and X2=0 (reference case);
- reports coefficient X1 (X1=1 and X2=0);
- reports coefficient X2 (X1=0 and X2=1);
- reports coefficient for the interaction, as detailed above (X1=1 and X2=1).

[ruby swinney]

is it also possible that significant coefficients on their own, used in an interaction term give up an insignificant coefficient ? like B1 and B2 are statistically significant, B3 is also statistically significant but wald t test of B1+B3 shows a p value above 0.05. how should I interpret it? Thank you

[Carlo Lazzaro]

Ruby:
please post what you typed and what Stata gave you back (via CODE delimiters, please). Thanks.

[ruby swinney]

Here it is

xi: areg Y X1 X2 X1_X2 $x i.year, absorb(firm_id) cluster(firm_id)
estimates store reg1

test X1+X1_X2==0

	Y
X1	0.021***
	3.29
X2	-0.010***
	-7.79
X1_X2	-0.019**
	-2.54
control 1	0.377***
	96.57
control 2	-0.001***
	-34.75
control 3	0.146***
	41.34
control 4	-0.060***
	-29.89
control 5	0.029***
	57.53
control 6	0.036***
	26.22
control 7	-3.023***
	-7.13
_cons	-0.006
	-1.33
wald t test B1+B3=0	0.4363
N	3.75E+05
r2_a	0.677

[Carlo Lazzaro]

Ruby:
I see the issue now (but please use CODE delimiters. Thanks).

. use "https://www.stata-press.com/data/r16/nlswork.dta"
(National Longitudinal Survey.  Young Women 14-26 years of age in 1968)

. areg ln_wage i.south##i.not_smsa, vce(cluster idcode) abs(idcode)

Linear regression, absorbing indicators         Number of obs     =     28,526
Absorbed variable: idcode                       No. of categories =      4,711
                                                F(   3,   4710)   =      12.06
                                                Prob > F          =     0.0000
                                                R-squared         =     0.6270
                                                Adj R-squared     =     0.5532
                                                Root MSE          =     0.3196

                               (Std. Err. adjusted for 4,711 clusters in idcode)
--------------------------------------------------------------------------------
               |               Robust
       ln_wage |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
---------------+----------------------------------------------------------------
       1.south |   -.070137   .0235179    -2.98   0.003    -.1162432   -.0240309
    1.not_smsa |  -.1188317   .0239774    -4.96   0.000    -.1658386   -.0718248
               |
south#not_smsa |
          1 1  |   .0691241   .0334817     2.06   0.039     .0034843    .1347639
               |
         _cons |    1.72652   .0102768   168.00   0.000     1.706373    1.746668
--------------------------------------------------------------------------------

. mat list e(b)

e(b)[1,9]
             0b.           1.          0b.           1.    0b.south#    0b.south#    1o.south#     1.south#            
          south        south     not_smsa     not_smsa  0b.not_smsa  1o.not_smsa  0b.not_smsa   1.not_smsa        _cons
y1            0   -.07013703            0   -.11883173            0            0            0    .06912414    1.7265204

. test 1.south+1.south#1.not_smsa=0

 ( 1)  1.south + 1.south#1.not_smsa = 0

       F(  1,  4710) =    0.00
            Prob > F =    0.9753

.

-test- outcome simply states that the linear combination of these coefficients does not reach joint statistical significant.

However, I fail to get what are the consequences (I mean in substantive terms) of this result for any regression.

[ruby swinney]

if you had seen my regression results with B1, B2 and B3 all being statistically significant, without using the wald t test, what kind of inferences would you be able to make?

Can I just blindly say that when X1=1 and X2=2, Y decreases by (0.019)?

[Carlo Lazzaro]

Ruby:
your interaction reaches statistical significance, B2 and B3 is already reported in the regression output. They all reach statistical significance.
You can test any (linear or non-linear) combination of your coefficients if this is the goal of your research, but that's all.

[ruby swinney]

I was advised to use wald t test to test B1+B3=0

I just dont understand 1: why do I need it or when is it generally used, 2. what can I take away with a p value of above 0.05 when all variable coefficients are statistically significant

I am new to all of this, so I am not sure what's the best course of action