Interpretation of interaction with time in FE model

[Gary Sean]

Dear all,

Would you help me to interpret the following models?

This is 5 waves data with time-varying treatment (0 to 2 scale) and two covariates.

In M1, I run two-way fixed effect model. But I could not find any significant treatment effect (-.002). As an alternative specification, I run the following model - I created the interaction term between treatment and time. In M2, a one unit increase in treatment variable "on average" is associated with .28*** increase in math achievement. Yet, the interaction term in M2 shows that every year treatment effect decreases by .09.

Here is my questions:

1) In M2, it seems that the main treatment effect (.28) corresponds to "year 1 effect". So after 4 years, the treatment effect turns to be negative (-.36+.28=-.08). Am I right?

2) In M1, treatment effect is found to be -.002. It is usually interpreted as "on overage" one unit increase in X is associated with Y. What does "on average" exactly stand for? In every year, treatment variable has -.002 effect? Thus, over 4 years, it has -.002*4 total effects?

3) If I rely on M3, it seems clear that treatment effect become more detrimental over years. If I want to manually calculate the treatment effect over 4 years, should I only employ significant coefficients associated with treatment variable? e.g., .19+(-.18-.30-.33) ?


Your reply will be greatly appreciated. Thank you!


Sean




M1

Math Coef. Std. Err. t P>t [95% Conf. Interval]

Treatment -.0018498 .0220533 -0.08 0.933 -.045076 .0413765
IRT_R .0006786 .0013973 0.49 0.627 .0020602 .0034174
IRT_M -.0060498 .0015313 -3.95 0.000 .0090514 -.0030483

year
_W4 .5765732 .0505503 11.41 0.000 .4774903 .6756561
_W6 1.355762 .0760639 17.82 0.000 1.20667 1.504853
_W7 2.224214 .094319 23.58 0.000 2.039341 2.409087
_W8 3.107438 .1060079 29.31 0.000 2.899653 3.315222

_cons 16.75458 .1074408 155.94 0.000 16.54399 16.96517



M2

Math Coef. Std. Err. t P>t [95% Conf. Interval]

cov1 .0002283 .0013969 0.16 0.870 -.0025098 .0029664
cov2 -.0057629 .0015299 -3.77 0.000 -.0087616 -.0027641

year
_W4 .5955138 .0505553 11.78 0.000 .4964211 .6946065
_W6 1.400425 .0762236 18.37 0.000 1.25102 1.549829
_W7 2.303995 .0948584 24.29 0.000 2.118064 2.489925
_W8 3.233523 .1073338 30.13 0.000 3.023139 3.443906

Treatment .2819542 .0454272 6.21 0.000 .1929129 .3709955

c.year#c.Treatment -.0880109 .012321 -7.14 0.000 -.112161 -.0638608

_cons 16.72663 .107375 155.78 0.000 16.51616 16.93709




M3

Math Coef. Std. Err. t P>t [95% Conf. Interval]

cov1 .0002339 .0013978 0.17 0.867 -.0025059 .0029736
cov2 -.0057813 .0015312 -3.78 0.000 -.0087826 -.0027799

year
_W4 .5898591 .051925 11.36 0.000 .4880817 .6916365
_W6 1.401499 .077243 18.14 0.000 1.250096 1.552902
_W7 2.317785 .0956012 24.24 0.000 2.130399 2.505172
_W8 3.22343 .1076846 29.93 0.000 3.012359 3.434501

year#c.Treatment
_W4 -.0665615 .0551084 -1.21 0.227 -.1745786 .0414557
_W6 -.1763148 .0546061 -3.23 0.001 -.2833474 -.0692823
_W7 -.2952571 .0555001 -5.32 0.000 -.4040421 -.1864721
_W8 -.3270771 .0556028 -5.88 0.000 -.4360634 -.2180909

Treatment .1907644 .045256 4.22 0.000 .1020589 .27947

_cons 16.72785 .1080214 154.86 0.000 16.51612 16.93958

[Clyde Schechter]

1) In M2, it seems that the main treatment effect (.28) corresponds to "year 1 effect". So after 4 years, the treatment effect turns to be negative (-.36+.28=-.08). Am I right?

Your model M2 is mis-specified and should not be interpreted. If you want to treat year as a continuous variable, then you must do so in all parts of the model. In particular, there needs to be a year term (not the separate year indicator ["dummy"] variables). You can have separate year indicator in addition to a continuous year variable, but you can't omit the continuous year variable here. The model should be something like

regress Math i.Treatment##c.year

(I'm assuming here that Treatment is not really a continuous variable but is a 0/1 variable.) When you run that model, the coefficient of Treatment will be the treatment effect in Year 0. If there is no Year 0 then it is a meaningless number. If you want to see what the treatment effect in each year is, from 0 through 4, you can run

margins, dydx(Treatment) at(year = (0(1)4))

What does "on average" exactly stand for?

It doesn't stand for anything. It's an abuse of language. Actually, it is what the treatment effect would be if we impose the modeling constraint that the treatment effect is the same in all years. Model M1 is inconsistent with model M2. Model M1 imposes this constraint of constant treatment effect, whereas model M2 explicitly requires that the treatment effect change by a certain amount each year. The results from model M1 are not in any mathematical sense, the average results of the yearly effects in model M2. Models M1 and M2 cannot both be correct models. At least one of them must be a mis-specification of the data generating process as they contradict each other.

In every year, treatment variable has -.002 effect? Thus, over 4 years, it has -.002*4 total effects?

I don't know what you're talking about. I don't see the number -0.002, nor anything close to it, in the output you show for Model M1. Please clarify.

3) If I rely on M3, it seems clear that treatment effect become more detrimental over years. If I want to manually calculate the treatment effect over 4 years, should I only employ significant coefficients associated with treatment variable? e.g., .19+(-.18-.30-.33) ?

No, no, no, no, no. Statistical significance of these individual terms is completely irrelevant. You must include them all, whether statistically significant or not.

Please read the Forum FAQ, with special attention to #12 for excellent advice on how best to format Stata code and output here in the forum so it presents readably. While there, also read about the -dataex- command for posting example data, as you are likely to need to do that in future posts as well.

[Gary Sean]

Dear Clyde Schechter,

Thanks so much for your clear answer. May I ask you a question? You pointed out that "the results from model M1 are not in any mathematical sense, the average results of the yearly effects in model M2. Models M1 and M2 cannot both be correct models. At least one of them must be a mis-specification of the data generating process as they contradict each other." Would you explain why you think one of models is mis-specified? I rerun the models with a continuous form of time variable (please see below). Thanks again!


Sean

[Carlo Lazzaro]

Gary:
as an aside to Clyde's as always excellent advice, the code of your secon code looks somehow weird to me:
- as Clyde wisely highlighted, -Treatment- is intrisically a categorical variable (each treatment=1 level of the categorical variable). I find really hard to present this predictor as a continuos one, because it would sound meaningless. Hence your interaction should be:

c.Year##i.Treatment

;
- you applied the cluster/robust standard error; did you detect heteroskedasticity and/or autocorrelation in your data?;
-eventually, you can check whether your model has no evidence of misspecification just running an auxiliary -xtreg,fe- regression with the same dependent variable and fitted value up to the fourth power as predictors, like in the following toy-example:

. use "http://www.stata-press.com/data/r15/nlswork.dta"
(National Longitudinal Survey.  Young Women 14-26 years of age in 1968)

. xtreg ln_wage age tenure, fe

Fixed-effects (within) regression               Number of obs     =     28,101
Group variable: idcode                          Number of groups  =      4,699

R-sq:                                           Obs per group:
     within  = 0.1296                                         min =          1
     between = 0.1916                                         avg =        6.0
     overall = 0.1456                                         max =         15

                                                F(2,23400)        =    1742.76
corr(u_i, Xb)  = 0.1302                         Prob > F          =     0.0000

------------------------------------------------------------------------------
     ln_wage |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         age |   .0121949   .0004131    29.52   0.000     .0113852    .0130045
      tenure |   .0211313   .0008015    26.37   0.000     .0195604    .0227023
       _cons |   1.256467   .0109792   114.44   0.000     1.234947    1.277987
-------------+----------------------------------------------------------------
     sigma_u |  .39034493
     sigma_e |  .29808194
         rho |  .63165531   (fraction of variance due to u_i)
------------------------------------------------------------------------------
F test that all u_i=0: F(4698, 23400) = 8.02                 Prob > F = 0.0000

. predict fitted, xb
(433 missing values generated)

. g sq_fitted= fitted^2
(433 missing values generated)

. g cu_fitted=fitted^3
(433 missing values generated)

. g qu_fitted=fitted^4
(433 missing values generated)

. xtreg ln_wage fitted sq_fitted cu_fitted qu_fitted , fe

Fixed-effects (within) regression               Number of obs     =     28,101
Group variable: idcode                          Number of groups  =      4,699

R-sq:                                           Obs per group:
     within  = 0.1414                                         min =          1
     between = 0.1890                                         avg =        6.0
     overall = 0.1512                                         max =         15

                                                F(4,23398)        =     963.22
corr(u_i, Xb)  = 0.1252                         Prob > F          =     0.0000

------------------------------------------------------------------------------
     ln_wage |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      fitted |   261.9079   41.66072     6.29   0.000     180.2502    343.5656
   sq_fitted |  -209.8997   34.82085    -6.03   0.000    -278.1509   -141.6486
   cu_fitted |   74.91571   12.87572     5.82   0.000     49.67845     100.153
   qu_fitted |  -10.01921    1.77708    -5.64   0.000    -13.50241    -6.53602
       _cons |  -121.2816   18.60721    -6.52   0.000     -157.753   -84.81025
-------------+----------------------------------------------------------------
     sigma_u |  .38891971
     sigma_e |  .29607691
         rho |  .63309262   (fraction of variance due to u_i)
------------------------------------------------------------------------------
F test that all u_i=0: F(4698, 23398) = 8.21                 Prob > F = 0.0000

Then you can test whether the three higher order polynomials are jointly different from zero:

. test sq_fitted cu_fitted qu_fitted

 ( 1)  sq_fitted = 0
 ( 2)  cu_fitted = 0
 ( 3)  qu_fitted = 0

       F(  3, 23398) =  106.67
            Prob > F =    0.0000

In this case, the generalized version of Ramsey's RESET test (which is avilable in Stata as a -postestimation- test after -regress-; see -help ovtest-) tells us thet the model, despite having two highly significant predictors, is unreliable due to its misspecification.

[Clyde Schechter]

I agree with Carlo's comments in #4.

Regarding #3, I will continue to assume that notwithstanding the specification c.Treatment in the model, that Treatment is actually a 0/1 variable. The mis-specification with c.Treatment will not harm the regression results shown, but it would lead to incorrect results in subsequent applications of the -margins- command.

So the second model says that in year 0 (whatever actual calendar year that is), the treatment group's Math scores were, on average 0.177 units higher than the control group's. However, in subsequent years, that effect declined, and if we want to look, say, 4 years out, the treatment group's Math scores would, on average, differ from the controls by 0.177 - 4*.080 = -0.143. So, four years out the expected Math score in the treatment group is lower than that of the controls.

The first model, which estimates treatment effect subject to the constraint that it must be the same in each year, says that this difference is -0.002.

Again, you need to come to a decision about which model is appropriate for the data. A graphical exploration of how scores trend over time in each group might prove helpful if there is no scientific theory to guide you. But models 1 and 2 cannot both be correct: you must choose between them.

[Gary Sean]

Thanks so much! This is really a great site with wise advisors. I should be more clearly stated that the treatment variable ranges from 0 to 2, that's why I specified it as a continuous variable to exploit its variation in FE model. Ramsey's RESET test seems also very useful. Thanks again.

Sean

[Clyde Schechter]

If the treatment ranges from 0 to 2 in the sense that it can be 0, 1, or 2, and if treatment 2 is in some meaningful sense twice as much treatment as treatment 1, then it is fine to use it as a continuous variable. But if you just have 3 treatment groups, and there is no real sense in which treatment 2 is twice as much as treatment 1, then you should use i.treatment, not c.treatment.

If on the other hand treatment ranges from 0 to 2 in the sense that it can, in principle, take on any numeric value between 0 and 2 and is, in some meaningful sense, a measure of the intensity with which some treatment is being applied, then you are fine using c.treatment.