Generalized Diff-in-Diff to capture treatment effect on relation between Y and X

[Harald Leber]

I use a diff-in-diff design to capture the effect of a treatment on the coefficient b1 of the following model: Y = b0 + b1*X + u.
The full model therefore is: Y = b0 + b1*X + b2*POST+b3*TREAT+b4*(POST*TREAT)+b5*(POST*X)+b6*(T REAT*X)+b7*(POST*TREAT*X)+u with the coefficient of interest being b7.

Now I wanted to increase the precision of my DiD estimates by adding other explanatory variables for Y (i.e., decrease standard errors). To do so, I first use a "generalized" diff-in-diff model, that is, I add a full set of firm-fixed and time-fixed effects to soak up variation. Therefore, I use the following Stata regression:

xi: areg Y X 1.POST 1.TREAT 1.POST#1.TREAT 1.POST#c.X 1.TREAT#c.X 1.POST#1.TREAT#c.X i.year, cl(firmid) absorb(firmid)

1.POST 1.TREAT and 1.POST#1.TREAT should drop out (because they are perfectly colinear with the fixed effects), but the rest should survive.

However - my results on b7 seem to suffer a lot and get insignificant when i use firm- and year-fixed effect to soak up variation - in my opinion these fixed effect should not hurt the DiD triple-interaction; am I doing something wrong here?

Furthermore, I thought of adding further controls that vary in time, such as firmsize (i skipped the variables that will drop out anyway here):

xi: areg Y X 1.POST#c.X 1.TREAT#c.X 1.POST#1.TREAT#c.X c.FIRMSIZE i.year, cl(firmid) absorb(firmid)

Is that recommendable or am I getting into new econometrical problems here?

[Clyde Schechter]

1.POST 1.TREAT and 1.POST#1.TREAT should drop out (because they are perfectly colinear with the fixed effects)

In the standard DID model, the same observational units are followed from a pre-intervention state to a post-intervention state. Though you don't specifically state that the firmid variable indexes these observational units, it seems strongly implied by the post as a whole. But if that is true, each level of firmid will be associated with both POST = 0 and POST = 1, so 1.POST should not be colinear with the fixed effects. And neither should POST#TREAT. So if these really are being dropped for colinearity, then either there is something wrong with your data, or there is more complexity to your design than you have conveyed in your post.

Apart from that, ditch the -xi:- prefix. It is obsolete, and since you are (almost correctly) using factor variable notation, it accomplishes nothing. To get the full advantages of factor variable notation, you should use it correctly. Your command should look like

areg Y i.POST##i.TREAT##c.X i.year, cl(firmid) absorb(firmid)

Also, you may need to heed the warning in the -areg- help file:

areg fits a linear regression absorbing one categorical factor. areg is designed for datasets with many groups, but not a number of groups that increases with the sample size. See the xtreg, fe command in [XT] xtreg for an estimator that handles the case in which the number of groups increases with the sample size.

Without knowing your design, it is not possible for me to say which is appropriate for you, -areg-, or -xtreg-, but designs where the number of groups does increase with sample size are far more common in practice.

Finally, if your third-order interaction term coefficient is shrinking when you adjust for time, it suggests that that effect is confounded with time. The question then is whether the inclusion of time as a covariate is appropriate. If the nature of the intervention is such that its effects vary over time (e.g. they might slowly take hold and increase, then level off, and maybe even attenuate later on), then time, in a sense, lies on the causal chain between intervention and outcome, and adjusting for time would be inappropriate. But if the nature of the intervention is such that its effects should be more or less immediate and persistent, then the loss of the interaction when adjusting for time suggests that the apparent influence of X on the treatment effect is, at least in part, confounded with some secular trend in your outcome.

Note that in the previous paragraph I speak of the interaction term coefficient shrinking, not whether it loses statistical significance. Loss of statistical significance can occur simply because you are, in effect, chopping up your data set into more and more pieces that are smaller and smaller, and you may just not have enough data left in each piece to sustain statistical significance. Understanding what is going on here really is better served by focusing more on whether the coefficient is changing much.

[Harald Leber]

Thank you Clyde for your quick and helpful response!

Maybe a couple of words about the sample/design: The sample comprises 2,800 firm-year observations of 500 unique firms for the years 2000-2005 (unbalanced, but each firm needs at least one firm-year observation in the pre- and post period, respectively). The pre-period is 2000-2002, the post-period is 2003-2005.

1. Regarding the POST variable: I agree w.r.t. the firm-fixed effects, but I thought including i.year (for each of the years 2000-2005; except one that gets dropped) should drop POST and POST#TREAT due to collinearity. The idea was somehow to use two-way fixed effects (firmid and year) to soak up a lot of the unexplained variation in Y.

2. Yes, the three-way interaction term 1.POST#1.TREAT#c.X is shrinking when I adjust for time (i.e., when I include i.year) - in magnitude and significance; as the effect of the intervention (a regulatory change) should be more or less immediate and persistent - I guess I have to include i.year?

So my revised model would be:

xtreg Y i.POST##i.TREAT##c.X i.year, fe cl(firmid)

And with multiple explanatory variables:

xtreg Y i.POST##i.TREAT##c.X1 i.POST##i.TREAT##c.X2 i.year, fe cl(firmid)

Does this seem reasonable ?

Thanks!!!

[Clyde Schechter]

1. Yes, when you have both firm and time fixed effects in the model, then POST will be collinear with those and will be dropped. I misunderstood your original post.

2. It sounds like including year is appropriate, as time apparently confounds the effect of X. And your suggested code looks right to me. If you have a large number of explanatory variables you can save yourself a little typing (and reduce the risk of making an error):

xtreg Y i.POST##i.TREAT##c.(X1 X2 X3 X4 X5 X6 X7 X8 X9 X10) i.year, fe cl(firmid)

(provided all the X's are continuous).

[Harald Leber]

Thanks for all the effort Clyde - may I ask you one more general question regarding covariates in DiD models:

If I understand the DiD design correctly, the last model you show,

xtreg Y i.POST##i.TREAT##c.(X1 X2 X3 X4 X5 X6 X7 X8 X9 X10) i.year, fe cl(firmid)

would be the right choice if the coefficients for X1, X2, X3 are expected to vary with the treatment, i.e. I would need a hypothesis/theory why/how these coefficients change due to treatment (even if they should only act as controls).

Instead, if I simply use the following model:

xtreg Y i.POST##i.TREAT##c.(X1) X2 X3 i.year, fe cl(firmid)

I would assume that X2 and X3 have a (time-varying) effect on Y, but that this effect is independent of the treatment (and constant across groups).

Now what I read from the literature is, that you would add X2 and X3 only to increase the precision of the DiD model (i.e., to reduce standard errors) and the actual triple interaction DiD-term of interest should be unaffected. Is that right?

I'm also happy about if anyone has references to literature that can help me to understand covariates in DiD models - a Google search revealed somehow very different opinons about this topic...

[Clyde Schechter]

Code:

xtreg Y i.POST##i.TREAT##c.(X1 X2 X3 X4 X5 X6 X7 X8 X9 X10) i.year, fe cl(firmid)
would be the right choice if the coefficients for X1, X2, X3 are expected to vary with the treatment,

No, that's not correct. You can have modification of treatment effect by a variable X even if X itself does not vary with treatment. For example, in a medical context, a treatment's effectiveness (i.e. impact on disease) might well depend on the patient's age, or weight. But there is no general reason to think that age or weight would vary based on treatment assignment. What is true is that if the distribution of X is independent of treatment, then X will not confound the treatment-outcome relationship. But it could still be an effect modifier. They are two different concepts, and you need to be clear which you are dealing with in your problem. Confounding requires entering the variable as a covariate. Effect modification calls for an interaction term.

Instead, if I simply use the following model:
Code:

xtreg Y i.POST##i.TREAT##c.(X1) X2 X3 i.year, fe cl(firmid)
I would assume that X2 and X3 have a (time-varying) effect on Y, but that this effect is independent of the treatment (and constant across groups).

This code fits a model in which X2 and X3 are assumed to have an effect on Y that does not vary with time and is the same regardless of treatment. If the effect of an X variable on Y is time-varying, then you need to include X#i.year interaction terms to capture that. As for being constant across groups, I don't what groups you are referring to here.

Now what I read from the literature is, that you would add X2 and X3 only to increase the precision of the DiD model (i.e., to reduce standard errors) and the actual triple interaction DiD-term of interest should be unaffected. Is that right?

If X2 and X3 do not vary with treatment, then the purpose of including them in the model is to reduce the "extraneous" variance they contribute to the outcome, which may result in improved precision. Note, however, that if they do not actually substantially covary with the outcome, including them in the analysis actually can add noise and reduce precision. As for the triple interaction term, you have to assume that every term can be affected by any other term in the model unless the variables involved in those terms are completely independent. So, for example, even if X2 and X3 are independent of TREAT, if their distributions differ according to POST, then their inclusion in the model could change the value of the coefficient of
i.POST##i.TREAT##c.(X1)