Difference-in-Differences with Multiple Groups and Time Periods

[Adam King]

Hello,

I know that the basic setup for a difference-in-differences estimation that involves two time periods and a control and treatment group is:

Y = B0 + B1(treatment) + B2(posttreatment) + B3 (treatment*posttreatment),

where treatment = 1 if the observation is in the treatment group, and posttreatment = 1 if the time period is after treatment -- with B3 being the parameter of interest/ estimate of the impact of the treatment.

I have 8 years of monthly consumption data (2007-2015) for 16,000 households. I split these households up into 6 treatment groups (binned by lawn size), and want to see the differential impact of a policy that occurred for a two year period (2009-2011), call the dummy "restrictions." I also want to see if there is some habit formation that occurs after the policy ends (as there are 4 years of data after it ends).

Is the correct way to do this to generate a dummy for each lawn size bin (6 treatment groups), then generate 6 interactions between each treatment group dummy and the "restrictions" dummy, and another 6 interactions with a "restrictions_off" dummy that equals 1 for the 4 year time period after the restrictions ended? If so, is this all in one regression, or do I estimate these separately for each lawn size grouping? I am also wondering if I need to have a reference group of some sort? Formally, my options (I believe) are:

A) Y = B0 + B1(treatment_1) + B2(restrictions) + B3(treatment_1*restrictions) + B4(treatment_2) + B5(treatment_2*restrictions) + B6(treatment_3) + B7(treatment_3*restrictions)...

or

B) For i = 1 to 6 separately: Y = B0 + B1(treatment_i) + B2(restrictions) + B3(treatment_i*restrictions)

C) And then run either A) or B) a "restrictions_off" dummy instead of a "restrictions" dummy, or D) include the "restrictions_off" dummy in one of the above?

If I can provide any more detail, please let me know. Thank you.

[Christos Makridis]

I might be misunderstanding, but it sounds like your policy affected everyone? While one approach is to define the treated group as those owning a home, and the control as those renting, that would suffer from a serious selection problem unless you can do a good job predicting selection into homeowner status. Presumably, if you observed some switching of people going from renting to owning, person fixed effects would be ideal. An alternative would be to find out whether the restrictions were only enforced in one area, or if they were only binding for a certain set of homes -- for example, perhaps homes in some blocks dont have much grass.

In either case, I don't think your setup is right -- you just have dummy variables on different bins, but from your description I don't think those bins are groups that received the treatments along different magnitudes.

[Adam King]

Thanks for your response, and sorry for not providing more detail. The two year policy was mandatory outdoor water restrictions, which only affects those households with lawns, and previously, I had one treatment group (those with lawns), and one control group (those without lawns), and that was correct. Everyone is a single family homeowner, but thats not what I am looking for -- water consumption is the left hand side variable. Now, I wanted to see how these restrictions affected those with different lawn sizes, which I was I split lawn sizes up (at various percentiles). Should the control group remain those without lawns every time?

[Clyde Schechter]

It's really not clear to me that homes without lawns represented an appropriate control group in the first place, but be that as it may.

Your research question now, as I understand it, is to determine whether the policy impacted water consumption differently depending on lawn size. For the same reason that it's not clear to me that no-lawn houses were an appropriate control group, it isn't clear to me how including them now would contribute to that goal. But I suppose one could argue that the promulgation of the water-restriction rules created a level of awareness of water conservation that would rub off even on those with no lawn and cause some change in their water consumption as well. So let's go with including them.

Then you have a complicated design. You have a three level time variable: before restrictions, during restrictions, after restrictions were lifted--call that variable era. You have 7 treatment groups (no treatment plus 6 lawn sizes)--call that variable group. I assume your data is longitudinal with periodic (?monthly ?annual ?something else) data over time and the unit of observation is the house Then the analysis goes something like this:

xtset house_id time
xtreg water_consumption i.era##i.group, fe // OR MAYBE re?

I would imagine in this situation there are covariates to add to this model. And I also imagine that there may be time variables that need to be expressed, including the possibility of continuous trends over calendar time, and seasonal variation.

With group having 7 levels and era having 3, you're going to have 6X2 = 12 interaction terms. You will probably lose your mind trying to figure out exactly what each term in the model does. So I think to interpret the results you will want to run -margins- after that, probably also with the -contrast- option.

[Anuradha Saikia]

Clyde Schechter I have considered the time period as era for pre , during and after effect. My basic question is how do I interpret the era and treatment group interaction if it is significant(positive). Does it imply during and after periods of policy implementation were successful?

[Clyde Schechter]

The coefficient of the interaction term is the difference-in-differences estimator of the policy implementation effect. I do not think its statistical significance is of much interest one way or another. If your sample is very large, a tiny, meaningless difference might be statistically significant. If your sample is small, or of good size but outcome measures are noisy, then an effect might be too imprecisely estimated by the data and could lead to a non-statistically-significant estimate.

Moreover, statistical significance is a test of a straw-man null hypothesis that the effect of the policy was zero. But you are, rightly, asking a different question: was the policy successful. It is seldom, if ever, the case that any non-zero effect, no matter how small, would be considered success. Nearly always there is some minimum positive level of effect that one would want to see to declare a policy implementation successful. So first compare the coefficient to whatever the minimum criterion for success is. If the coefficient is bigger, then your best estimate of its effect suggests it is successful, otherwise no. But crucially, examine also the 95% confidence interval. The data are compatible with any actual level of effectiveness in that interval. Do both limits of that confidence interval represent success? If so, then the data are strongly supportive of success. But if the ends of the interval disagree, one is at a successful level and the other unsuccessful, then the data are inconclusive about whether the policy was successful. And of course, if the entire confidence interval consists of values that would be considered failures, then the data suggest the policy failed.

[Anuradha Saikia]

Thank you Clyde Schechter . very helpful information. Also , completely off-topic is it necessary to test for parallel trends in the case of triple difference model (DDD).

[Clyde Schechter]

Also , completely off-topic is it necessary to test for parallel trends in the case of triple difference model (DDD).

Yes.