Two way fixed effects and interaction terms

[Milla Hanzon]

Hello Stataforum,

I'm working on a generalized difference difference model with multiple groups and multiple treatment dates:

ln_sales = alpa_i + lambda_t + d*treatment_it

Coded in stata as:

xtreg ln_sales i.week i.store treatment

The variable treatment is an interaction term takes the value 0 if a treatment has happened for store in week t.

I want to see if the treatment effect on sales that happens at night/ the weekends is different from the effect on sales during at daytime an the weekdays.

I have split the sample in two groups, one for daytime and one for nighttime, and run the regression on the two samples.
I want to test wether the effects differ by including the dummy night inn my original dataset and code something like this:

reg ln_sales i.week i.store night treatment treatment#night

And then run the test command:

test treatment = treatment#night

However, as I figured, night must interact with all the variables in my model if I want to obtain the same coefficients using interaction terms as when I ran separate models. The results obtained by running separate regressions make sense in theory, and the results obtained by including the dummy night do do not make much sense. As much as I wish my initial approach is the best one, it feels odd to interact time and individual fixed effects with the dummy variable.

My question is wether it is wrong to run a regression like this:

reg ln_treatment i.week i.store i.week#night i.week#night treatment night treatment#night

It certainly feels like I'm "cheating" in order to obtain the results I want. However, I have never seen an interaction term included in a model with two way fixed effects. Can someone help me understand wether and why it is ok or not to do this?

[Clyde Schechter]

So, first, your proposed -test treatment = treatment#night- is legal Stata syntax, but the meaning of that test is, well, bizarre, and has nothing to do with determining whether or not the treatment effect differs between day and night. That is tested with just treatment#night itself. Just look at that interaction term in the regression model to see how much the treatment effect differs between day and night.

There is absolutely no reason you can't have interaction effects in a model with two-way fixed effects. I really can't explain why because I can't think of any reason why not that might need explaining away.

Now, if you want to replicate the results you got using two separate samples for day and night, you do have interact night with everything in the model. The simplest way to write that code is:

regress ln_sales i.night##(i.week i.store i.treatment)

Note that this includes interaction between night and the store effects, which you did not include in your model. But if you don't include them in your model, you won't get the same results as you got with the two separate samples.

I will note that in most situations, people don't interact the fixed effects because they assume that those fixed effects do not differ between day and night. That may well be why you don't see this done very often. But it is based on an assumption. Based on your understanding of the processes that affect ln_sales in the real world, you need to think about whether that assumption is realistic or not. If you think it is realistic, then you can take the i.week and i.store terms outside the parentheses in my code and have a simplified model that will give results similar to, but not the same as, the two separate regressions for day and night. If you think it is not realistic, then stick with interacting night with the fixed effects.

[Milla Hanzon]

Thank you so much, this makes a lot more sense now. May I ask another question related to this model? I have googled and looked at previous research, but cannot seem to find anything about it.

I'm working with a panel data set where four out of eight stores are subject to treatment. The treatments are at different dates, hence the generalized DiD model specification. I've noticed that the treatment occurs very late for one of the stores relative to the other three stores. This is a store with higher average sales than the other stores. When I export the data to excel and compute average weekly sales for the whole treatment group together, I have a lot of observations of high sales before treatment and few observations of high sales after treatment. Therefore, the average total sales for the stores seen as one group are high before and low after treatment, and it looks as if the percentage increase in sales is very low compared to the average of the percentage increases for the four stores analyzed separately.

Will the fixed store effects correct for this, or do I have to drop observations in my data set in order for it include the same amount of weeks for each store before and after treatment?

[Clyde Schechter]

Because you are using a fixed-effects model, your estimates of treatment effect will be based on what happened within-stores, so it will not be affected by the level of sales in the stores: just on how they changed within-store from before treatment to after.

If you have different numbers of observations for each store, and if the number of observations for a store is related to the change in sales from pre- to post-treatment, then you may well get a distorted estimate of the average treatment effect because the stores with more observations will exert more influence on the overall estimate. In that case, and if the parameter you interested in estimating is the equally weighted average treatment effect across all stores,then it would make sense to restrict the data to an equal number of observations per store. Ideally, that decision would have been made prior to analysis, because there is always the concern that whatever way you go about removing "excess" observations could be construed as cherry-picking the data to get the results you want. So you need a clearly explained and obviously unbiased plan for how you would go about weeding out the excess observations. I would also urge you to see if the effect within store really is associated with the number of observations for the store: it very well may not be. If it isn't, then there is no problem and you can proceed with your unbalanced data.

Now, I may have answered a question you weren't asking. If you have equal total numbers of observations per store, and the concern is just that some of the observations are more or less equally balanced between before and after, but in other stores there are more pre- observations and fewer post-observations, that would not bias your results. It would somewhat decrease the precision (efficiency) of your estimation, but it would not be a bias and I wouldn't make any changes to the data or the analysis in response to it.

[Milla Hanzon]

Again, thank you so much!

Back to my original question. Why is this bizarre? I see tests like this all the time. My tutor recommended doing this, maybe I misunderstood him?

So, first, your proposed -test treatment = treatment#night- is legal Stata syntax, but the meaning of that test is, well, bizarre, and has nothing to do with determining whether or not the treatment effect differs between day and night. That is tested with just treatment#night itself. Just look at that interaction term in the regression model to see how much the treatment effect differs between day and night

[Clyde Schechter]

Really, you see this all the time? I've never seen it.

-test treatment = treatment#night- tests whether or not the difference between the treatment and control groups in daytime is equal to the difference between the treatment group in day time and the treatment group at night. Again, legal, and possible, but it's hard to imagine a situation where that would be of interest. And, as I say, I've never seen it.

[Milla Hanzon]

I guess I need to talk to my tutor about it, then. Again, thanks for all the help!