Counterfactual analysis using a correlated random effects approach

[Maxime Bercholz]

Dear Statalisters,

I am investigating the impact of a policy on a series of non-negative outcomes using panel data on over 2,000 households observed over up to 78 months (unbalanced panel). I am using an interrupted time series design because everyone in the sample was exposed to the policy at the same time. The generic model allows for an immediate intercept change and a change in the time trend after the policy came into effect.

An exponential conditional mean model makes more theoretical sense than a linear model, so I am estimating Poisson models. Specifically, following advice from this thread, I am using correlated random effects models that I estimate by pooled Poisson.

I am interested in the average difference between the fitted values and counterfactual predictions in the policy period, so my question is whether I should set both the policy variables (an indicator variable and its interaction with time) and their panel-level averages to zero in the counterfactual scenario or only the policy variables, leaving the panel-level averages untouched. I assume that because the linear combination of the panel-level averages acts as a control for the household fixed effects and those are not supposed to have changed, it should be the latter, but I am not entirely sure.

To make things a little more concrete, here is what I am doing:

poisson y c.month##policy x1 ... xk month_bar policy_bar monthxpolicy_bar x1_bar ... xk_bar d2-d78, vce(cluster panel_id)
margins if policy == 1, expression(predict(n) - predict(n) / exp(_b[1.policy] + _b[month#1.policy] * month))

where policy is an indicator variable for the policy period and d2-d78 are dummies for the number of observations on each household (as recommended in the aforementioned thread).

predict(n)

gives the fitted values exp(a1 * month + a2 * policy + a3 * month * policy + xb), and

predict(n) / exp(_b[1.policy] + _b[month#1.policy] * month)

gives the counterfactual predicted values exp(a1 * month + xb), with x = x1 ... xk month_bar policy_bar monthxpolicy_bar x1_bar ... xk_bar d2-d78.

[Maxime Bercholz]

Never mind, I think this paper by Wooldridge has the answer in a section on average partial effects: leave the panel-level averages untouched.

[Jared Greathouse]

If everyone was exposed, you really can't do counterfactual analysis because there's no control group you can evaluate the treated group against. If they were treated at different times arguably there's a case to be made, but if literally there's no pure control, then this isn't good.


If we want to test a cancer drug, and we get 1000 people and give all of them the drug, how can we know that the changes we see are due to the drug? No, we'd need to compare them against comparable units that either were never treated or had not yet been treated, so ITS isn't a legal description for your design here.

[Maxime Bercholz]

Thanks Jared. I am aware of this limitation. While a controlled analysis is (always) needed to infer causality, it's not the case that nothing can be learned from a non-controlled ITS analysis. Although ITS is generally applied to time series data, I don't see why the basic idea can't be extended to panel data.

[Jared Greathouse]

Well wait a minute, I'm not saying it can't work with panel data, that's what CITS is for. However, if we do have panel data, then this does not absolve us from needing to construct a counterfactual in a principled way.


I guess my question for you, is are you sure that the normal ITS single assumption makes sense here, that the previous trends of the treated unit are a good comparison for the postintervention trends of the target unit?