ATE ATT and ATU vs. Difference in Differences

[Marc Smith]

Dear all:

We are applying a Difference in Differences (DiD) estimator to study the treatment effect of a healthcare program.
We would like to discuss our results in terms of the usual program evaluation effects like 'average treatment' or 'average treatment on the treated' etc.
To this end, we would like to show how the DiD estimator relates to ATE and/or ATT (and under what precise assumptions).

We are confused because here (http://en.wikipedia.org/wiki/Average_treatment_effect) it says essentially that DiD=ATE, whereas here (http://www.ucl.ac.uk/~uctp39a/Blunde...s-Dec-2007.pdf) on page 18 it says that DiD=ATT.

We would be grateful if anyone can help us with a reference where we can find precise conditions such as 'DiD=ATE if xy holds' and 'DiD=ATT if nn holds'.
(We already read the advanced Wooldridge text but did not get to the answer.)


Thanks for your help!

Marc

[Melissa Garrido]

Marc, you might find the following paper by Andrew Ryan and colleagues useful:

Ryan AM, Burgess, JF, Dimic JB. Why we should not be indifferent to specification choices for difference-in-differences. Health Services Research 2014; online ahead of print (doi: 10.1111/1475-6773.12270).

In particular, it includes this quote:
"DID estimates are typically considered to be average treatment effects on the treated, rather than average treatment effects. This is because DID estimates are generally thought of as applying to a particular group that was treated (rather than to a population that could have been treated)"

Hope this helps,
Melissa

[Marc Smith]

Melissa: Thanks a lot, the article you mention is certainly of interest to us! However, as to DiD and ATT it only formulates a claim, not a technically clean answer...
Kind regards, Marc

[Steve Samuels]

It depends on how the control and treated subjects were selected: 1) If controls were either all untreated or a random sample of untreated; and the treated were all treated or a random sample of all treated; then DID estimates ATE. 2) If controls were selected to resemble the treated subjects, then DID estimates ATT. If, by chance, the treated subjects were chosen to resemble controls, then DID estimates ATU.

[Marc Smith]

Steve: Thanks for your reply! Intuitively, I completely understand your argument. Do you know of a formal proof (e.g. somewhere in an article/journal)?
Kind regards, Marc

[Steve Samuels]

Sorry, I don't. My idea was that in the treatment effect definitions, where one sees, (E(Y(1)) - E(Y(0)), replace each term by the expected before/after difference, e.g.: \( E(Y(1)) = E(Y_{after}(1))-E(Y_{before}(1))\).

[Steve Samuels]

I don't think that my statement about ATE was correct.

1) If controls were either all untreated or a random sample of untreated; and the treated were all treated or a random sample of all treated; then DID estimates ATE.

I believe that the only design that will estimate ATE is a randomized experiment. However, I think that in Situation (1), any of the three treatment effects can be estimated analytically: by reweighting (Nichols, 2008) or by predictive margins.

Reference:
Nichols, Austin. 2008. Erratum and discussion of propensity-score reweighting. Stata Journal 8, no. 4: 532-539.

[Marc Smith]

Thanks, Steve. It will take some time for me to check the references in the paper you mention.

[Paul Visintainer]

Marc, I don't think you'll find a formal proof (famous last words, right?) because it is an interpretation. Guo and Fraser: Propensity Score Analysis, 2010 (Sage) may help. They relate ATE to the intent-to-treat principle. (They reference the text by Shadish, Cook, and Campbell: Experimental and quasi-experimental designs for generalize causal inference. Boston:Houghton-Mifflin, 2002 for this view). Conversely, Guo and Fraser site Heckman & Vytlacil (2005) (Structural equations, treatment effects, and econometric policy evaluation. Econometrica, 73, 669-738), as arguing for ATT for policy contexts. Their argument is that for policy, the average benefit of the intervention among all individuals (ATE) is less important than knowing the benefit specifically among those who participate or among those to whom the intervention is applied.

[Navid Asgari]

Hi Marc,

DiD's final result is essentially an ATET.

I suggest you to read Stata 13's introduction to treatment effects: http://www.stata.com/manuals13/te.pdf

Though the manual says nothing about DiD, it explains the difference between ATE and ATET in great details.

Navid

[Steve Samuels]

Here's the proof that plain DID (no reweighting, no predictive margins) estimates ATT. See Athey and Imbens (2006).

Definitions: \(i\) indexes individuals; Group \(G_i = 0,1\); time \(T_i =0,1\). \(Y_i(0)\) is the, possibly counterfactual, response of person \(i\) in the absence of intervention. \(Y_i(1)\) is the possibly counterfactual, response if person \(i\) got the intervention.

The standard (linear) DID model is:

\[
E(Y_i) = \alpha +\beta\, T_i + \gamma\, G_i + \tau \,G_i T_i \quad \quad \quad \quad (1)
\]
where \(\tau\) is the DID, to be estimated by least squares regression.

In the (possibly counterfactual) absence of intervention, the expected outcome is:
\[
E(Y_i(0)) = \alpha +\beta\, T_i + \gamma\, G_i \quad \quad \quad \quad (2)
\]
In the (possibly counterfactual) presence of intervention, the expected outcome is:
\[
E(Y_i(1)) = \text{E}(Y_i(0)) + \tau \quad \quad \quad \quad (3)
\]
Now, ATT is the expected difference in \(Y_i(0) -Y_i(1)\) for those treated by time 1, i.e. with \(G = 1\) and \(T = 1\). Plugging these values into Equation 1 or Equations 2 & 3 we get:
\[

\text{ATT} = E(Y_i(1) -Y_i(0)\, | \,G=1,T =1) = \\
\\
E(Y_i(1)| \,G =1, T=1) -E(Y_i(0)\, | \,G=1, T =1) = \tau
\]

References:

Athey, Susan, and Guido W Imbens. 2006. Identification and inference in nonlinear difference in differences models. Econometrica 74, no. 2: 431-497.

[Marc Smith]

Steve, that's exactly what I was looking for, thank you very much!
We also overlooked the reference you quote so far, but it is highly relevant to us!
Thanks again,
Marc

[Steve Samuels]

You are welcome, Marc. The statement that the standard DID model estimates ATT is on page 436, but no proof is offered, since its an immediate consequence of that model. I've just spelled it out.

[Melkamu Tadesse]

But, what if one get insignificant ATET and significant ATE after running teffects? And suppose if the research is on national scale program?

[Steve Samuels]

You've responded to a topic about DID that was closed over a year ago. Please start a new thread.