Heterogenous Treatment Effects

[Timo Vogel]

Hi there,

I did a completely randomized experiment and have pre and post experimental data. The average treatment effect of the experiment is basically zero. Moreover, I checked as much of other observable variables for potential heterogenous treatment effects. Without any effect. However, you can think about tons of unobservable variables that may influence the treatment effect and lead to heterogeneity. I was wondering whether you are aware of an easy tool to check this?

Of course I did my homework and checked the web. Solutions that appear there seem to require some decent knowledge on baysian statisitcs (which I do not have).

My idea was then to try to estimate individual treatment effects and then look at the distribution of those. However, estimating an individual treatment effect (this, 1 treatment observation) (even with many pre treatment periods and many control individuals) seem to be highly biased using OLS with time fixed effects.

Anyhow, I would love to discuss this issue on how to tackle potential unobservalbe heterogeneity in treatment effects (preferable with methods that are implemented in Stata).


Looking forward to your answers and many thanks!

[Clyde Schechter]

I would probably do something like this. Let's call your outcome variable outcome (I'll assume it's continuous). You have longitudinal data on a bunch of subjects, the subjects being identified by a variable I'll call id. Each subject has a pre- and a post-randomization observation. You have a variable called pre_post that distinguishes these. And a variable, call it arm, distinguishes the treatment and control groups. I would run the overall trial analysis with a mixed-effects model and a random slope for the pre-post variable. Then I would calculate the random slopes for each person. That will give you an individual-level treatment effect, and then you can explore that graphically or analytically as desired. So something like this:

mixed outcome i.arm##i.pre_post || id: pre_post
predict u v, reffects

The variable u now contains the random component of the slope for each subject. That is, the individual treatment effect is given by _b[1. pre_post] + _b[1.pre_post#1.arm]*arm + u. If you want to calculate the later expression, of course, you can. But it's probably simpler for your purposes just to work with u as a proxy for individual treatment effect.

Note: In exploratory analyses of u (or u + etc.) be aware that each subject has two observations, each containing the same value of u. So all your analyses should be qualified by something like -whatever_command_involving_u if pre_post == 1- so you don't double count everything.

All of that said, I'm not endorsing the pursuit of this goal. It sounds misguided overall to me, except perhaps as a way to explore data that might provoke a hypothesis that will be tested in a completely separate trial.

[Marcos Almeida]

I did a completely randomized experiment and have pre and post experimental data. The average treatment effect of the experiment is basically zero. Moreover, I checked as much of other observable variables for potential heterogenous treatment effects. Without any effect. However, you can think about tons of unobservable variables that may influence the treatment effect and lead to heterogeneity. I was wondering whether you are aware of an easy tool to check this?

I hope you won't take it amiss. But, it seems everything was done properly: a completely randomized experiment, with pre and post data, and yet, well , a non-significant difference in the effect of treatment between groups..

All in all, I gather it truly is a trial with "negative results". So what?

Searching on "tons of unobservable variables that may influence the treatment effect and lead to heterogeneity" would not provide a positive result, at least for two reasons: lack of power and, shall this be a subgroup analysis, it is not allowed to may inferences related to the primary outcome, secondary outcome, etc. That could be only useful for "raising hypotheses" to be tested in future studies.

Last but not least, in spite of the "high credulity" towards Bayesian analysis (NB: mine, as well!), I'm afraid this amazing counterpart of the frequentist approach wouldn't do wonders under the scenario described in #1.

To end this post on a realistic note, I wonder why not simply publishing the trial' s negative results. Mostly refused in the past, there are a bunch of journals that "care much" for such results nowadays.

Hope that helps.

P.S.: edited to comment that, provided the study is "completely randomized", we may surely consider that the observed as all as the unboservable variables are equally distributed between treatment groups.

[Jeff Wooldridge]

It's really a difficult task, and I doubt you can make it convincing. Looking for heterogeneity as a function of observables is always a good idea, but, like Clyde, I think it is a bit misguided to try to determine if there's unobserved heterogeneity. For one, how would you use this information if it is unobserved? In non-experimental studies, heterogeneity is discovered when there's a difference between the average treatment effect and the ATE on the treated. But those must be the same in your case.

I like Clyde's suggestion, but with one caveat: you can only learn about heterogeneity if you are willing to assume the shocks over time are independent and identically distributed. It's usually a strong assumption. So, you might conclude there's unobserved heterogeneity when in fact it's just heteroskedasticity or serial correlation.

My view is that if you have a bunch of good background and pre-treatment socioeconomic variables, and the effect seems not to depend on them, then it would be strange if the treatment effect depended on heterogeneity unrelated to all unobservables. And, again, what would you do with such information from a policy perspective? "Some people are affected by the treatment but I can't tell you which ones"?

Edit: And now that I read Marcos' post, I find myself in agreement with him, too.

[Catalina Saavedra]

Hello everyone,

I am really new to Treatment Effects. I am trying to perform a 2sls model in a cross-sectional data where both my outcome and treatment endogenous variables are binary. I have two instrumental variables, one discrete (it can only take 5 values) and the other one is binary. My control variables includes continuous and binary variables.

I know that one way of testing for heterogeneous treatment effects is estimating a linear Marginal Treatment effect to test if it has a constant slope.
However, I would like to know other way of testing for heterogenous treatment effects and therefore the external validity of LATE, without estimating MTE.

Many thanks in advance,