Proper statistical procedure for repeated RCTs on the same group (without replacement)

[Lucas Mation]

I posted this question on stats.stackexchange, here, but don't think the answers there are adequate . I am cross-posting que question her on StataList hoping the broader audience (epidemiologists, applied econometricians, econometricians, etc.. ) may be able to help.

##############Question #####################

Proper statistical procedure for repeated RCTs on the same group (without replacement)


Setting:

• Longitudinal data on outcome Yi,t of a group of individuals, i={1,...,N}, over time, t={1,...,T}
• On this gruop, a sequence of RCTs (r={1,...,R}) staggered over time is applied (with R< N).
• Each RCT is measuring the effect of a specific treatment and treats a small percentage of the population.
• Ence an individual is treated in a certain round (r=x) then he is no longer eligible for other future RCTs (rounds).

I am trying to measure the specific impact of each treatment (eq 1), as well as the average impact of the treatments (eq 2).

1. Yi,t,r = ALPHAr*TREATEDi,r + BETAr(TREATEDi,r*I_POST_TREATMENTt,r)
2. Yi,t = ALPHA*TREATEDi + BETA *(TREATEDi *I_POST_TREATMENTt)

Take estimation of equation 1 for round r=x. Part of the control group of that round will eventually be "contaminated" being treated in a future rounds of the RCT. Because trials treat a small percentage of the population each round (without replacement), and there aren't that many rounds, a significant proportion of the control group remains untreated over the entire period.


What is the correct statistical procedure to adopt here?

Is there a technical term for this sort of setting?

Any relevant literature I should look at?

[Clyde Schechter]

If an individual is treated in round r, is the treatment continued on that individual in subsequent rounds? If no, is the effect of the treatment administered in round r expected to persist in subsequent rounds?

Is there some common factor involved in the different treatments? For example, are they different doses or intensities of the same agent/intervention? If not, what is the rationale for attempting to estimate an overall combined effect of these treatments?

If it were the same treatment being initiated at different times, this would be called a stepped wedge design. There is an abundant literature on the stepped wedge. However, these are all different treatments (or at least they are partially heterogeneous), and I'm not aware of any particular name for this, nor have I ever seen it used before.

[Lucas Mation]

Clyde, tks for this.

The actual study is measuring the impacts different of housing projects aimed at poor people, which are allocated through lotteries. Lotteries draw from the same pool of applicants (excluding previous applicants who received homes). I measure outcomes in every period, but lotteries are spread unevenly over time (and do not occur in most periods). In the end, just 20% of the applicants get treated over the different housing project lotteries, so a significant control group remains after the end of the study. And, yes, treatment is persistent.

So the setting is somewhat different from a stepped wedge design, but I will take a look at that literature, thanks for pointing that out.

I would like to estimate project specific impacts and an overall impact. Or estimate impacts by project level variables, such as distance to downtown (which could be a project characteristic interacted with treatment in Eq 2).

Do you, or any one else, recognize this setting?

When estimating, project specific effects, Eq1, for round r, could I just remove control individuals who end up selected in future rounds? My reasoning is that this future contamination will be driven by a random process, so it will not bias the composition of the control group in any way.

[Clyde Schechter]

Actually, to me this simply sounds like multiple staggered implementations of the same intervention and it sounds like a classic example of the stepped wedge design. I don't get why you consider each round to be a different intervention. The only thing that seems like it might not fit is that you say that in some rounds there is no lottery. If so, what does happen in that round? Or is it just a round of outcome measurements without any new interventions? If so, it's still a stepped-wedge design.

There is an extensive literature on the analysis of the stepped wedge design, including analyses of step-specific and overall treatment effects. I think this sounds suitable for you, unless I am missing something. It is, however, a bit unwieldy to summarize in a Forum post. If you do a search on stepped-wedge design you will have no difficulty finding articles about it at all levels of statistical sophistication.

[Lucas Mation]

Tks again.


Answering your specific questions:

"If so, what does happen in that round? Or is it just a round of outcome measurements without any new interventions?" This is correct. We measure outcomes, say, monthly. And some months have lotteries for a specific housing project (in fact there can be more than one lottery, for multiple projects, in a month). Others months have nothing.

"sounds like multiple staggered implementations of the same intervention". Housing projects can be very different and I would like to look at heterogeneity in effects. They vary in distance to town, quality of surrounding schools, sanitation conditions. These project characteristics I can measure. But there are others dimensions that I may now be able to measure, or anticipate at this point. This is the rationale for housing project specific effects.

From a very superficial search of stepped wedge design (Wikipedia) I was under the impression that eventually all individuals got treated. In my case I do have a control group that never gets treated. But I will definitively look into it more. Tks!

[Clyde Schechter]

I was under the impression that eventually all individuals got treated.

That is probably the most common case, but it is not a necessary aspect of stepped wedge designs. You certainly can have a control group that never gets treated.

[Lucas Mation]

Tks a lot!