How to define treatment & control groups properly?

[Anna Bokun]

I’m working on a project examining the effect of a 2016 cash transfer on fertility.

Who is eligible for the cash?
All families with: 1.) 2+ children, or 2.) 1 low-income or disabled child.

The data doesn’t have a variable indicating who got the cash transfer, so as I understand it, I would be doing an “intent to treat” analysis by defining the treatment & control groups based on eligibility.

However, I keep getting stuck on how to define the treatment & control groups. I guess my question is since the cash transfer is universal for ALL families with 2+ kids, what would be the control group then? Theoretically, there should be two similar groups of families with 2+ kids (one who get the cash transfer and the other who don’t), but that’s not possible in this case?

Comparing eligible families (2+ kids or 1 poor/disabled kid) to ineligible families (1 kid that is not poor/disabled or zero kids) would violate one of the core assumptions of causal inference (that the treatment and control groups be similar and only differ in the “treatment”).

I think I’m getting tripped up by how the cash transfer is both universal and birth-dependent.

I’m exploring using a linear probability model with FE or a DID model, but not sure if a DID makes sense? Is synthetic control more appropriate? Any thoughts on modeling strategies?

More context: The data comes from a household survey, which I’ve organized into a panel with fertility histories for each childbearing-aged woman (e.g. each woman has 17 observations, or 18 years containing her time-variant birth information). I have data from 2010-2018 and the program started in 2016. The program grandfathers in anyone who falls in either one of two eligibility categories. The cash transfer is not means or work tested.

[Jared Greathouse]

I love a good, well posed theoretical stats question. You are interested in what's called the difference-in-discontinuities design. Why?

In DD, we have treatment groups defined only by time, before and after treatment. In RD, you have treatment defined by a cutoff. Here, you have both. You have data on folks before the had over 2 kids, or 1 disabled kid.... and after. The ideal control group in this instance, are families who are JUST like the treated families on the other side of the discontinuity. I'll give a different example.


Let's say we give ice cream to kids based off how they do on an exam. If you get over an 87, boom, chocolate ice cream for you. Well, if we have 10,000 kids......... how different is a kid who made an 85 or an 86, from one who made an 87? Or even an 89 or 90? In large enough numbers, and over a long enough time series, we'd ideally expect confounding to be eliminated at the threshold (that is, by adjusting for gender or other predictors).


In your case, your forcing/running variable is how many kids you have. So, in large enough numbers and adjusting for appropriate predictors, you want a mixture of DD an MD RD

[Clyde Schechter]

You are not getting tripped up at all. You are facing squarely the fact that in this data it is not possible to define a control group. When a program is implemented in a way that is universally applied to everyone eligible, no good experimental or quasi-experimental methods can be applied to evaluate its effects.

Let's think about some other ways of approaching it. Was the cash benefit ongoing, or a one time thing? And was it initiated for all recipients at the same time? If it was ongoing and some recipients received it later than others, you could consider a stepped-wedge design, which does not require a control group, only that different groups of people undertake the intervention in different time periods.

Is there some other jurisdiction that was covered by the survey and did not enact a similar cash benefit but had similar fertility trends prior to 2016 and is, in a general sense, reasonably comparable to the one you are dealing with? If so, a difference-in-differences analysis might be applied, with the other jurisdiction as the control group.

It is hard to imagine that there could be a good instrumental variable to use for a relationship between initial number of children and subsequent fertility, but perhaps your imagination in this area is broader than mine?

[Jared Greathouse]

I don't understand why defining a control group isn't possible here. Unless I've misunderstood, which is quite possible, if we have data on 10k (say) families and 2k are eligible, wouldn't the control group be the families that are similarly situated to the 2k families (i.e, only one kid and alike on background covariates)?

Anna writes

the cash transfer is universal for ALL families with 2+ kids

so unless this dataset only consists of families with 2 or more kids or a disabled child, surely there are families in this dataset that are not eligible for the program, right?


Again I may be misunderstanding, but it seems like the dataset must consist of eligible and ineligible families, right?

ADDED:

Comparing eligible families (2+ kids or 1 poor/disabled kid) to ineligible families (1 kid that is not poor/disabled or zero kids) would violate one of the core assumptions of causal inference (that the treatment and control groups be similar and only differ in the “treatment”).

This simply isn't true. Causal inference is sorta what my specialty is, in certain applications. We can have differences in the treated and control groups and, so long as you can argue that the groups are balanced, you're fine,


Comparing eligible families to ineligible families is quite literally at the heart of what you're doing. That's precisely what causal inference is here, you're using a well selected comparison group to compare treated units to. The whole point of adjusting for covariates in matching or other applications is so we can make the treated and untreated groups as alike as possible. So if I were you, I would construct my control group around folks who a) have only 1 kid and are b) alike the treated families on all other observable characteristics as you can possibly match on.

[Clyde Schechter]

Re #4: Yes, but given that the outcome under study is fertility, it is simply not credible that anybody with only one kid or none, no matter how alike on background covariates, can be considered to be similarly situated for the purposes of this research question. Having 0 or 1 kids vs 2+ kids is nothing like having an 85 or 86 vs 87 or 88 on a test when what's being compared is fertility. Now, every discipline has its own conventions and standards. Maybe that would be acceptable somewhere. But in any journals on epidemiology or reproductive health a paper based on that control group would just get a desk-rejection from an associate editor on day one. Although maybe the associate editor would get a belly-laugh out of it and write the rejection note with a kind thank you.

Added: My #3 crossed with #2.

[Fei Wang]

I tend to agree with Clyde, for a different reason. A control group unit should not be affected by the policy while it seems every family has actually been influenced by the policy no matter whether it's eligible at the very beginning. Families with 2+ children or one low-income/disabled child may increase fertility due to income effects from the cash transfer, while families with one child who is not poor or disabled or with zero kid tend to increase fertility to earn the cash transfer -- an incentive effect. So it's difficult to find pure control group units as the fertility behavior of every family has changed upon the policy, despite different mechanism.

I would suggest looking for an outside control group. For example, if the policy is effective in a state of the U.S., then you may treat families with 2+ kids or 1 low-income/disabled kid from another state which is similar but not having the policy as control group units. If the policy is implemented in a European country, then you may find control families from other similar European countries without the policy.

[Anna Bokun]

Thanks to everyone for their thoughts.

Jared, you write:

"In DD, we have treatment groups defined only by time, before and after treatment. In RD, you have treatment defined by a cutoff. Here, you have both."

While it's true there's a "cutoff" of having 2+ kids or 1 low-income/disabled child, as Clyde points out, having 0/1 kids is very different from having 2+ kids and would violate the assumption that the two groups are characteristically similar. For example, the decision to go from zero children to one child is fundamentally different than higher-order children. A first birth is less elastic to financial incentives, but more sensitive to factors such as the subjective well-being and marital satisfaction of new parents (Twenge, Campbell, and Foster 2004; Testa, Cavalli, and Rosina 2014; Bauer and Kneip 2014; Margolis and Myrskylä 2015; Glass, Simon, and Andersson 2016)

Clyde–

Was the cash benefit ongoing, or a one time thing?

It's an ongoing monthly cash transfer starting in 2016.

And was it initiated for all recipients at the same time?

Yes, any families with 2+ kids or 1 low-income/disabled child qualified at the same time (so there should be no issues with program selection).

Is there some other jurisdiction that was covered by the survey and did not enact a similar cash benefit but had similar fertility trends prior to 2016 and is, in a general sense, reasonably comparable to the one you are dealing with?

The context is a European country and the cash transfer applied to all of the country's regions equally at the same time.

Fei–
Thanks for highlighting the possibility that the fertility behavior of every family changed in response to the cash transfer, which complicates isolating the true impact of the cash transfer. When you suggest using control families from similar countries, are you referring to the synthetic control method?


References
Bauer, Gerrit, and Thorsten Kneip. 2014. “Dyadic Fertility Decisions in a Life Course Perspective.” Advances in Life Course Research 21:87–100.
Glass, Jennifer, Robin W. Simon, and Matthew A. Andersson. 2016. “Parenthood and Happiness: Effects of Work-Family Reconciliation Policies in 22 OECD Countries.” AJS; American Journal of Sociology 122(3):886–929. doi: 10.1086/688892.
Margolis, Rachel, and Mikko Myrskylä. 2015. “Parental Well-Being Surrounding First Birth as a Determinant of Further Parity Progression.” Demography 52(4):1147–66. doi: 10.1007/s13524-015-0413-2.
Twenge, Jean M., W. Keith Campbell, and Craig A. Foster. 2003. “Parenthood and Marital Satisfaction: A Meta-Analytic Review.” Journal of Marriage and Family 65(3):574–83. doi: 10.1111/j.1741-3737.2003.00574.x.
Testa, Maria Rita, Laura Cavalli, and Alessandro Rosina. 2014. “The Effect of Couple Disagreement about Child-Timing Intentions: A Parity-Specific Approach.” Population and Development Review 40(1):31–53.

[Fei Wang]

When you suggest using control families from similar countries, are you referring to the synthetic control method?

Anna, what in my mind is a DiD setting with individual-level datasets from the country of study and from other similar countries. In other words, you may pool families from different countries and work on them with DiD.