How to allow for time-varying treatment effects and heterogeneous treatment subgroup effects in generalized DD?

[Sofie Appelkvist]

Hi all,

I am running a generalized DD to estimate the effect on infant mortality rate (imr) of a health policy implemented at the municipal-level with staggered implementation 2010 - 2017.

Following advice on a previous thread, I have run the regression as follows:
Code:

xtreg imr i.active_treat i.year Z, fe vce(cluster mun)

where treat =1 if the municipality ever implemented the policy and =0 otherwise, active_treat =1 in all periods when the policy is active and =0 for all periods before policy adoption, as well as =0 in control municipalities for all periods. Z is a vector of municipal time-variant characteristics, such as population, GDP, literacy rate, etc.

My first question is how to allow for time-varying treatment effects? Can I simply create an index variable, s, indexing lead periods before treatment and a variable, m, indexing lag periods? For example:

xtreg imr i.active_treat i.year i.s i.m Z, fe vce(cluster mun)

Secondly, the policy differed across treated municipalities. Treated municipalities received treatment type1, type2, or both type1 and type2. Is there a way to incorporate this into the model to evaluate the heterogeneous treatment effects across treatment type? A DDD wouldn't work considering only treatment groups receive type1 or type2. Is the only way to estimate the regression on a subsample by treatment type?

Many thanks for the help!

[Clyde Schechter]

As you don't really explain how you would set up your s and m variables, I won't comment on that part other than to see that, in principle, you can do this. Whether it makes sense is another question. In most environments there are also secular trends in infant mortality, which would be better represented by having a continuous variable for time, or possibly something continuous but very flexible like a cubic spline. (-help mkspline-)

Concerning the different types of treatments, create two dichotomous 0/1 variables, treat1, and treat2. The controls are set to zero on both of those. Let active be a variable that is 1 during time periods when any treatment is active, and 0 otherwise (and hence always 0 for controls). Then your model becomes:

xtreg imr i.active##i.treat1##i.treat2 Z, fe vce(cluster mun)

I have left out i.year for the reasons I set out in the first paragraph of this response, but if you disagree, you can put it back.

One other thought: if you have not just the calculated infant morality rate, but both the numerator and denominator thereof, a somewhat better model would be:

xtpoisson numerator i.active##i.treat1##i.treat2 Z, exposure(denominator) fe vce(cluster mun)

[Sofie Appelkvist]

Thank you for your response.

To clarify regarding s and m variables: I have yearly panel data for 8 periods. s would be a variable indicating lead periods prior to treatment. i.e. if a municipality adopted treatment in 2013, s=3 in 2010, s=2 in 2011, and so forth. m would indicate periods after treatment, i.e. for a municipality adopting treatment in 2013, m=1 in 2014, m=2 in 2015, and so forth. I would like to examine if the treatment effect dissipates over time (as outlined here). I am also struggling with how to formally assess the common trends assumption behind my DD framework and was thinking of regressing on the lead variables to test if the coefficient estimates are statistically different from zero. But like you said, this would likely pick up the secular trends in infant mortality.

When running:

 
 xtreg imr i.active##i.treat1##i.treat2 Z, fe vce(cluster mun)

how do I interpret the treatment effect on a municipality receiving both treat1 and treat2? Is it the coefficient on active + coefficient on active x treat1 + coefficient on active x treat2? (In my original post I mistakenly assumed the triple-difference approach would not work in a case with two treatment types, but thank you for clarifying.)

I have calculated the numerator and denominator of the IMR, I will give the poisson a try.

Many thanks.

[Clyde Schechter]

OK, your s and m variables make sense that way, although, again, I would question the use of i.year and argue in favor of c.year or maybe a cubic spline on year to capture the secular trend.

To interpret the treatment effect when both treatments are given:

It's the coefficient of active + coefficient of active#treat1 + coefficient of active#treat2 + coefficient of active#treat1#treat2.

However, easier than doing all that yourself:

margins treat1#treat2, dydx(active)

run after the regression will give you the treatment effect in all four groups: control, treat1 only, treat2 only, and both treat1 and treat2. And this will be easier still than hand-calculating marginal effects if you do the Poisson regression.

[Sofie Appelkvist]

I am able to run the regression and receive coefficient estimates for active, active#treat1, active#treat2, and active#treat1#treat2. The variables are defined as mentioned above. However, I have problems with the margins command.

Stata returns

. xtreg imr i.active##i.treat1##i.treat2 c.year $xlist, fe vce(cluster mun)
note: 1.treat1 omitted because of collinearity
note: 1.treat2 omitted because of collinearity
note: 1.treat1#1.treat2 omitted because of collinearity

Fixed-effects (within) regression               Number of obs     =     10,599
Group variable: mun                             Number of groups  =      2,643

R-sq:                                           Obs per group:
     within  = 0.0048                                         min =          1
     between = 0.0223                                         avg =        4.0
     overall = 0.0077                                         max =          8

                                                F(13,2642)        =       2.07
corr(u_i, Xb)  = -0.1685                        Prob > F          =     0.0133

                                        (Std. Err. adjusted for 2,643 clusters in mun)
--------------------------------------------------------------------------------------
                     |               Robust
                 imr |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
---------------------+----------------------------------------------------------------
            1.active |   3.260755   3.208429     1.02   0.310    -3.030532    9.552042
            1.treat1 |          0  (omitted)
                     |
       active#treat1 |
                1 1  |   -2.85559   3.208041    -0.89   0.373    -9.146116    3.434936
                     |
            1.treat2 |          0  (omitted)
                     |
       active#treat2 |
                1 1  |   3.084889   4.593839     0.67   0.502    -5.922997    12.09278
                     |
       treat1#treat2 |
                1 1  |          0  (omitted)
                     |
active#treat1#treat2 |
              1 1 1  |    .398865   5.466633     0.07   0.942    -10.32045    11.11818
                     |
                year |  -.7095146   .7115828    -1.00   0.319     -2.10483    .6858013
               lnpop |  -18.74337   23.35126    -0.80   0.422    -64.53197    27.04523
               lnpib |   8.239275   4.214173     1.96   0.051    -.0241383    16.50269
              school |   .8244576   .4481574     1.84   0.066    -.0543174    1.703233
            lit_rate |    .653547   .3133427     2.09   0.037      .039125    1.267969
               water |  -.0704599   .0959504    -0.73   0.463    -.2586054    .1176857
               waste |  -.1375418   .0630175    -2.18   0.029    -.2611104   -.0139731
                 bhu |  -.2800495    .439762    -0.64   0.524    -1.142362    .5822632
                 bfp |   .2858642   .8888372     0.32   0.748    -1.457023    2.028751
               _cons |   1520.498   1249.737     1.22   0.224    -930.0646    3971.061
---------------------+----------------------------------------------------------------
             sigma_u |  130.18798
             sigma_e |  52.633233
                 rho |   .8595145   (fraction of variance due to u_i)
--------------------------------------------------------------------------------------
​​​​​​

Margins returns:

. margins treat1#treat2, dydx(active)

Average marginal effects                        Number of obs     =     10,599
Model VCE    : Robust

Expression   : Linear prediction, predict()
dy/dx w.r.t. : 1.active

-------------------------------------------------------------------------------
              |            Delta-method
              |      dy/dx   Std. Err.      z    P>|z|     [95% Conf. Interval]
--------------+----------------------------------------------------------------
0.active      |  (base outcome)
--------------+----------------------------------------------------------------
1.active      |
treat1#treat2 |
         0 0  |          .  (not estimable)
         0 1  |          .  (not estimable)
         1 0  |          .  (not estimable)
         1 1  |          .  (not estimable)
-------------------------------------------------------------------------------
Note: dy/dx for factor levels is the discrete change from the base level.

I have also tried running the regression as its full factorial interaction, but it did not solve the problem.

xtreg imr i.active i.treat1 i.treat2 i.active#i.treat1 i.active#i.treat2 i.active#i.treat1#i.treat2 c.year $xlist, fe vce(cluster mun)

Any idea as to what I am doing wrong?

[Clyde Schechter]

Oh, sorry, you need to add the -noestimcheck- option to the -margins- command here.

[Sofie Appelkvist]

Ah, now it works. Thanks again for all the help, much appreciated!

[Sofie Appelkvist]

Hi again Clyde, I am a bit confused in interpreting the outputs of the -margins- command and was hoping you could help clarify.
To quickly recap - I have yearly panel data at the municipal level. I am investigating the effect of a policy on infant health outcomes. The policy has two variations, treat_b and treat_c, and municipalities received either treat_b, treat_c, or treat_b and treat_c. I am estimating:

 xtreg pt_rate i.active##i.treat_b##i.treat_c i.active##ib0.m i.uf#c.year c.year $xlist, fe vce(cluster mun)

('active' is a dummy indicating if treatment is active in a given municipality, interaction between 'active' and post-treatment period index 'm' allows for varying treatment effects across post-treatment periods, and finally, I include state-specific linear time trends and a vector of municipal time-varying characteristics.)

I run the regression estimating preterm birth rate, 'pt_rate', followed by the -margins- command to examine total treatment effect for treat_c municipalities.

. xtreg pt_rate i.active##i.treat_b##i.treat_c i.active##ib0.m i.uf#c.year  c.year $xlist,
>  fe vce(cluster mun)
note: 1.treat_b omitted because of collinearity
note: 1.treat_c omitted because of collinearity
note: 1.treat_b#1.treat_c omitted because of collinearity
note: 1.active#0b.treat_b#0b.treat_c identifies no observations in the sample
note: 1.active#1.treat_b#1.treat_c omitted because of collinearity

Fixed-effects (within) regression               Number of obs     =     41,259
Group variable: mun                             Number of groups  =      4,594

R-sq:                                           Obs per group:
     within  = 0.2835                                         min =          1
     between = 0.0000                                         avg =        9.0
     overall = 0.0003                                         max =         11

                                                F(47,4593)        =          .
corr(u_i, Xb)  = -1.0000                        Prob > F          =          .

                                          (Std. Err. adjusted for 4,594 clusters in mun)
----------------------------------------------------------------------------------------
                       |               Robust
               pt_rate |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-----------------------+----------------------------------------------------------------
              1.active |  -12.73694   1.774357    -7.18   0.000    -16.21553   -9.258347
             1.treat_b |          0  (omitted)
                       |
        active#treat_b |
                  1 1  |   .6007268   1.344369     0.45   0.655    -2.034883    3.236336
                       |
             1.treat_c |          0  (omitted)
                       |
        active#treat_c |
                  1 1  |    9.08057   1.166875     7.78   0.000     6.792934    11.36821
                       |
       treat_b#treat_c |
                  1 1  |          0  (omitted)
                       |
active#treat_b#treat_c |
                1 0 0  |          0  (empty)
                1 1 1  |          0  (omitted)
                       |
                     m |
                    1  |  -8.893779    2.32937    -3.82   0.000    -13.46046   -4.327094
                    2  |  -22.69337   2.295547    -9.89   0.000    -27.19375     -18.193
                    3  |  -28.22951   2.641436   -10.69   0.000    -33.40799   -23.05103
                    4  |   -38.6572   2.501469   -15.45   0.000    -43.56128   -33.75311
                    5  |  -50.84688   2.674828   -19.01   0.000    -56.09083   -45.60293
                       |
              active#m |
                  1 1  |  -4.604368   2.400766    -1.92   0.055    -9.311024    .1022878
                  1 2  |  -1.661484   2.342567    -0.71   0.478     -6.25404    2.931073
                  1 3  |  -5.955851   2.617789    -2.28   0.023    -11.08798   -.8237263
                  1 4  |  -5.794992   2.432464    -2.38   0.017    -10.56379   -1.026193
                  1 5  |    -5.3163   2.591158    -2.05   0.040    -10.39622   -.2363847
                       |
             uf#c.year |
                   12  |   5.120935   1.914184     2.68   0.007     1.368214    8.873656
                   13  |   3.034471   1.400332     2.17   0.030     .2891476    5.779794
                   14  |   -2.30205   3.712947    -0.62   0.535    -9.581211    4.977111
                   15  |   3.646681   .8730351     4.18   0.000     1.935112    5.358249
                   16  |   2.003491   2.047442     0.98   0.328     -2.01048    6.017462
                   17  |   1.259055   .7419595     1.70   0.090    -.1955421    2.713652
                   21  |   2.639142    .690205     3.82   0.000     1.286009    3.992276
                   22  |   .4396392   .6766646     0.65   0.516    -.8869485    1.766227
                   23  |   3.599579   .5969541     6.03   0.000     2.429262    4.769896
                   24  |    2.29521   .7111873     3.23   0.001     .9009406    3.689478
                   25  |   -4.54995   .8262489    -5.51   0.000    -6.169795   -2.930105
                   26  |   1.754983     .59359     2.96   0.003     .5912615    2.918705
                   27  |   1.785575   .7477182     2.39   0.017     .3196881    3.251462
                   28  |  -1.318051   .6893006    -1.91   0.056    -2.669411    .0333096
                   29  |   1.598073   .5981723     2.67   0.008     .4253675    2.770778
                   31  |  -.3550174   .5829512    -0.61   0.543    -1.497882    .7878471
                   32  |  -1.104455   .6534799    -1.69   0.091     -2.38559    .1766798
                   33  |   .4474965   .6670983     0.67   0.502    -.8603368     1.75533
                   35  |  -.6057525   .5792639    -1.05   0.296    -1.741388    .5298832
                   41  |  -.4212407   .5977123    -0.70   0.481    -1.593044    .7505626
                   42  |  -.2926705   .6164231    -0.47   0.635    -1.501156     .915815
                   43  |  -1.238908   .6175603    -2.01   0.045    -2.449623   -.0281929
                   50  |   2.401418   .9375664     2.56   0.010     .5633372    4.239499
                   51  |  -.9786431   .7757805    -1.26   0.207    -2.499546    .5422595
                   52  |  -.8429493   .6404185    -1.32   0.188    -2.098477    .4125787
                   53  |  -.6089324   .5501442    -1.11   0.268    -1.687479    .4696146
                       |
                  year |   8.929403   .5983327    14.92   0.000     7.756383    10.10242
                 lnpop |   -4.50226   5.635892    -0.80   0.424    -15.55132    6.546797
                 lnpib |   10.14913   1.397575     7.26   0.000     7.409215    12.88905
            mother_age |  -.1625143   .1066819    -1.52   0.128    -.3716621    .0466335
                school |   .1396663   .2149947     0.65   0.516    -.2818265    .5611592
                 water |   .0405115   .0294792     1.37   0.169     -.017282    .0983049
                   bfp |   .1560142   .0692123     2.25   0.024     .0203249    .2917035
                  vacc |  -.0323114   .0111262    -2.90   0.004     -.054124   -.0104987
                   bhu |  -.4539506    .314297    -1.44   0.149    -1.070124    .1622226
                 _cons |  -18594.54   499.9495   -37.19   0.000    -19574.68    -17614.4
-----------------------+----------------------------------------------------------------
               sigma_u |  3565.3255
               sigma_e |  34.190725
                   rho |  .99990804   (fraction of variance due to u_i)
----------------------------------------------------------------------------------------

. margins treat_c, dydx(active) noestimcheck

Average marginal effects                        Number of obs     =     41,259
Model VCE    : Robust

Expression   : Linear prediction, predict()
dy/dx w.r.t. : 1.active

------------------------------------------------------------------------------
             |            Delta-method
             |      dy/dx   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
0.active     |  (base outcome)
-------------+----------------------------------------------------------------
1.active     |
     treat_c |
          0  |  -14.02784   1.282081   -10.94   0.000    -16.54067     -11.515
          1  |  -4.947265   .8897804    -5.56   0.000    -6.691203   -3.203328
------------------------------------------------------------------------------
Note: dy/dx for factor levels is the discrete change from the base level.

According to my understanding, I interpret the regression output as follows: The effect of active treatment is a decrease in pt_rate of 12.737, on average. The additional treatment effect from being in a municipality receiving treat_c is 9.081 compared to overall treatment effect. The total treatment effect for treat_c municipalities is -12.737 + 9.081 = -3.656, i.e. a reduction in pt_rate of 3.656 as compared to overall treatment effect.
When running the -margins- command, the estimate on 1.treat_c is -4.947.

What is the command telling me here? Is my interpretation of the regression output incorrect?

[Clyde Schechter]

The reason for the discrepancy is that variable active is also involved in other interactions, not just the one with treat_b and treat_c. Consequently, there is, in your model, no such thing as the marginal effect of active among those receiving treat_c. There is a separate marginal effect for each value of m. What -margins- is doing is considering the fact that active also interacts with m, and is giving you an average marginal effect over the different values of m, whereas what you calculated by summing coefficients is just the marginal effect when m = 0.

Aside: One of the warnings that Stata gave you worries me:

note: 1.active#1.treat_b#1.treat_c omitted because of collinearity

This implies that, at least after removing any observations that have missing values for one or more of the variables in the regression command, either there are not actually any entities that received both treat_b and treat_c, or, if there are, they are either always active or never active (or perhaps all of them are active, or all of them inactive for some particular values of m, or uf. Since you specifically said in your description that some entities did receive both treatments, I worry that there is a problem with your data. You might want to look at:

by m, sort: tab treat_b treat_c if e(sample)
by uf, sort: tab treat_b treat_c if e(sample)

You might similarly see what's going on with the variables in $xlist. This condition would not invalidate your model, I raise it only because it appears to contradict your description of the study design.

[Sofie Appelkvist]

Thank you for the explanation of margins, this clears up my confusion.

I was also concerned about the collinearity warning on 1.active#1.treat_b#1.treat_c, but assumed it was because there is no reference group. The variable active =1 when either treat_b or treat_c policy is active in a given municipality and year. treat_b takes the value of =1 for all periods if the municipality ever received treat_b. The same goes for treat_c. Hence, by design, there is no observation where active=1 where either treat_b or treat_c or both are also not =1. Is it because with the lack of a reference group, Stata has nothing to estimate in relation to? I want to allow for the variation in treatment effect in municipalities receiving both treatments if possible, but am not sure if I am doing it correctly in my definition of the variables? In the current output I interpret the estimated treatment effect of receiving both treatments as the sum of coefficients: active + active x treat_b + active x treat_c.

I have checked the commands you provided and there are observations in each group for all variables. All vars in $xlist are continuous.

[Clyde Schechter]

Yes, your reasoning is correct, and you do have the proper number of degrees of freedom in the model.

In the current output I interpret the estimated treatment effect of receiving both treatments as the sum of coefficients: active + active x treat_b + active x treat_c.

But that's wrong. That is the estimated treatment effect of receiving both treatment effects only when m == 0. There is no such thing a the effect of receiving both treatments. There are different effects at each level of m. No one of them is privileged unless there is something in the design of your study that makes the effects when m == 0 more important than the effects at other values of m. (But if that's true, why bother observing at other values of m?) If you want to give a name to that coefficient sum, call it the marginal effect of receiving both treatments conditional on m == 0. Or, consider using the average marginal effect (which averages over observations at all values of m), which is what the -margins- command gave you.