Question pertaining CSDID package for estimating heterogenous treatment effects

[SiQing Xu]

Hi. I am studying the effect of a policy "active_treat" on outcome "acceptance". I have attached an excerpt of the data I'm working with. I am using repeated cross sections between 2010-2019. Each observation is a health care facility, and some facilities may be observed in multiple years (but I cannot confirm this as the data does not include facility level identifiers). Depending on my specification N ranges between 70,000-90,000 approximately. (approximately 10,000 observations per year)

active_treat is a binary that is equal to 1 if my policy of interest is in place
acceptance is my outcome of interest, basically whether the facility accepts a certain type of payment.
UmemployPercent is a continuous measure of a state's unemployment rate
FPLbelow200 is a continuous measure of a percent of state's population living at or below 200% of federal poverty level (FPL)
dem_gov is a binary indicator for whether the governor of the state, at the year of interview, was a democrat
private_profit is a binary indicator for whether a facility is for-profit or non-profit
State is a string variable indicating State
State_num is a numeric variable indicating State
year_num is the year the facility interviewed

The timing of policy adoption (the treatment) varies, leading to concerns regarding heterogenous treatment effects when using typical TWFE.

However I obtain the following with many values omitted. I understand that always treated units are ignored, however, I'm not sure why so many cohorts are completely omitted. I know in my data, that many states implemented the policy in 2014.

I would appreciate any insights regarding this. Thank you.

. tab year_expand

year_expand |      Freq.     Percent        Cum.
------------+-----------------------------------
          0 |     32,935       33.81       33.81
       2010 |      2,206        2.26       36.08
       2011 |      1,967        2.02       38.10
       2014 |     46,951       48.20       86.30
       2015 |      8,055        8.27       94.57
       2016 |      3,848        3.95       98.52
       2017 |      1,441        1.48      100.00
------------+-----------------------------------
      Total |     97,403      100.00

. csdid accept_medicaid active_treat UnemployPercent FPLbelow200 dem_gov private_p
> rofit, t(year_num) g(year_expand) cluster(State_num)
Units always treated found. These will be ignored
xxxxxxxxxx.x......xxxxxxxxxxxxx...x.xxxx.....xxxx.
.xxx
Difference-in-difference with Multiple Time Periods

                                                        Number of obs = 42,326
Outcome model  : least squares
Treatment model: inverse probability
                             (Std. err. adjusted for 51 clusters in State_num)
------------------------------------------------------------------------------
             | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
g2010        |
 t_2010_2011 |          0  (omitted)
 t_2010_2012 |          0  (omitted)
 t_2010_2013 |          0  (omitted)
 t_2010_2014 |          0  (omitted)
 t_2010_2015 |          0  (omitted)
 t_2010_2016 |          0  (omitted)
 t_2010_2017 |          0  (omitted)
 t_2010_2018 |          0  (omitted)
 t_2010_2019 |          0  (omitted)
-------------+----------------------------------------------------------------
g2011        |
 t_2010_2011 |          0  (omitted)
 t_2010_2012 |   .0248805   .0225803     1.10   0.271    -.0193762    .0691371
 t_2010_2013 |          0  (omitted)
 t_2010_2014 |   .0856161   .0135796     6.30   0.000     .0590006    .1122316
 t_2010_2015 |   .0367183   .0317725     1.16   0.248    -.0255546    .0989912
 t_2010_2016 |  -.0099473   .0389002    -0.26   0.798    -.0861903    .0662956
 t_2010_2017 |   .0954174   .0898442     1.06   0.288    -.0806741    .2715088
 t_2010_2018 |   .1302272    .093643     1.39   0.164    -.0533097    .3137641
 t_2010_2019 |   .1726064   .1083423     1.59   0.111    -.0397406    .3849534
-------------+----------------------------------------------------------------
g2014        |
 t_2010_2011 |          0  (omitted)
 t_2011_2012 |          0  (omitted)
 t_2012_2013 |          0  (omitted)
 t_2013_2014 |          0  (omitted)
 t_2013_2015 |          0  (omitted)
 t_2013_2016 |          0  (omitted)
 t_2013_2017 |          0  (omitted)
 t_2013_2018 |          0  (omitted)
 t_2013_2019 |          0  (omitted)
-------------+----------------------------------------------------------------
g2015        |
 t_2010_2011 |          0  (omitted)
 t_2011_2012 |          0  (omitted)
 t_2012_2013 |          0  (omitted)
 t_2013_2014 |          0  (omitted)
 t_2014_2015 |  -.0567201   .0603586    -0.94   0.347    -.1750209    .0615806
 t_2014_2016 |   -.023359    .042255    -0.55   0.580    -.1061774    .0594593
 t_2014_2017 |  -.0333143   .0558177    -0.60   0.551     -.142715    .0760863
 t_2014_2018 |          0  (omitted)
 t_2014_2019 |   .1916928   .1054545     1.82   0.069    -.0149942    .3983798
-------------+----------------------------------------------------------------
g2016        |
 t_2010_2011 |          0  (omitted)
 t_2011_2012 |          0  (omitted)
 t_2012_2013 |          0  (omitted)
 t_2013_2014 |          0  (omitted)
 t_2014_2015 |  -.0078914   .0168854    -0.47   0.640    -.0409861    .0252033
 t_2015_2016 |  -.0108443   .0084924    -1.28   0.202     -.027489    .0058004
 t_2015_2017 |   .0162566   .0249807     0.65   0.515    -.0327047    .0652179
 t_2015_2018 |    .036864   .0352495     1.05   0.296    -.0322237    .1059517
 t_2015_2019 |   .0185099   .0383815     0.48   0.630    -.0567164    .0937363
-------------+----------------------------------------------------------------
g2017        |
 t_2010_2011 |          0  (omitted)
 t_2011_2012 |          0  (omitted)
 t_2012_2013 |          0  (omitted)
 t_2013_2014 |          0  (omitted)
 t_2014_2015 |   .0036289   .0063906     0.57   0.570    -.0088965    .0161544
 t_2015_2016 |   .0434556   .0183421     2.37   0.018     .0075057    .0794054
 t_2016_2017 |          0  (omitted)
 t_2016_2018 |          0  (omitted)
 t_2016_2019 |          0  (omitted)
------------------------------------------------------------------------------
Control: Never Treated

See Callaway and Sant'Anna (2021) for details

. estat all
Pretrend Test. H0 All Pre-treatment are equal to 0
chi2(3) =     6.9191
p-value  =     0.0745
Average Treatment Effect on Treated
------------------------------------------------------------------------------
             | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
         ATT |   .0294693   .0350939     0.84   0.401    -.0393135     .098252
------------------------------------------------------------------------------
ATT by group
------------------------------------------------------------------------------
             | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
    GAverage |   .0239749    .032544     0.74   0.461    -.0398102      .08776
       G2011 |   .0765026    .036721     2.08   0.037     .0045308    .1484744
       G2015 |   .0135714    .054365     0.25   0.803    -.0929821    .1201249
       G2016 |   .0151966   .0203914     0.75   0.456    -.0247698    .0551629
------------------------------------------------------------------------------
ATT by Calendar Period
------------------------------------------------------------------------------
             | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
    CAverage |    .035675   .0282508     1.26   0.207    -.0196956    .0910456
       T2012 |   .0248805   .0225803     1.10   0.271    -.0193762    .0691371
       T2014 |   .0856161   .0135796     6.30   0.000     .0590006    .1122316
       T2015 |  -.0330791   .0572429    -0.58   0.563    -.1452731    .0791149
       T2016 |  -.0168588   .0254302    -0.66   0.507    -.0667011    .0329835
       T2017 |   .0038231   .0516333     0.07   0.941    -.0973763    .1050224
       T2018 |   .0645174   .0588036     1.10   0.273    -.0507355    .1797704
       T2019 |   .1208257    .072803     1.66   0.097    -.0218656     .263517
------------------------------------------------------------------------------
ATT by Periods Before and After treatment
Event Study:Dynamic effects
------------------------------------------------------------------------------
             | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
     Pre_avg |   .0017626   .0103904     0.17   0.865    -.0186022    .0221274
    Post_avg |   .0593985   .0376136     1.58   0.114    -.0143228    .1331198
         Tm2 |   .0036289   .0063906     0.57   0.570    -.0088965    .0161544
         Tm1 |  -.0001037    .017558    -0.01   0.995    -.0345167    .0343093
         Tp0 |  -.0362626   .0388409    -0.93   0.351    -.1123892    .0398641
         Tp1 |   -.001911   .0278988    -0.07   0.945    -.0565916    .0527696
         Tp2 |  -.0037375   .0477223    -0.08   0.938    -.0972715    .0897965
         Tp3 |   .0383862   .0333676     1.15   0.250     -.027013    .1037855
         Tp4 |   .1498078   .0965945     1.55   0.121    -.0395139    .3391295
         Tp5 |  -.0099473   .0389002    -0.26   0.798    -.0861903    .0662956
         Tp6 |   .0954174   .0898442     1.06   0.288    -.0806741    .2715088
         Tp7 |   .1302272    .093643     1.39   0.164    -.0533097    .3137641
         Tp8 |   .1726064   .1083423     1.59   0.111    -.0397406    .3849534
------------------------------------------------------------------------------

* Example generated by -dataex-. For more info, type help dataex
clear
input float(acceptance active_treat) double UnemployPercent float(FPLbelow200 dem_gov private_profit) str20 State long State_num int year_num
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 1 "Alabama" 1 2010
. 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
0 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 1 "Alabama" 1 2010
. 0 10.4 .4023 0 0 "Alabama" 1 2010
0 0 10.4 .4023 0 1 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 1 "Alabama" 1 2010
0 0 10.4 .4023 0 1 "Alabama" 1 2010
1 0 10.4 .4023 0 1 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
. 0 10.4 .4023 0 0 "Alabama" 1 2010
0 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 1 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
. 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 1 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 1 "Alabama" 1 2010
. 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 1 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
0 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 1 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 1 "Alabama" 1 2010
. 0 10.4 .4023 0 1 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
. 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
0 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
0 0 10.4 .4023 0 1 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
. 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 1 "Alabama" 1 2010
. 0 10.4 .4023 0 0 "Alabama" 1 2010
0 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 1 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
0 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
0 0 10.4 .4023 0 1 "Alabama" 1 2010
1 0 10.4 .4023 0 0 "Alabama" 1 2010
end
label values accept_medicaid accept_medicaid
label def accept_medicaid 0 "No", modify
label def accept_medicaid 1 "Yes", modify
label values dem_gov dem_gov
label def dem_gov 0 "Non Democrat", modify
label values private_profit private_profit
label def private_profit 0 "Non Profit", modify
label def private_profit 1 "For Profit", modify
label values State_num State_num
label def State_num 1 "Alabama", modify

[FernandoRios]

Can you do the following
reg accept_medicaid active_treat UnemployPercent FPLbelow200 dem_gov private_profit year_num year_expand State_num
tab year_num year_expand if e(sample)

[SiQing Xu]

Hi Fernando. Thank you for responding. I have attached the output from your suggestion.

. reg accept_medicaid active_treat UnemployPercent FPLbelow200 dem_gov private_profit year_num year_ex
> pand State_num

      Source |       SS           df       MS      Number of obs   =    62,647
-------------+----------------------------------   F(8, 62638)     =    195.98
       Model |  144.184296         8   18.023037   Prob > F        =    0.0000
    Residual |  5760.49111    62,638  .091964799   R-squared       =    0.0244
-------------+----------------------------------   Adj R-squared   =    0.0243
       Total |   5904.6754    62,646  .094254628   Root MSE        =    .30326

---------------------------------------------------------------------------------
accept_medicaid | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
----------------+----------------------------------------------------------------
   active_treat |  -.0004239   .0048101    -0.09   0.930    -.0098517    .0090039
UnemployPercent |  -.0204144    .001224   -16.68   0.000    -.0228135   -.0180153
    FPLbelow200 |   .1612686   .0283754     5.68   0.000     .1056528    .2168844
        dem_gov |  -.0269903   .0025302   -10.67   0.000    -.0319495   -.0220311
 private_profit |  -.0878189   .0036844   -23.84   0.000    -.0950404   -.0805975
       year_num |  -.0084222   .0009629    -8.75   0.000    -.0103095   -.0065349
    year_expand |  -.0087781   .0012328    -7.12   0.000    -.0111944   -.0063617
      State_num |   .0018861   .0000937    20.12   0.000     .0017024    .0020698
          _cons |   35.59906   2.816561    12.64   0.000      30.0786    41.11953
---------------------------------------------------------------------------------

. tab year_num year_expand if e(sample)

           |                            year_expand
  year_num |      2010       2011       2014       2015       2016       2017 |     Total
-----------+------------------------------------------------------------------+----------
      2010 |       206        210      4,499        811        344        132 |     6,202 
      2012 |       348        278      6,327        995        535        198 |     8,681 
      2014 |       287        233      6,154      1,034        519        172 |     8,399 
      2015 |       278        234      5,980      1,039        499        182 |     8,212 
      2016 |       271        231      5,682        989        483        175 |     7,831 
      2017 |       250        220      5,493        957        441        172 |     7,533 
      2018 |       262        230      5,504        959        456        180 |     7,591 
      2019 |       269        266      5,989      1,022        463        189 |     8,198 
-----------+------------------------------------------------------------------+----------
     Total |     2,171      1,902     45,628      7,806      3,740      1,400 |    62,647

[FernandoRios]

There is the problem
you have no data for 2013 so there is no g-1 for 2014, thus the attgt can’t be estimated

[SiQing Xu]

Hi, thank you for that insight. That makes sense. With this specific survey I'm working with there is no data for the years 2011 and 2013.

In that case, do you know if there is an appropriate way to account for concerns about dynamic TE with the csdid package with data like this?

One way I am considering getting around it is by limiting consideration to states that adopted the policy in a short period of time, say in 2014 and the never adopters.

[FernandoRios]

Hi SiQing
So, the results from CSDID will still be consistent. It will just not use any information from 2014 for the estimation.
However, one way you "fix" this is to copy duplicate 2012 into 2013. This only if the data is for an untreated period (either never treated or not yet treated).
Doing this still allows you estimate ATTGTs (same assumptions) but with the caveat that the Pretrend is slightly off for this group (since you are using a base different from the True one.
Hope this helps
Fernando

[SiQing Xu]

Thank you, that's very helpful. I have three follow up questions:

1. I want to clarify I'm understanding this what you've said above correctly. Do you mean states that adopted the policy in 2014, as of right now, are dropped from analyses because there is no 2013 data observed?

2. Within each cohort, there are indicators such as t_2014_2015. Do you mind explaining what each of these indicators mean? I would like to have a better understanding for understanding what is going on when values of these are omitted.

3. When using the csdid package, is there ever a concern about statistical power?

[FernandoRios]

1. Yes, that is what happens because ATTs are estimated using the period before treatment. (2013 for those treated in 2014)
2. Those means which years are used for the analysis. T2014_2015 means that the "post_period" comes from 2015, whereas the "pre_period" comes from 2014.
3. Statistical power is always a problem. csdid is very flexible in avoiding standard DID problems. but, because is flexible and conservative, it also losses a lot of data that other approaches use for ATT estimation.
For instance. Standard TWFE use ALL pretreatment periods to estimate post treatment ATTGTs. CSDID only uses the one before treatment. However, (doing what i suggest) it could use the last one available before treatment).
HTH
Fernando

[SiQing Xu]

Hi Fernando. I really appreciate all this help. I have ruminated about what was going on with the csdid package for quite a bit, and your clarifications and suggestions were incredibly helpful. Once again thank you.

[Richard Thomas Boylan]

In matching estimators, one is usually required to assess covariate balance. My understanding is that CSDID is a matching estimator of some sorts. How does one go about assessing covariate balance? Presumable the estimator would have to provide the weights used so that one can check whether the weighed Xs are the same for the treated and control units.

[FernandoRios]

you are correct
the methods dripw and drimp both produce doubly robust estimators, which basically use an IPW to balance characteristics so the controls look like recipients.
Now, because each ATTGT (the long table you see when using csdid) is estimated separately, each one has different weights, None are stored.

The closer you can get to answering this question is to examine them case by case using drdid. Where the logit and ipt models are saved, and where you can produce the IPW weights (one of the post estimation commands).

Now, the huge advantage of drimp method is that when you use it, your covariates are matched perfectly at means. So, you may not need to even worry about balance.

See Sant'Anna and Zhao 2020 paper on DR DID, where they provide the appropriate reference for the properties of IPT estimator (inverse probability tilting).

HTH
Fernando

[Richard Thomas Boylan]

I understand that CSDID ensures proper balance. However, the question is how to come up with a table of summary statistics for the estimates obtained in CSDID: if one uses the entire sample, the summary statistics are misleading since they end up using units that may not be used in CSDID or given very little weight. So, if I understand your suggestion correctly, I would convert the staggered DID into a series of DID models. I then estimate each DID separately with drdid and save the weights. Then, I compute the summary statistics for CSDID by only looking at the units that get some minimal average weight.

[FernandoRios]

Yes
but even then it may be difficult
for different comparisons,(cohort and year) different subsamples are used with different weights
so not sure what would be the best approach to do a balance comparison

[Richard Thomas Boylan]

I guess one can first use a propensity score to determine the likelihood that a unit is treated at any time period (as a function of the time-average values of the Xs). Then, eliminate all units for which the probability of being treated is below some threshold (say 1%). Finally, re-run CSDID on only the remaining control units. It would be worrisome if the results changed too much. So, that would provide a robustness test. Moreover, one could compute the average value of the Xs for that particular subsample of the treated units. [I am examining the case with a small number of treated units and a very large number of control units. So, this is why I would keep the number of treated units unchanged, but reduce the number of control units]

[Mateus Maciel]

Thanks for this post, Richard! I am facing the same issue with a dataset I am working on. FernandoRios, do you know if there is any other version of the staggered estimators that we can use to deal with the fact that there is a gap in the treatment?