Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • -sdid- with long data and few treated units

    Hi,
    I am trying to run a synthetic difference-in-differences (as in Arkhangelsky et al, 2021) analysis in a case where I have many units observed over 20 years, but only two treated units. I am trying to use Stata's user-created -sdid- command to do this.

    Unfortunately, the command generates a "<istmt>: 3200 conformability error" error message after about 15-20 placebo replications.

    Is this a flaw of the command or am I not able to use the synthetic difference-in-differences approach in this data context?

    I have attached my data and pasted my analysis code below.

    Any help is much appreciated - thanks!


    Code:
    * Load data
    // use ".../dataex", clear
    
    * Declare panel
    xtset id year
                                            
    * Run analysis
    sdid dv id year treatment, vce(placebo)
    Attached Files

  • #2
    Hey, can you just show the dataex in the post please so that way we don't need to download it? If it helps, just do
    Code:
    dataex, count(400)

    Comment


    • #3
      Hi Jared,
      I don't think the error will replicate with a smaller version of the dataset. The length seems to be causing the issue, so I wanted to post the whole dataset.

      Comment


      • #4
        Code:
        sdid dv id year treatment, vce(placebo)
        Placebo replications (50). This may take some time.
        ----+--- 1 ---+--- 2 ---+--- 3 ---+--- 4 ---+--- 5
        ..................................................     50
        
        
        Synthetic Difference-in-Differences Estimator
        
        -----------------------------------------------------------------------------
                  dv |     ATT     Std. Err.     t      P>|t|    [95% Conf. Interval]
        -------------+---------------------------------------------------------------
           treatment | -3.05e+02   42.40591    -7.19    0.000   -3.88e+02   -2.22e+02
        -----------------------------------------------------------------------------
        95% CIs and p-values are based on Large-Sample approximations.
        Refer to Arkhangelsky et al., (2020) for theoretical derivations.
        Having looked at this, here's my opinion as an SCM user. You have an unholy amount of observations. It almost amazes me! Way more than we typically see in SCM settings. I don't know if it's causing the error, but even if it weren't, I would not buy your estimates. Why?

        There's no donor selection mechanism beyond the Ridge regression we use to assign the weights, and you have 4001 donors. Holy Mother of God. In other words, I would imagine that SDID would overfit, or be biased, due to such high dimensionality. There are SCM methods that address this, such as principal component regression or Robust PCA SYNTH (the former written by researchers at Columbia and the latter by my friend and colleague, programmed into Python by me).

        Both these methods rely on extracting the low rank donor matrix and use the low rank structure to predict the counterfactual in the post period. Robust PCA Synth even picks a suitable donor pool before estimating (so if you start with 1000 donors, you may end up with only 500 after the donor selection mechanism, and of those, only say 5 may even get any weight). In synthetic studies, Robust PCA SYNTH and PCR out-perform normal SCM. My point is, what you really want to use here is Python! Someday, but not today, I'll have all these newer SCM methods wrapped into a single module that Stata users can access/run on their own.

        Beyond this, I'm quite curious: what are you studying? What calls for such a large dataset for SCM? I'm most curious.

        Comment


        • #5
          Thanks Jared!

          It looks like you were able to run the code that was giving me an error. Can I ask what version of Stata you are using? I am using Stata 17 BE; maybe that's causing the issue?

          I want to study the impacts of policy changes in Oklahoma City and Tulsa that dramatically reduced student suspensions in schools. The data I attached is youth arrest data for each reporting American police department by year. I also have another dataset with average standardized test scores in each school in Oklahoma that has about 500 schools observed annually. I'll look into principal component regression and robust PCA synth more; I am not familiar with them.

          Comment


          • #6
            I'm using 17 MP.

            And yeah, PCR and PCA-SYNTH need python, and even though the former is public, the most straightforward implementation is what I've written for Python.

            Okay, so the idea is that these policies decreased arrests, what I'll do on my end, is I'll run this on my machine.

            Comment


            • #7
              Yeah, that's the idea! I think they might have 17 MP at my school's library; I'll see if I can try it there.

              Comment


              • #8
                Hi Noah Spencer , We fixed this issue which was actually an iteration error when we applied the placebo. Reinstall sdid from github
                Code:
                net install sdid, from("https://raw.githubusercontent.com/daniel-pailanir/sdid/master") replace
                We tested with some examples and yours and it works perfectly, please let me know if this problem was fixed for you!

                Code:
                . sdid dv id year treatment, vce(placebo) seed(2023)
                Placebo replications (50). This may take some time.
                ----+--- 1 ---+--- 2 ---+--- 3 ---+--- 4 ---+--- 5
                ..................................................     50
                
                
                Synthetic Difference-in-Differences Estimator
                
                -----------------------------------------------------------------------------
                          dv |     ATT     Std. Err.     t      P>|t|    [95% Conf. Interval]
                -------------+---------------------------------------------------------------
                   treatment | -3.05e+02   31.77757    -9.59    0.000   -3.67e+02   -2.43e+02
                -----------------------------------------------------------------------------
                95% CIs and p-values are based on Large-Sample approximations.
                Refer to Arkhangelsky et al., (2020) for theoretical derivations.

                Comment


                • #9
                  Originally posted by Jared Greathouse View Post
                  Code:
                  sdid dv id year treatment, vce(placebo)
                  Placebo replications (50). This may take some time.
                  ----+--- 1 ---+--- 2 ---+--- 3 ---+--- 4 ---+--- 5
                  .................................................. 50
                  
                  
                  Synthetic Difference-in-Differences Estimator
                  
                  -----------------------------------------------------------------------------
                  dv | ATT Std. Err. t P>|t| [95% Conf. Interval]
                  -------------+---------------------------------------------------------------
                  treatment | -3.05e+02 42.40591 -7.19 0.000 -3.88e+02 -2.22e+02
                  -----------------------------------------------------------------------------
                  95% CIs and p-values are based on Large-Sample approximations.
                  Refer to Arkhangelsky et al., (2020) for theoretical derivations.
                  Having looked at this, here's my opinion as an SCM user. You have an unholy amount of observations. It almost amazes me! Way more than we typically see in SCM settings. I don't know if it's causing the error, but even if it weren't, I would not buy your estimates. Why?

                  There's no donor selection mechanism beyond the Ridge regression we use to assign the weights, and you have 4001 donors. Holy Mother of God. In other words, I would imagine that SDID would overfit, or be biased, due to such high dimensionality. There are SCM methods that address this, such as principal component regression or Robust PCA SYNTH (the former written by researchers at Columbia and the latter by my friend and colleague, programmed into Python by me).

                  Both these methods rely on extracting the low rank donor matrix and use the low rank structure to predict the counterfactual in the post period. Robust PCA Synth even picks a suitable donor pool before estimating (so if you start with 1000 donors, you may end up with only 500 after the donor selection mechanism, and of those, only say 5 may even get any weight). In synthetic studies, Robust PCA SYNTH and PCR out-perform normal SCM. My point is, what you really want to use here is Python! Someday, but not today, I'll have all these newer SCM methods wrapped into a single module that Stata users can access/run on their own.

                  Beyond this, I'm quite curious: what are you studying? What calls for such a large dataset for SCM? I'm most curious.
                  "I would imagine that SDID would overfit, or be biased, due to such high dimensionality."

                  @Jared: would you mind explaining that, intuitively and perhaps mathematically? I ask because I'm about to apply synthetic DiD to a dataset with 10 time periods and 147 countries, and I'm quite new to the synthetic control / doubly robust methods... Curious to read your answer!

                  I am curious as well to understand this:

                  Robust PCA Synth even picks a suitable donor pool before estimating (so if you start with 1000 donors
                  If you are implementing a doubly robust method with a matching component based on minimising Euclidian distance in the difference between treated outcome and weighted control outcome, why do you reduce the donor pool straight off bat before starting to apply weights?

                  Thanks in advance for your help! I have a lot to learn in this (modern) strand of literature!

                  Comment


                  • #10
                    Thank you Daniel PV!

                    Comment


                    • #11
                      Tomorrow (or maybe tonight if I'm bored), I will go into details of what I mean. I'll even give examples to illustrate. Essentially, machine learning methods can help quite a lot in SCM settings (I'll even send you my paper, if you'd like).

                      Comment


                      • #12
                        Some of these applications seem to be synthetic control gone wild. I have difficulty believing that one cannot narrow down the donor set substantially in many applications. Surely geography plays a key role in choosing similar schools or similar countries. Why are we giving everything over to the computer when it isn't clear what the statistical properties will be? My understanding is that the inference underlying sdid requires that the treated cells are "growing" with the total sample size. In Noah's example, the total sample size is over 8,000 but the number of treated cells is about 40 -- and most are over time. Do we have any evidence that the placebo variance estimator works well in such cases?

                        Comment


                        • #13
                          I have difficulty believing that one cannot narrow down the donor set substantially in many applications.
                          To me, the issue is not whether it can be done, the issue is how do we do so in an objective and rigorous manner. Different people will have different standards about what specific aspects of geography/population are "good enough" for a unit to be a donor. In a paper I'm currently working on, we discuss donor selection in detail and offer solutions (and simulations).

                          Maxence Morlet if you can't see what I'd inboxed you, email me (see my page) and I'll resend my reply.

                          Comment


                          • #14
                            I partly agree, Jared, but there are tradeoffs. Having lots of donors doesn't suddenly make the asymptotics work better -- likely the opposite, as I think you suggested above. And yet people will report those standard errors and confidence intervals as if they are the truth. I have to believe that we could limit the number of police departments relevant for two school districts in Oklahoma substantially. Are the departments/districts in Oregon, Michigan, and Maine really contenders here? Presumably rules for student suspensions vary a lot by state, so why would one even look outside Oklahoma in this kind of example? Maybe throw in Texas and Kansas as a robustness check. If we're going to use the entire United States, why not throw in Canada and Mexico, too?

                            Comment


                            • #15
                              Originally posted by Jared Greathouse View Post
                              To me, the issue is not whether it can be done, the issue is how do we do so in an objective and rigorous manner. Different people will have different standards about what specific aspects of geography/population are "good enough" for a unit to be a donor. In a paper I'm currently working on, we discuss donor selection in detail and offer solutions (and simulations).

                              Maxence Morlet if you can't see what I'd inboxed you, email me (see my page) and I'll resend my reply.
                              Jared Greathouse : received, thank you very much! I will give all these papers a read, thanks a lot for the references & your comments!

                              Comment

                              Working...
                              X