Conducting leave-one-out test (Synthetic control method).

[Sebastian Hansen]

Dear Statalist

I want to ask you if anyone are familiar with conducting the leave-one-out robustness tests for a synthetic control method analysis in STATA - like Abadie, Diamond & Hainmueller 2015?

I have conducted the placebo studies in accordance to this post: https://www.statalist.org/forums/for...control-method, but I have not been able to find any help online regarding the leave-one-out test.

I know i am supposed to create a loop, where I iteratively reestimate the baseline model to construct the synthesization by excluding high weighted control units. - But I simply cannot get the coding correct.

I hope you can help

Thanks in advance
Sebastian Hansen

[Dimitriy V. Masterov]

I think the post you cited is doing a different exercise. It sets aside the treated unit(s), samples one control unit from the donor set and estimates the SC effect for it as if it was treated (the gray lines). Then it puts it back and samples another one. If these false placebo effects look as large as the estimated one, your actual effect is probably just noise. This procedure is actually automated by wrapper command for -synth- called -synth_runner- that you can find at SSC, rather than doing it with a loop yourself. I think one of the example in the help file replicates the figure in the post.

It sounds like you want to repeat the baseline analysis while excluding one or more highly weighted units from the potential donor set. This is a different kind of robustness check. But it is not clear how you want to go about, so you will need to clarify.

[Sebastian Hansen]

Hi Dimitriy,

Thanks for your reply.

Exactly what you say - I want to repeat the baseline analysis while excluding the highest weighted units from the donor pool successively. My thought was to create a loop (alike the false placebo studies) that repeats the baseline analysis while excluding the highest weights.

I have made some progress (I think)

I have tried by typing this (I exclude donor unit 37, then 22 and then 3). Stata does the estimation, but I cannot grasp how to merge the graphs into one:

forval i=1/1 {
synth p s r bnp indb k BNPind skatpct p(2006) p(2005) p(2004) p(2003) p(2002) p(2001) p(2000) p(1999) p(1998) p(1997), trunit(`i') trperiod(2007) counit(2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 38 39 40 41 42 43 44 45 46 47 48) xperiod(1997(1)2006) nested fig keep(synth_`i')

synth p s r bnp indb k BNPind skatpct p(2006) p(2005) p(2004) p(2003) p(2002) p(2001) p(2000) p(1999) p(1998) p(1997), trunit(`i') trperiod(2007) counit(2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48) xperiod(1997(1)2006) nested fig keep(synth_`i')

synth p s r bnp indb k BNPind skatpct p(2006) p(2005) p(2004) p(2003) p(2002) p(2001) p(2000) p(1999) p(1998) p(1997), trunit(`i') trperiod(2007) counit(2 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48) xperiod(1997(1)2006) nested fig keep(synth_`i')

synth p s r bnp indb k BNPind skatpct p(2006) p(2005) p(2004) p(2003) p(2002) p(2001) p(2000) p(1999) p(1998) p(1997), trunit(`i') trperiod(2007) xperiod(1997(1)2006)nested fig keep(synth_`i')
}

I am a novice in working with Stata, and it is my first time with SCM so I apologise if my question or procedure is completely off.

Thanks in advance
Sebastian

[Dimitriy V. Masterov]

When you exclude 22 in round 2, is 37 back in? Or is do you want to exclude both 22 and 37 in round 2, and then 22, 37, and 3 in round 3?