RMSPE in Synth package

[Nicole Gorton]

Hi all,

I am wondering anyone has been able to replicate manually the values for the Root Mean Squared Prediction Error as calculated by Hainmuller's Synth package. I have not been able to do so. If I compute the pre-period RMSPE by hand, using only the values output by the program (Y_treated and Y_controlled, the time series used to generate the plots) and calculating, for the pre-period only, the sum of the difference between the treated outcome and the synthetic control outcome (then that difference squared), divided by the number of pre-treatment periods (then the square root of that).

But based on the below, it should match the program's output. Below is an excerpt from the ado file -- the code that generates the RMSPE:

/* Compute loss and transform to RMSPE */
 mat `loss' = (`Ztr' - `Zco' * `wsol')' * ( `Ztr' - `Zco' * `wsol' )
 mat `loss' = `loss' / rowsof(`Ztr')
 mata: roottaker("`loss'")
 mat rowname `loss' = "RMSPE"

I believe these are as follows:

• Ztr -- the matrix of pre-treatment characteristics for the treated unit
• Zco * wsol, the pre-treatment characteristics of the control units times the weights that come from the optimization section of the program
• rows of Ztr -- the number of pre-treatment period

This should yield the same pre-period RMSPE that I described calculating by hand above (I think -- please correct me if I am wrong on any of the above).

Any ideas about what is going on?

[Nicole Gorton]

In case anyone else had this issue, I've figured it out (that is, was able to match their results). It seems that when they calculate the RMSPE (which is in fact a pre-period RMSPE, a fact that is not obvious from the documentation) they exclude the year of implementation. This seems reasonable, but it consequently also seems strange that the literature matches in the year of implementation, suggesting that that is considered a pre-year over which we would want to calculate the RMSPE. Any thoughts on that? Have others found the same?

[Jesse Wursten]

Sorry for the necromancy, but I ran into this post while looking into synth. I just wanted to mention that AFAIK the literature does not match on year of implementation. This would be silly, as then you'd try to match out any treatment effect. (just to avoid confusion for potential future visitors)