Different ordering of predictors yields different answers using synth in Stata

[Mike Wallace]

When using the *synth* command in stata and specifying the nested option, changing the order of the predictors yields different weights on the control units. Below is an example:

clear
sysuse smoking
tsset state year
synth cigsale beer(1984(1)1988) lnincome retprice age15to24 cigsale(1988) cigsale(1980) cigsale(1975) ///
, trunit(3) trperiod(1989) nested

synth cigsale cigsale(1988) cigsale(1980) cigsale(1975) beer(1984(1)1988) lnincome retprice age15to24///
, trunit(3) trperiod(1989) nested

I should note that the results don't change when you do not specify the nested option. I'd appreciate if anyone could give their thoughts about why the results change when simply reordering the predictors when using the nested option.

[Joseph Coveney]

What's synth?

[Mike Wallace]

synth is a program used to do synthetic controls.

[Nick Cox]

As the FAQ Advice asks,

If you are using user-written commands, explain that and say where where they came from: the Stata Journal, SSC, or other archives. This helps (often crucially) in explaining your precise problem, and it alerts readers to commands that may be interesting or useful to them.

[Mahta Norouzi]

Mike, did you find any answer for your question? I have recently started using synth command and found the same problem you mentioned here; the order of predictors matter! Please kindly advise if you know why it's like that and how one should deal with that. Thanks!

[Chris Auld]

Hi folks,

No one appears to have answered the OP's question. A student of mine has encountered the same problem: the order in which the covariates are listed affects the estimated weights and thus the results when using the synth command with the "nested" option. As far as I can tell this appears to be an error in the command. Does anyone know of a discussion of, or a solution to, this issue?

Thanks, Chris.

[Michael Makovi]

This is a subject of ongoing research.

One recent paper claims this is a flaw in the synthetic control method's algorithm for optimizing the V-matrix. They claim this problem exists in R, so it's not a problem with Stata, but a problem with the synthetic control method itself.
"Design Flaw of the Synthetic Control Method" by Kuosmanen et. al.
https://mpra.ub.uni-muenchen.de/1063...per_106390.pdf

They also say that changing the order of the donors will change the results as well. This is harder to check in Stata because if you use encode to automatically convert string names into numeric IDs, you cannot control the order of the donors. You would have to manually re-number the donors, or else change the string names to change their alphabetical order. You would have to go out of your way to do this in Stata.

However, I am conducting my own research on this as well. So far, I find that with careful use of different numerical techniques (NR, DFP, BFGS), and use of convergence parameters (margin, tolerance, ltolerance, nrtolerance), the instability of estimates can be mitigated or even eliminated. That is, changing the order of the indicator variables will NOT change the estimates IF you have carefully chosen the right numerical technique and/or convergence parameters. Therefore, I am concluding that the problem is not with the synthetic control method. Rather, it is a consequence of the fact that floating point math is approximate, and numerical optimizers are approximate as well. There is no closed form solution to the synthetic control problem, and different numerical techniques will yield different results.

When you change the order of the indicator variables, you are changing the matrices by moving columns around. Similarly, when you change the order of the donors, you are changing the matrices by moving the rows around. And this can affect how the numerical optimizer converges on an optimum.