rbounds reports non significant results at gamma=1

[Chiara Pollio]

Dear Statalists,
I am using rbounds to assess the sensitivity of the results of a matching to unobservables. Although my results are significants, when I run the command the significance level is never below 0.1, and of course the point estimate is outside the confidence interval since the beginning. Does it mean that the estimate is certainly biased by unobservables? Here I provide an example of the output I obtained.

psmatch2 spinoff, outcome(pricediff) caliper(0.1) pscore(ps1l)

----------------------------------------------------------------------------------------
Variable Sample | Treated Controls Difference S.E. T-stat
----------------------------+-----------------------------------------------------------
----------------------------+-----------------------------------------------------------
pricediff
Unmatched | .407894737 .109204368 .298690369 .061607262 4.85
ATT | .402985075 .15920398 .243781095 .099344525 2.45
----------------------------+-----------------------------------------------------------

. rbounds delta_pricediff, gamma(1 (0.05) 2)
Rosenbaum bounds for delta_pricediff (N = 201 matched pairs)

Gamma sig+ sig- t-hat+ t-hat- CI+ CI-
----------------------------------------------------------------------
1 .99994 .99994 -.09392 -.09392 -.108593 -.081033
1.05 .999983 .999804 -.095892 -.092257 -.112745 -.079246
1.1 .999995 .999448 -.097718 -.090564 -.118823 -.076756
1.15 .999999 .998622 -.099477 -.08866 -.124092 -.073862
1.2 1 .996908 -.101197 -.086956 -.127451 -.071171
1.25 1 .993678 -.10304 -.085319 -.130488 -.067251
1.3 1 .988088 -.104942 -.083561 -.132839 -.0625
1.35 1 .97912 -.107217 -.081998 -.135704 -.052632
1.4 1 .965673 -.109842 -.080499 -.137919 .044753
1.45 1 .946693 -.112745 -.079246 -.141078 .086538
1.5 1 .92132 -.117045 -.07763 -.143247 .185897
1.55 1 .889021 -.121107 -.075324 -.144914 .210046
1.6 1 .84969 -.124355 -.073549 -.146729 .285592
1.65 1 .803683 -.126613 -.072053 -.148268 .310159
1.7 1 .751802 -.129116 -.069184 -.149791 .334762
1.75 1 .69522 -.130952 -.066667 -.151423 .343548
1.8 1 .63537 -.132578 -.063462 -.152804 .348682
1.85 1 .573815 -.134044 -.059003 -.154593 .353084
1.9 1 .512118 -.136478 -.045833 -.15642 .357154
1.95 1 .451733 -.137812 .038333 -.15814 .361538
2 1 .393916 -.139706 .075226 -.160192 .366247

* gamma - log odds of differential assignment due to unobserved factors
sig+ - upper bound significance level
sig- - lower bound significance level
t-hat+ - upper bound Hodges-Lehmann point estimate
t-hat- - lower bound Hodges-Lehmann point estimate
CI+ - upper bound confidence interval (a= .95)
CI- - lower bound confidence interval (a= .95)


Thank you,
Chiara

[Panos Markou]

Hi Chiara,

I am in the same predicament, so I will bump your post up to the top of the thread. I also obtain significance, however the results at Gamma = 1 are insignificant. Hopefully, someone more experienced will be able to solve this for us.

HOWEVER, if I am interpreting this correctly, what this means is that the results are HIGHLY sensitive to omitted variables. In other words, the results you obtain may not hold for even minor unobservables.

In other words, better matching variables need to be found.

Again, hopefully someone else can chime in...

Best and good luck,
Panos

[Chiara Pollio]

Dear Panos,
I finally solved the issue by writing to one of the stata command programmers. He basically told me that it is well possible that rbounds gives such results, but most of the cases it happens when your main results are significant in the limit. This happens because while the estimation of the ATTs is usually done through parametric tecnique, the Rosenbaum bound's approach is basically a non parametric one. Before admitting that the results are high sensitive to unobservables, however, it is necessary to check that the mean value of the delta variable is equal to the ATT effect.

Best,
Chiara

[Erik Alda]

Hi Chiara
Thanks for clarifying. I just run into the same problem. Did it happen that your results were highly sensitive to unobservables after checking the mean value of delta?
Many thanks
Erik

[Sijia Du]

Dear Statalist users,
After running Rosenbaum bounds, I find the results at Gamma=1.05 become insignificant. Does that mean that my result is highly vulnerable to hidden selection bias? In that case, do I need to use the iv approach even if propensity score matching has been performed? However, one effect was significant for PSM methods and non-significant for IV methods (other estimations were consistent for both methods). Then how to deal with this discrepancy result from the two methods? Any guidance and suggestions would be greatly appreciated.
Best wishes, Sijia.