psmatch2 versus 'teffects psmatch'

[Ali Wang]

I've used psmatch2 in one analysis looking at the effect of health insurance enrollment on use of health services. A nice feature of psmatch2 is that I can call on pstest after psmatch2 to assess the comparability of the treament and control groups in terms of the specified covarites before and after matching.

Lately I was suggested to use 'teffects psmatch' because of its advantage of taking into account the fact that propensity scores are estimated when calculating standard errors. But my questions are:
1. How would I know the quality of matching with teffects psmatch. I could not figure out how to use pstest after it.

2. The second problem I had with 'teffects psmatch' is I got an error message: no propensity-score matches for observation xx(xx is the observation #) within caliper 0.01; this is not allowed. I know psmatch2 automatically excludes unmatched cases but teffects psmatch does not, so I did the following-only using the sample on support when running teffects psmatch. However it did not fix the problem. My stata commands are:

psmatch2 insurance covarlist, outcome(service) caliper(0.01) logit ties
teffects psmatch (service) (insurance covarlist, logit) if _support==1, atet caliper(0.01) vce(iid)

If anyone can help answer these two questions, I would greatly appreciate!!!

[Melissa Garrido]

As far as question #1, I'd recommend a two-step process. You can use the user-written -psmatch2- (from ssc) to evaluate your propensity score's ability to balance covariates across treatment groups and within strata. Once you've fine-tuned your propensity score, you can use -teffects psmatch- to get a treatment effect estimate with corrected standard errors.

More details about this are in an article I recently published with my colleagues -
(Garrido et al. Methods for Constructing and Assessing Propensity Scores. Health Services Research 2014 Apr 30. doi: 10.1111/1475-6773.12182. [Epub ahead of print])

Hope this helps!

[Ali Wang]

Hi Melissa, thanks for your reply. I am going to read your article. Actually we are doing what you suggested-using psmatch2 to check the balance and use -teffects psmtach- to estimate SE. I understood the average effect of treatment on the treated should be the same between psmatch2 and -teffects psmatch-, despite the different SE they give. owever, we did not get exactly the same ATT although it is very close for some outcomes. For my second question above, do you have any suggestion? Does what I did seem right to you? I limited the sample to if _support==1 when running -teffects psmatch-, so that the analysis sample is the same as for the psmatch2. Thanks again for your help!!!

[Melissa Garrido]

My guess (without seeing your output) is that the two commands are choosing slightly different matches to construct your ATT. I'd suggest trying to run -teffects psmatch- on the sample with _support ==1 and _weight !=. That should give you the same sample for both commands (and will hopefully solve your second question as well).

[Ali Wang]

Hi Melissa, thanks!
I did what you suggested. But still i got the message with -teffects psmatch: no propensity-score matches for observation 742 within caliper 0.01; this is not allowed.
But I found there should be a match for obserrvation 742 within the radius of 0.01
. list _support _weight in 742

+----------------------+
| _support _weight |
|----------------------|
742. | On support 1 |
+----------------------+

Interestingly, when I changed to cliper=0.011 (not far from 0.01), -teffects psmatch- worked. Do you think there is something wrong with the program when caliper=0.01?
The balance is essentially the same between caliper=0.01 and 0.011. So it is probably fine to use 0.011

With caliper=0.011 specified in both psmatch2 and -teffects psmatch- I expected to have the exactly same ATT between the two, but they are not, the SE is not so different.


----------------------------------------------------------------------------------------
Variable Sample | Treated Controls Difference S.E. T-stat

.--------------------------+-----------------------------------------------------------
facbirth Unmatched | .739762219 .470904208 .268858012 .022405751 12.00
ATT | .73414305 .624395288 .109747762 .029842774 3.68
----------------------------+-----------------------------------------------------------
Note: S.E. does not take into account that the propensity score is estimated.

-teffects psmatch--results are as below:


facbirth | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
ATET |
v481 |
(yes vs no) | .0960414 .0320514 3.00 0.003 .0332218 .158861
------------------------------------------------------------------------------


Do you think the results from the two should be exactly the same if using the same sample? thanks!!

[Melissa Garrido]

The results should be the same if you are using the same sample. Have you verified that exactly the same observations were chosen for both -teffects psmatch- and -psmatch2-?

I think rounding might be responsible for your differences according to caliper choice. Within -teffects psmatch-, there is an option to change the tolerance of the caliper. From the help file:
"pstolerance(#) specifies the tolerance used to check the overlap assumption. The default value is pstolerance(1e-5). teffects will exit with an error if an observation has an estimated propensity score smaller than that specified by pstolerance()."

Hope this helps!

[T Jäger]

Melissa, Thank you. Your article mentioned above contributed significantly to the understanding of propensity score matching.