checking matching using pscore vs pstest?

[Hannah Koenker]

Hi everyone,

I am trying to better understand the differences in using pscore to check the balancing property after matching, and in using pstest to check the means of covariates before and after matching. The two approaches are being endorsed by two of my colleagues, but pstest gives a nice matched group, while pscore reveals that the balancing property is not met.

We have run the psmatch2 as follows, where "media_ipc" is the treatment (exposure to a health communication message), and the remaining covariates are age, education, region, ses etc, among households that own at least one net, with replacement and the common support:

psmatch2 media_ipc agecat2 agecat3 educ2 educ3 newprov2 newprov3 newprov4 child6 wealthcat irs_district urban3 if anynet==1, outcome(itnlastnite) neighbor(1) common
There are observations with identical propensity score values.
The sort order of the data could affect your results.
Make sure that the sort order is random before calling psmatch2.
----------------------------------------------------------------------------------------
Variable Sample | Treated Controls Difference S.E. T-stat
----------------------------+-----------------------------------------------------------
itnlastnite Unmatched | .858974359 .733648542 .125325817 .017275211 7.25
ATT | .858974359 .325641026 .533333333 .177182168 3.01
----------------------------+-----------------------------------------------------------
Note: S.E. does not take into account that the propensity score is estimated.

| psmatch2:
psmatch2: | Common
Treatment | support
assignment | On suppor | Total
-----------+-----------+----------
Untreated | 2,538 | 2,538
Treated | 780 | 780
-----------+-----------+----------
Total | 3,318 | 3,318

when we run pstest following this, we get a nice statistically similar matched group:
pstest agecat1 agecat2 agecat3 educ1 educ2 educ3 newprov1 newprov2 newprov3 newprov4 child6 wealthcat irs_district urban3, treated(_treated) support(_support)

However, when I run pscore on the same model, the balancing property is not met, in 37 instances.
pscore media_ipc agecat2 agecat3 educ2 educ3 newprov2 newprov3 newprov4 child6 wealthcat irs_district urban3 if anynet==1, pscore( itnlastniteps1) blockid( bloklast) comsup

I am hoping for some advice on which of these approaches is appropriate or if there is something I am missing. The ATT is wildly different from the unmatched result, which leads us to think there is something very different about the exposed group, possibly hidden by the common support (?).

My understanding is that we can estimate the ATT after the pscore command by doing the following:
attnd itnlastnite media_ipc if anynet==1, pscore( itnlastniteps1) comsup

which gives a smaller estimated ATT - but still with all the blocks not being balanced?

ATT estimation with Nearest Neighbor Matching method
(random draw version)
Analytical standard errors

---------------------------------------------------------
n. treat. n. contr. ATT Std. Err. t
---------------------------------------------------------

780 1443 0.331 0.058 5.676

---------------------------------------------------------
Note: the numbers of treated and controls refer to actual
nearest neighbour matches

many thanks,
Hannah

[Melissa Garrido]

Hi Hannah,
You need to run the two (user-written) commands in conjunction with each other. You should run -pstest- first in order to acquire a propensity score that balances covariates across treatment and comparison groups. The -pstest- command will help you balance covariates across the sample as a whole as well as within strata of your propensity score. Following that, use -psmatch2- (with the model you develop with -pstest-) to create your matched sample.

Examples of the code are available in the following online appendix: http://onlinelibrary.wiley.com/doi/1...12182/suppinfo (Garrido et al. HSR 2014).

Hope this helps,
Melissa

[Phyu Zin]

Hi Melissa

How to interpret the ATT t-stat from "psmatch2" output.? With the coefficients obtained from ATT, how to judge is it significant or not?
Thank you

[Amare Terefe Gashaye]

Hello Melissa, would you give me brief explanation of the dotted (.) values of the post-PSM test results in the table: Thank you!

[Melissa Garrido]

The dot indicates the value is missing - do you have any variables with no variability in the control group?

[Amare Terefe Gashaye]

Thank you for your response, Melissa. In fact, the variables have different values, may be the variability within a variable might be quite small.