Hi everyone,
I am trying to better understand the differences in using pscore to check the balancing property after matching, and in using pstest to check the means of covariates before and after matching. The two approaches are being endorsed by two of my colleagues, but pstest gives a nice matched group, while pscore reveals that the balancing property is not met.
We have run the psmatch2 as follows, where "media_ipc" is the treatment (exposure to a health communication message), and the remaining covariates are age, education, region, ses etc, among households that own at least one net, with replacement and the common support:
psmatch2 media_ipc agecat2 agecat3 educ2 educ3 newprov2 newprov3 newprov4 child6 wealthcat irs_district urban3 if anynet==1, outcome(itnlastnite) neighbor(1) common
There are observations with identical propensity score values.
The sort order of the data could affect your results.
Make sure that the sort order is random before calling psmatch2.
----------------------------------------------------------------------------------------
Variable Sample | Treated Controls Difference S.E. T-stat
----------------------------+-----------------------------------------------------------
itnlastnite Unmatched | .858974359 .733648542 .125325817 .017275211 7.25
ATT | .858974359 .325641026 .533333333 .177182168 3.01
----------------------------+-----------------------------------------------------------
Note: S.E. does not take into account that the propensity score is estimated.
| psmatch2:
psmatch2: | Common
Treatment | support
assignment | On suppor | Total
-----------+-----------+----------
Untreated | 2,538 | 2,538
Treated | 780 | 780
-----------+-----------+----------
Total | 3,318 | 3,318
when we run pstest following this, we get a nice statistically similar matched group:
pstest agecat1 agecat2 agecat3 educ1 educ2 educ3 newprov1 newprov2 newprov3 newprov4 child6 wealthcat irs_district urban3, treated(_treated) support(_support)
However, when I run pscore on the same model, the balancing property is not met, in 37 instances.
pscore media_ipc agecat2 agecat3 educ2 educ3 newprov2 newprov3 newprov4 child6 wealthcat irs_district urban3 if anynet==1, pscore( itnlastniteps1) blockid( bloklast) comsup
I am hoping for some advice on which of these approaches is appropriate or if there is something I am missing. The ATT is wildly different from the unmatched result, which leads us to think there is something very different about the exposed group, possibly hidden by the common support (?).
My understanding is that we can estimate the ATT after the pscore command by doing the following:
attnd itnlastnite media_ipc if anynet==1, pscore( itnlastniteps1) comsup
which gives a smaller estimated ATT - but still with all the blocks not being balanced?
ATT estimation with Nearest Neighbor Matching method
(random draw version)
Analytical standard errors
---------------------------------------------------------
n. treat. n. contr. ATT Std. Err. t
---------------------------------------------------------
780 1443 0.331 0.058 5.676
---------------------------------------------------------
Note: the numbers of treated and controls refer to actual
nearest neighbour matches
many thanks,
Hannah
I am trying to better understand the differences in using pscore to check the balancing property after matching, and in using pstest to check the means of covariates before and after matching. The two approaches are being endorsed by two of my colleagues, but pstest gives a nice matched group, while pscore reveals that the balancing property is not met.
We have run the psmatch2 as follows, where "media_ipc" is the treatment (exposure to a health communication message), and the remaining covariates are age, education, region, ses etc, among households that own at least one net, with replacement and the common support:
psmatch2 media_ipc agecat2 agecat3 educ2 educ3 newprov2 newprov3 newprov4 child6 wealthcat irs_district urban3 if anynet==1, outcome(itnlastnite) neighbor(1) common
There are observations with identical propensity score values.
The sort order of the data could affect your results.
Make sure that the sort order is random before calling psmatch2.
----------------------------------------------------------------------------------------
Variable Sample | Treated Controls Difference S.E. T-stat
----------------------------+-----------------------------------------------------------
itnlastnite Unmatched | .858974359 .733648542 .125325817 .017275211 7.25
ATT | .858974359 .325641026 .533333333 .177182168 3.01
----------------------------+-----------------------------------------------------------
Note: S.E. does not take into account that the propensity score is estimated.
| psmatch2:
psmatch2: | Common
Treatment | support
assignment | On suppor | Total
-----------+-----------+----------
Untreated | 2,538 | 2,538
Treated | 780 | 780
-----------+-----------+----------
Total | 3,318 | 3,318
when we run pstest following this, we get a nice statistically similar matched group:
pstest agecat1 agecat2 agecat3 educ1 educ2 educ3 newprov1 newprov2 newprov3 newprov4 child6 wealthcat irs_district urban3, treated(_treated) support(_support)
However, when I run pscore on the same model, the balancing property is not met, in 37 instances.
pscore media_ipc agecat2 agecat3 educ2 educ3 newprov2 newprov3 newprov4 child6 wealthcat irs_district urban3 if anynet==1, pscore( itnlastniteps1) blockid( bloklast) comsup
I am hoping for some advice on which of these approaches is appropriate or if there is something I am missing. The ATT is wildly different from the unmatched result, which leads us to think there is something very different about the exposed group, possibly hidden by the common support (?).
My understanding is that we can estimate the ATT after the pscore command by doing the following:
attnd itnlastnite media_ipc if anynet==1, pscore( itnlastniteps1) comsup
which gives a smaller estimated ATT - but still with all the blocks not being balanced?
ATT estimation with Nearest Neighbor Matching method
(random draw version)
Analytical standard errors
---------------------------------------------------------
n. treat. n. contr. ATT Std. Err. t
---------------------------------------------------------
780 1443 0.331 0.058 5.676
---------------------------------------------------------
Note: the numbers of treated and controls refer to actual
nearest neighbour matches
many thanks,
Hannah
Comment