Error r(504) from -teffects- with propensity scoring and a caliper distance

[Mike Lacy]

Greetings,

I am finding a problem, conceivably a bug, in the behavior of the -teffects- command,
It occurs when I try to enforce a caliper distance on matches in a propensity score
analysis.

As a secondary matter, I'm encountering a convenience issue related to excluding
from analyses treatment observations that have no match within a caliper range.


"Problem" first: I get error r(504) when I think I shouldn't

Here is the relevant part of my code:

local caliper = 0.0345
local nomatch = "1093, 2540, 2541, 5440, 5727, 7697" // some observation numbers to exclude
teffects psmatch (collgrad) (treatment i.teenparent siblings meanfmedu faminc*22) ///
if female & !inlist(_n, `nomatch'), ///
caliper(`caliper') control(0) ate nn(5) gen(match)

When I run the preceding, -teffects- halts prior to producing any output and reports:
"estimates post: matrix has missing values r(504)"

This error goes away if I relax the caliper size to a larger value.
0.0345 is not an outrageous caliper; it's about 1/4 of the standard deviation
of the propensity score, a common caliper choice.

Any suggestions of what I might be doing wrong?

I'm sorry, but I haven't yet been able to find a way to reliably reproduce
this error with a publicly available data set.


"Convenience" issue second:

In any attempt at a propensity score analysis, if I allow inclusion of treatment observations
that have no match within the caliper range, -teffects- halts and notes this fact,
"no propensity-score matches for observation 1093 within caliper 0.0345; this is not allowed."

I find the halting both inconvenient and perplexing. It's not uncommon or horrible, to my knowledge,
to have some unmatched treatment subjects in a propensity score or similar analysis. I'd rather that
-teffects- report these and go ahead with the analysis, as -psmatch2- does. My solution to this
behavior currently is to find the problem observations by trial and error, and exclude them, as I've
done above. This is tedious if one has, say, 10 match failures out of 1,000 treatment subjects.
Am I missing some option on -teffects-? If not, does anyone have any idea for a workaround?

Thanks for your advice,

Mike Lacy
Fort Collins CO

[Richard Gates (StataCorp)]

I have been been able to reproduce the -teffects psmatch- error 504 that Mike Lacy has discovered. This is an issue with the robust standard error computation. The robust standard errors require two sets of matches: matches to the opposite group and matches in the same group. It also requires at least 2 matches in each match set in order to compute two variances. Our matching algorithm is not enforcing the minimum number of matches specified in nn(#) or vce(robust,nn(#)). It attempts to achieve # matches, but will settle for #>1 if that is all it can find. It should be enforcing the minumum. In your case an observation has only one match and its contribution to the standard error is a missing values which propegates through the rest of the computations.

Regarding the `convenience issue', I feel your pain. I spent two hours trying to repoduce your error, playing the same mark-out game that you did. However, if -teffects psmatch- marks out unmatched observation the new sample is no longer the sample that generated the propensity scores. We decided to require the user to mark out these observations to emphasize that the propensity score model has changed.

-Rich

[Mike Lacy]

I have been been able to reproduce the -teffects psmatch- error 504 that Mike Lacy has discovered. This is an issue with the robust standard error computation. The robust standard errors require two sets of matches: matches to the opposite group and matches in the same group. It also requires at least 2 matches in each match set in order to compute two variances. Our matching algorithm is not enforcing the minimum number of matches specified in nn(#) or vce(robust,nn(#)). It attempts to achieve # matches, but will settle for #>1 if that is all it can find. It should be enforcing the minumum. In your case an observation has only one match and its contribution to the standard error is a missing values which propegates through the rest of the computations.

Regarding the `convenience issue', I feel your pain. I spent two hours trying to repoduce your error, playing the same mark-out game that you did. However, if -teffects psmatch- marks out unmatched observation the new sample is no longer the sample that generated the propensity scores. We decided to require the user to mark out these observations to emphasize that the propensity score model has changed.

-Rich

Thanks for the straightforward and thoughtful answer.

Regards,

Mike Lacy
Fort Collins CO

[Joseph Thomas]

G'day from Australia by way of Denver, CO,

I am using the teffects psmatch command and would like to impose a caliper of .2 x the std. dev. of the propensity score as per common guidance. The standard deviation of the propensity score of the unrestricted data set is .1111484 , giving me a caliper of ~.02223.

However, when I attempt to impose this caliper using this syntax:

teffects psmatch (y) (treat xvarlist, probit), atet nn(1) caliper(.02223) vce(robust, nn(2)) gen(match) osample(osample)

I get the following error: "

11 observations have no propensity-score matches within caliper .02223; they are identified in the osample() variable r(459)

"

As per your previous thread, it seems this is error is at least partly a function of the calculation of robust standard errors. Dropping the observations identified with osample simply reproduces this error ad infinitum; I assume the dropped observations were matched in previous iterations of the standard error calculation. Obviously trimming the data also changes the population parameter and can't be good practice in principle.

As I am using panel data pertaining to student schooling and drop out rates, I don't believe the i.i.d assumption is likely to be met. Running the estimation without calipers and saving the propensity scores reveals that only 2 of 295 treated observations in the outcome model have match distances outside the aforementioned caliper. What would be an appropriate approach to calculating standard errors in this case? Is calculation of the robust standard errors compromised if using (apparently 11) matched observations outside the recommended caliper?

Many thanks for your help!

Kind regards,

Joseph Thomas
Queensland Australia

Greetings,

I am finding a problem, conceivably a bug, in the behavior of the -teffects- command,
It occurs when I try to enforce a caliper distance on matches in a propensity score
analysis.

As a secondary matter, I'm encountering a convenience issue related to excluding
from analyses treatment observations that have no match within a caliper range.


"Problem" first: I get error r(504) when I think I shouldn't

Here is the relevant part of my code:

local caliper = 0.0345
local nomatch = "1093, 2540, 2541, 5440, 5727, 7697" // some observation numbers to exclude
teffects psmatch (collgrad) (treatment i.teenparent siblings meanfmedu faminc*22) ///
if female & !inlist(_n, `nomatch'), ///
caliper(`caliper') control(0) ate nn(5) gen(match)

When I run the preceding, -teffects- halts prior to producing any output and reports:
"estimates post: matrix has missing values r(504)"

This error goes away if I relax the caliper size to a larger value.
0.0345 is not an outrageous caliper; it's about 1/4 of the standard deviation
of the propensity score, a common caliper choice.

[Phurba Tamang]

Hi Richard
I am having a same issue like Mike.
I am getting this error "estimates post: matrix has missing values r(504)" whenever I am using a particular covariate along with other covariates. So, whenever I drop this particular covariate the same command gives me an ATE result.

Command used:
teffects psmatch ( Dependent Var ) (Treat x1 x2 x3..........., probit), gen (match) pstolerance(1e-10)
OR
teffects psmatch (Dependent Var) (Treat x1 x2 x3.........., probit ), pstolerance(1e-10)

I have checked this particular covariate, it is a categorical variable & has no missing values. I would like to use this particular covariate as it is an important control for my analysis.
kindly help me to overcome this error. You have been kind enough to post a solution but apologies for not being able to follow mainly due to new to STATA and econometrics.

Thank you.

Regards
P Tamang