Obtaining propensity score graphs in teffects without sufficient overlap

[Jeff Wooldridge]

I'm using Stata 14. My understanding is that the command

teffects overlap

is intended to be used after one of the teffects commands that uses the estimated propensity score in particular, -- ipw and psmatch. If one starts with a severe lack of overlap, none of the PS options will run -- which means, I think, that the option to use the overlap isn't available. Sometimes, especially for teaching purposes, it's useful to show the PS densities to illustrate just how serious the overlap problem can be. Therefore, Stata should allow one to estimate the propensity score and then obtain the overlap graphs. I can't see how to do this.

The user-written psmatch2 allows plotting of the histograms after PS matching regardless of the lack of overlap. If teffects does not do this, or does not do it easily, it should be changed. Probably I missed something simple and someone can clarify.

[Enrique Pinzon (StataCorp)]

Dear Jeff,

Below I describe what we are doing in teffects and provide some pedagogical alternatives, which I believe is your main motivation. I illustrate this using the inverse-probability weighted estimator.

If there are prefect predictors or close to perfect predictors, arising from the probit or logit equation, we error out. In the latter case we allow the tolerance level to be modified. Currently the tolerance level equates 1.00e-5 to zero. Obviously, under perfect prediction, changing the tolerance will not work.

The error message also suggests that you may use an option called osample(varname) which marks the observations that violate the overlap assumption. It is now the responsibility of the researcher to determine what to do with this particular observations in their sample. We do not tell researchers what to do in this case but provide the information. This may be helpful if they want to drop these observations from their sample, for instance.

As you know, the ATE is not identified when overlap is violated. An estimation command that drop the observations that violate overlap is changing the parameter
to fit the data, which is probably a bad idea. For this reason, we ask the researcher to drop the observations instead of doing so automatically.

Below I construct an example were the overlap assumption is characterized by "severe lack of overlap" and the true average treatment effect is equal to -4.00. In this example you can clearly illustrate that:

1. The estimates of the ATE are far away from the true values
2. The diagnostic graph provided by teffects overlap strongly suggest an overlap violation.

In the example below, before I estimate the treatment effects model I fit a probit and then tabulate predicted probabilities. This allows me to see how close I am to violating the overlap assumption. Then I use teffects ipw and capture the error message . In the next step I change the tolerance level and proceed with estimation and diagnostic testing.

. clear

. set seed 111

. set obs 1000
number of observations (_N) was 0, now 1,000

. // Generating data
.
. generate x1 = rnormal()

. generate x2 = rbeta(2,3)

. generate x3 = rbeta(2,5)

. generate e0 = rchi2(1)-1

. generate e1 = rchi2(1)-1

. generate u  = rnormal()

. generate y0 =  3 + x1 + x2 + e0

. generate y1 = -1 + x1 + x2 + e1

. generate t  = 3  + .8*x1 - x3 + u > 0

. generate y  = t*y1 + (1-t)*y0

. generate te = y1 - y0  //  This is equal to  (-4 + e1 - e0 ) which gives ATE = -4

. // Looking at potential overlap problems for tover

. quietly probit t x1 x3

. predict double pr1
(option pr assumed; Pr(t))

. gen double pr0 = 1 - pr1

. sort t

. by t: sum pr0 pr1

--------------------------------------------------------------------------------------------------
-> t = 0

    Variable |        Obs        Mean    Std. Dev.       Min        Max
-------------+---------------------------------------------------------
         pr0 |         13    .0854753    .0846016   .0002685   .2820809
         pr1 |         13    .9145247    .0846016   .7179191   .9997315

--------------------------------------------------------------------------------------------------
-> t = 1

    Variable |        Obs        Mean    Std. Dev.       Min        Max
-------------+---------------------------------------------------------
         pr0 |        987     .011894    .0242303   9.37e-07     .29999
         pr1 |        987     .988106    .0242303     .70001   .9999991


. // Estimating treatment effects

. cap noi teffects ipw (y) (t x1 x3, probit)
treatment 0 has 9 propensity scores less than 1.00e-05
treatment overlap assumption has been violated; use option osample() to identify the overlap
violators

. cap noi teffects ipw (y) (t x1 x3, probit), pstolerance(1e-7)

Iteration 0:   EE criterion =  6.856e-18  
Iteration 1:   EE criterion =  6.376e-25  

Treatment-effects estimation                    Number of obs     =      1,000
Estimator      : inverse-probability weights
Outcome model  : weighted mean
Treatment model: probit
------------------------------------------------------------------------------
             |               Robust
           y |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
ATE          |
           t |
   (1 vs 0)  |  -7.951854   .3039389   -26.16   0.000    -8.547563   -7.356145
-------------+----------------------------------------------------------------
POmean       |
           t |
          0  |   7.245205   .2992637    24.21   0.000     6.658659    7.831751
------------------------------------------------------------------------------

. teffects overlap

To summarize, we can do something about near violation of overlap by changing tolerance levels. When there are perfect predictors we provide information about the observations that violate overlap but do not proceed with estimation. The burden is on the researcher to decide what to do in this case.

[Ronald Obong]

how can one draw agraph like this in stata

[Nick Cox]

Ronald: That's a repeat of http://www.statalist.org/forums/foru...and-error-bars

Please act on the suggestions made by Marcos in that thread.