Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • teffects nnmatch, caliper() does not respect -atet- option

    The title essentially says it all. If you do teffects nnmatch ..., atet caliper(#) osample(toOmit) then the variable "toOmit" will include control observations. Yet, if I am interested in the average treatment effect on the treated, why would I throw away control observations? All I care about is matching my treated observations as well as possible.

    While we're on the topic, it would be nice if it were possible to run caliper(#, force) which will simply restrict the sample to the matches that fit the caliper. Right now, the caliper option is very counterintuitive to use, especially when using mahalanobis distances. Your code will fail if you do the following:

    Code:
    teffects nnmatch, atet caliper(1.5) osample(toOmit)
    teffects nnmatch if toOmit == 0, atet caliper(1.5)
    Why? Because once you omit the observations that do not fit the caliper, your sample covariances change, which means the mahalanobis distances change, which means there (almost always) will be new observations that do not fit the caliper.

    Moreover, you cannot run the above in one go, because the first line will give you an error message, e.g.
    Code:
    26 observations have no nearest-neighbor matches within caliper 1.5; they are identified in the osample() variable
    r(459);
    This means that instead you have to run
    Code:
    capture noisily teffects nnmatch, atet caliper(1.5) osample(toOmit)
    teffects nnmatch if toOmit == 0, atet
    And I don't know how many Stata users are aware of the -capture noisily- trick. All of it just feels very "unStata-ish".
    Last edited by Jesse Wursten; 21 Mar 2018, 07:33.

  • #2
    Jesse,
    I concur with your comments, as it is a multiple step procedure with changing variances (due to changing samples) with each pass. I am having trouble in understanding how to adjust the caliper settings to reduce/minimize sample selection bias. Cochran & Rubin (1973) and Rosenbaum & Rubin (1985) and others, have noted how to interpret this for propensity score models, but I do not think the same applies for Mahalanobis distances. Can you point me in a direction to better understand how to rationally adjust the caliper with Mahalanobis distances in stata to ensure more 'similar' matches between samples. My first thought was to evaluate the variance ratios from "tebalance summary" and adjust the caliper in accordance to the variances of the treatment and control group based on the most problematic variance ratio. But, these ratios are based on the variances of the variables, not the Mahalanobis distances which the caliper affects.
    Thanks
    JJ

    Comment


    • #3
      All I can say is that in the literature, this is often done similarly to outlier removal. I.e. generate your mahalonobis distances and then set your caliper to the 95th or 99th percentile.

      Comment


      • #4
        I have a query into stata about this, awaiting a response. I have calculated the Mahalanobis distances after a run and determined a cutoff for the max distance I want to be considered for a match. But, this does not work as anticipated with the caliper. For example, if I set the caliper to 2.7, which represents about 90% of the distances, nothing will be filtered out of the run. I have to set the caliper to around 0.5 to achieve a 10% trimming of matches. The caliper is not the Mahalabobis distance cutoff as indicated.

        Comment


        • #5
          Not sure what's going on there. Have you tried running the same thing with psmatch2? Despite the name I believe it also does Mahalanobis nn-matching.

          Comment


          • #6
            FYI...here is the response from stata tech support.

            "It looks like you have uncovered a bug related to the -caliper()- option that bites when exact matches are used as well as distance matches. What -teffects nnmatch- does is to first perform the exact matching, followed by distance matching within the exact matches. However, once -teffects nnmatch- has the exact matches, it apparently ignores the -caliper()- option and so does not apply the specified restriction on the covariate distance within the exactly matched sets.

            I have reported this issue to the relevant developers, and we will provide a fix for this as soon as possible in a future update."

            Comment

            Working...
            X