Presenting aggregated psm quality metrics across groups?

[Johannes de Ruig]

Hi all,

I have run a PSM matching procedure to match religious and secular nonprofits. In order to find close matches, I have written a loop that performs the matching procedure, including calculating propensity scores, within each industry-year group.
Now, when I look at matching quality diagnostics (e.g., "pstest"), at first glance it seems as if the matching did not do anything useful at all.
However, this might have been expected, as I matched and calculated propensity scores within industries. Indeed, if I look at the matching quality within a given industry (e.g., "pstest if industry==x"), the matching quality seems to be great.

Now I am struggling to find a way to present the matching quality in a concise manner. Given I have 25 industries over 14 years, denoting the quality metrics for all of them is simply not an option.
I thought about aggregating some of these metrics, but I am not sure whether (1) this would be a technically correct way to deal with these metrics and (2) how to do it, which are my questions.

The metrics I mean are basically the key metrics from "pstest", e.g., %bias/SMD, %reduction |bias|, variance ratio and pseudo R2.
Note that I work in Stata 18.

A (very stupid) example of what I did and mean:

webuse nlswork, clear

egen g = group(race year)

capture drop match_weight
gen match_weight = .

levelsof g, local(gr)
qui foreach j of local gr {

    capture drop _pscore _weight _treated _support _id _n1 _pdif

    capture noisily psmatch2 collgrad ///
        hours ln_wage union ///
        if g==`j', ///
        logit ///
        neighbor(1) ///
        caliper(0.05) ///
        ties

    if _rc == 0 {
        replace match_weight = _weight if g==`j'
    }
    
}

capture drop matched
gen byte matched = (match_weight != .)

This is the matching procedure I did, but where I did it in industry-year groups, here in race-year groups (note that I have some additional, here irrelevant, safeguards surrounding empty/small groups)
Then if you look at pstest output overall:

 pstest hours ln_wage union ///
    ,treated(collgrad) ///
    mweight(match_weight) ///
    support(matched) ///
    both

the matching quality should not be that good. In this case, it is not even that bad, but in my larger dataset with way more groups, there were some instances of bias increasing after the matching procedure.
Until I look at the matching quality within a given industry (or here, race):

 pstest hours ln_wage union ///
    if race==1 ///
    ,treated(collgrad) ///
    mweight(match_weight) ///
    support(matched) ///
    both

I would love to take e.g., the %bias from this one, and from race==2 and race==3, and aggregate them somehow.
Also note that the final regression code would now be something to the likes of

 reg msp collgrad hours ln_wage union i.race i.year [aweight=match_weight] if matched==1, cluster(idcode)

where msp is the outcome variable and collgrad the main variable of interest. (As I said, this is quite a stupid example, but I hope it serves good enough to explain my procedure.)

Kind regards, and thanks in advance,
Johannes

[Felix Bittmann]

Just as a general note, using 1:1 matching is rather outdated as it often performs poorly. Is there any special reason to stick to this? Other ados, like kmatch, provide simple ways to directly compare the matched groups, so it becomes very obvious whether the matching is successful or not. Furthermore, some alternatives, such as entropy balancing, usually outperform any PSM in general.