Balancing in propensity score matching

[Rob Wood]

Hi All,

I have two groups that I have propensity score matched, but before I analyse the outcomes I want to ensure my groups are adequately balanced on the covariates I used to generate the propensity score. Initially the groups were unbalanced with standardised differences outside the range of (-10%, 10%), therefore I made a few adjustments to the model, adding in interaction terms and higher order terms. I spent a long time on this, and could not achieve the level of balance I wanted across all covariates.

I have read somewhere that at this point it may be sufficient to perform a regression on the matched sample while controlling for all of the covariates, to investigate whether there is a difference in the outcome between my two groups. Is this an acceptable method when balance just cannot be achieved? Or do I just stop my analysis here and admit that balance could not be achieved?

Any advice on this matter would be greatly appreciated.

Thanks,

Rob.

[Melissa Garrido]

Hi Rob,
The answer depends on how much imbalance remains and whether the imbalance is in hypothesized strong or weak confounders. For instance, if you had a single relatively weak confounder with 11% standardized difference across groups, you could probably proceed with your analysis. However, if the covariate is thought to be a strong confounder or has a greater degree of imbalance, you can't be very confident that your propensity score is adequately adjusting for selection bias.
You may also consider weighting your sample of the propensity score and re-evaluting covariate balance; different matching and weighting strategies will have different impacts on covaraite balance.

Hope this helps,
Melissa

[Rob Wood]

Hi Melissa,

Thank you for your quick response. I have tried different matching and weighting strategies and some are better than others but the level of imbalance that remains on some covariates is still quite high.

Would you not advise a regression on the matched sample if covariates remain imbalanced? I've read it in a couple of papers, but not sure how widely used/accepted it is.

Thanks,

Rob.

[Melissa Garrido]

It doesn't sound like you should proceed with any analyses on your matched sample, because you know that it is not matched well. Combining regression adjustment with propensity score matching is meant mainly as a guard against potential (unknown) misspecification of the propensity score model.

[Rob Wood]

Thanks for your advice Melissa.

[Clyde Schechter]

Coming late to this conversation, with apologies. There is one thing that has been left unsaid here. If a covariate is not actually associated with the outcome you are ultimately trying to analyze, then there is no need to worry about whether it is balanced among the comparison groups. So your problem may be smaller than it appears. I would check that before concluding that all hope is lost here. If you have included some covariates that are actually independent of the outcome, then you may be able to find a propensity model that balances all the covariates that actually matter.