How to compare the performance of different firthlogit model?

[Ruowen Shen]

Hi,
I have a question regarding the comparison of performance between different firthlogit models. Just like using fitstat after running ologit model, I can see multiple r-square indexes, and AIC and BIC. Can anyone let me know how I can have such index information after running firthlogit model, so I can compare between different firthlogit models to picked the best one.

Thank you very much.

[Joseph Coveney]

You might have to calculate the information criteria manually from the output. I don't know about the multitude of R² indexes. I think that there's one that's been in fashion for a while, but I've forgotten its name. You can search the archives, as I recall that someone shows in a post how to compute it after -firthlogit- (or points to some URL where it's shown).

Why not create calibration plots and choose the model that predicts best in the region that you're most interested in? Or, if your interest is more in discrimination, then why not create receiver operating characteristic plots and choose the model with the best discrimination in your region of interest?

[Ruowen Shen]

Hi Joseph,
Thank you very much for the suggestions. Could you give me more clues about calibration plots and receiver operating characteristic plots? I don't have much ideas on these two plots. Thank you very much.

[Joseph Coveney]

You can use the search control at the top of the Stata List webpage to search for calibration plot, and a dozen or so posts are returned. Some of them have references, e.g., to a textbook by Frank Harrell, Jr. and some even have Stata code. For receiver operating characteristic plots, type

help roc

at the command line in Stata.

[Steve Samuels]

Try calibrationbelt, a contributed command. You can get it by typing in Stata:

net describe gr0071, from(http://www.stata-journal.com/software/sj17-4)

For an extended discussion, see:
Assessing the calibration of dichotomous outcome models with the calibration belt by G.Nattino,S.Lemeshow,G.Phillips,S.Finazzi,andG.Ber tolini. Stata Journal, Volume 17 Issue 4: