Clarification regarding O/E from melogit model

[Steven Spivack]

I believe I am doing this correctly but I would love confirmation from others more experienced with this type of analysis. I am attempting to calculate a practice-level observed over expected rate for mortality with data from two levels (patient and physician practice).

melogit mortality covariates  || practiceid:
predict yhat
by practiceid: egen yhatsum = sum(yhat)
gen expected = yhatsum/totalnumberpatients
gen observed = totaldeaths/totalnumberpatients
gen SRR = observed/expected

Is there anything I am missing or something else I should be aware of when calculating these rates?

[Clyde Schechter]

Your code looks correct to me, assuming that your variables totalnumberpatients and totaldeaths were correctly calculated in the first place (you didn't show that code) and the unit of observation here is an individual patient, and mortality is coded 0 for didn't die and 1 for did die..

One aside: the -egen-, sum()- function has been renamed -egen, total()- to distinguish it from the -gen, sum()- function which does something very different. While -egen, sum()- is still accepted and runs the -total()- function, it is better not to use -sum()- because of the potential for confusing yourself or somebody else reading the code. I recommend using -egen, total()- going forward.

[Steven Spivack]

Thank you for such a quick reply Clyde. I didn’t want to include code that I knew was correct. I also appreciate the tip about egen, I did not realize it had been updated.

One last question. Do you know if the predict command after running melogit calculates a “true” expected value or does it include a shrinkage factor? I seem to find conflicting evidence on this depending on where I look.

[Clyde Schechter]

The default behavior of -predict- following -melogit-, which, having specified nothing else, is what you are getting, is to calculate invlogit(Xb + u) where Xb is the linear predictor and u is the estimated random intercept at the practice id level. The estimate of u is an empirical Bayes mean.

[Carlos Diaz]

I would like to jump into this forum as I have a related question.

Dear Clyde, so the postestimation command "predict" after melogit , provides a reliability-adjusted prediction? What is the difference of including -fitted- to the command?
Thanks

[Clyde Schechter]

I don't know what you mean by a "reliability adjusted" prediction. And, as far as I know, -predict- does not support a -fitted- option after -melogit-. It does support an -eta- option which gives the value of the linear predictor, including the random effects, but does not apply -invlogit()-. So -eta- is in the log-odds metric.

[Carlos Diaz]

Thank you for the quick reply. the -fitted- option is supported after the -mixed- command. My mistake.

I am trying to estimate the "reliability adjusted- risk-standardized failure to rescue rate" after a surgical procedure. I already fit my multilevel regression mode with patient and hospital-level fixed parameters and hospitalsid as the random parameter.

Previous researchers have used the "reliability adjustment" (https://journals.lww.com/annalsofsur...spital.14.aspx or https://www.rand.org/pubs/technical_reports/TR653.html) to isolate the signal and reduce the noise in a dataset, particularly when sample sizes for a hospital or clinician are small.

My confusion was on the different post-estimation commands and what should I use to estimate reliability given the definition of: variance provider to provider / variance provider to provider + (provider-specific error /N)

Any word of wisdom from you is always very much appreciated.
thanks

[Clyde Schechter]

The notion of reliability as recounted in the Rand Corp. document you linked to is quite familiar to me. But the document does not mention reliability-adjustment, with which I am unfamiliar. Apparently this is detailed in the other document, but that one is paywalled. In any case, I do not know how one could calculate a reliability from a mixed-effects logistic model, because in that model, the "residual" variance is constrained to (pi^2)/3, so the intra-class correlation, although computable, does not have the same interpretation as the intra-class correlation of a normal-normal model or a beta-binomial model. In short, I do not know how to help you with this.

[Carlos Diaz]

thank you Clyde.