Intraclass correlations with meologit?

[Jeff Tree]

Hi all,

I'm revisiting my analysis for a revise & resubmit article but I'm having trouble replicating my own findings -- specifically reporting the ICC in my tables. Is it still possible to estimate intraclass correlations following a multilevel mixed-effects ordered logistic regression? I'm receiving the following error,

requested action not valid after most recent estimation command

Minimal working example as follows,

clear
use https://www.stata-press.com/data/r17/tvsfpors
meologit thk prethk cc##tv || school:
estat icc

Any thoughts / suggestions would be greatly appreciated!

[Weiwen Ng]

In general, in the help file for whatever command you're using, there's a link to the valid post-estimation commands. The one for meologit should indicate that estat icc is a valid post-estimation command. The help file for estat icc says this as well. I've never used this command, but your syntax appears correct and this should be a valid post-estimation command. I wonder if a previous version of Stata didn't implement an ICC calculation for ordered logit and you're on a previous version. If your version of Stata does support that command, it might be worth an email to Stata support.

That said, I believe you may be able to manually calculate the ICC, especially considering that you have only one random intercept. If you skim the methods and formulas of the estat icc help file, you'll see that the calculation is:

sigma^2 of random intercept / [gamma + sigma^2 of random intercept]

In a linear mixed effect model, gamma is the variance of the error term, which you get from the output table. Here, gamma is pi^2 / 3. The variance (i.e. sigma squared) of the random intercept is the one you need to plug in, and that's also presented in the regression output table.

[Leonardo Guizzetti]

Weiwen's advice is quite good. As for your example, I could run -estat icc- so I could not reproduce your problem on Windows with Stata 17 for Revision 09 Aug 2021.

[Jeff Tree]

Thank you Leonardo Guizzetti Weiwen Ng .

Given that I'm using version 15.1 perhaps the command is no longer available?



With respect to manual calculation, I believe gamma would correspond to the variance of residual but I don't see it listed in my output?

[Leonardo Guizzetti]

It would have been helpful to mention the verison of Stata you are using, as you are asked to do in the FAQ. I don't have access to Stata 15 on this machine, but you can see which commands are supported in your Stata by -estat icc- by reading the output of -help estat icc-. I don't know if -meologit- was introduced by version 15.1, but nevertheless, it is available since at least version 16 (so it's something StataCorp added, not took away).

Second, in that same help file, if you click the link at the top of the page to the full PDF documentation, there's a section called Methods and Formulas, with further details. Please review Weiwen's post in #2 for both the value of gamma and how to manually calculate (a level 2) ICC. You can see the output of -estat icc- and a manual calculation below for the model presented in #1.

. estat icc

Residual intraclass correlation

------------------------------------------------------------------------------
                       Level |        ICC   Std. err.     [95% conf. interval]
-----------------------------+------------------------------------------------
                      school |   .0218563   .0111416      .0079816    .0584298
------------------------------------------------------------------------------

. scalar var_school = [/]_b[var(_cons[school])]
. di var_school / (var_school + (c(pi)^2)/3)
.02185635

[Weiwen Ng]

You have a random effects model with only one level. Here, var(_cons) is the variance of the random intercept.

in a linear mixed model, you will also get the variance of the residual, e_ij. You do not have this in logistic or ordered logit mixed effects models. As I mentioned, you just substitute (pi^2)/3 in its place.

logistic models do not have error variances. Linear models do. In a linear model, you’re saying that y = XB + error. You have to have a random error term here, otherwise it’s not a statistical model, it’s a deterministic one.

in a probability model, you are saying that P(y = 1) is some function of XB. There’s no error term because probability is inherently stochastic.

[Clyde Schechter]

Weiwen Ng
There is an alternate formulation of the two-level logistic model in which there is an error variance:

y* = Xb + u + e
y = 1 if y* > 0, 0 otherwise.

In this model u is the random intercept at the group level. e is the observation level error variance and is assumed to have a standard logistic distribution. (If you instead assume that the distribution of e is standard normal, you have a similar formulation of the probit model.) In fact, the use of (pi^2)/3 as the error variance in the ICC formula comes precisely from the fact that (pi^2)/3 is the variance of the standard logistic distribution. (And in a multilevel probit model you do the analogous ting, with 1 instead.)

[Weiwen Ng]

Weiwen Ng
There is an alternate formulation of the two-level logistic model in which there is an error variance:

y* = Xb + u + e
y = 1 if y* > 0, 0 otherwise.

In this model u is the random intercept at the group level. e is the observation level error variance and is assumed to have a standard logistic distribution. (If you instead assume that the distribution of e is standard normal, you have a similar formulation of the probit model.) In fact, the use of (pi^2)/3 as the error variance in the ICC formula comes precisely from the fact that (pi^2)/3 is the variance of the standard logistic distribution. (And in a multilevel probit model you do the analogous ting, with 1 instead.)

I am less used to this formulation, but that is a good point.