Generating ICC for binomial distribution

[Rosh Mendis]

Hi everyone,

I have a dataset in which participants are nested within schools, and we seek to evaluate the efficacy of an intervention between participants while controlling for nesting. Three of our outcomes are linear, thus we are using multilevel mixed effects modelling with linear regression, which generates the ICC for us. However, two of our outcomes are binary (anxiety present/absent, depression present/absent), thus we are using multilevel mixed effects modelling with generalised linear modelling and selecting the binomial distribution cloglog option (as the absence of these outcomes is more common than their presence). Though Stata does generate the ICC for these latter models if you hit "estat ICC", my understanding is that this is not an accurate estimate of the ICC and that ICC calculation becomes much more complicated with the binomial distribution. Is there a way on Stata in which we can get an estimate of ICC with relation to the binomial distribution?

Many thanks in advance,
Rosh

[Joseph Coveney]

. . . we are using multilevel mixed effects modelling with generalised linear modelling and selecting the binomial distribution cloglog option . . . Though Stata does generate the ICC for these latter models if you hit "estat ICC", my understanding is that this is not an accurate estimate of the ICC . . .

I'm curious as to how you got that understanding.

In simulation shown below, where the true ICC is set at 0.5, the bias in the estimate returned in estat icc is zilch and its 95% confidence interval is spot on.

.ÿ
.ÿversionÿ18.0

.ÿ
.ÿclearÿ*

.ÿ
.ÿ//ÿseedem
.ÿsetÿseedÿ2058522378

.ÿ
.ÿprogramÿdefineÿsimem,ÿrclass
ÿÿ1.ÿÿÿÿÿversionÿ18.0
ÿÿ2.ÿÿÿÿÿsyntax
ÿÿ3.ÿ
.ÿÿÿÿÿdropÿ_all
ÿÿ4.ÿÿÿÿÿsetÿobsÿ300
ÿÿ5.ÿÿÿÿÿgenerateÿintÿpidÿ=ÿ_n
ÿÿ6.ÿÿÿÿÿgenerateÿdoubleÿpid_uÿ=ÿrnormal(0,ÿsqrt(_pi^2ÿ/ÿ6))ÿ//ÿICCÿ=ÿ0.5
ÿÿ7.ÿ
.ÿÿÿÿÿexpandÿ10
ÿÿ8.ÿÿÿÿÿgenerateÿdoubleÿxbuÿ=ÿcloglog(0.1)ÿ+ÿpid_u
ÿÿ9.ÿÿÿÿÿgenerateÿbyteÿoutÿ=ÿrbinomial(1,ÿinvcloglog(xbu))
ÿ10.ÿ
.ÿÿÿÿÿmecloglogÿoutÿ||ÿpid:
ÿ11.ÿ
.ÿÿÿÿÿestatÿicc
ÿ12.ÿÿÿÿÿtempnameÿicc
ÿ13.ÿÿÿÿÿscalarÿdefineÿ`icc'ÿ=ÿr(icc2)
ÿ14.ÿÿÿÿÿreturnÿscalarÿposÿ=ÿinrange(0.5,ÿr(ci2)[1,ÿ1],ÿr(ci2)[1,ÿ2])
ÿ15.ÿÿÿÿÿreturnÿscalarÿiccÿ=ÿ`icc'
ÿ16.ÿend

.ÿ
.ÿsimulateÿposÿ=ÿr(pos)ÿiccÿ=ÿr(icc),ÿreps(400)ÿnodots:ÿsimem

ÿÿÿÿÿÿCommand:ÿsimem
ÿÿÿÿÿÿÿÿÿÿpos:ÿr(pos)
ÿÿÿÿÿÿÿÿÿÿicc:ÿr(icc)


.ÿ
.ÿforeachÿvarÿofÿvarlistÿiccÿposÿ{
ÿÿ2.ÿÿÿÿÿsummarizeÿ`var',ÿmeanonly
ÿÿ3.ÿÿÿÿÿdisplayÿinÿsmclÿasÿtextÿ_newline(1)ÿ"`var':ÿ"ÿ%04.2fÿr(mean)
ÿÿ4.ÿ}

icc:ÿ0.50

pos:ÿ0.95

.ÿ
.ÿexit

endÿofÿdo-file


.

[Rosh Mendis]

Hi Joseph,

Thanks for your input. This is an example of one of my commands on Stata, for the multilevel model with GLM: meglm Depression_gr || school:, family(binomial) link(cloglog) eform.

At the start of this document, it excludes meglm as an estimation for which estat icc can be used (https://www.stata.com/manuals/meestaticc.pdf).

[Joseph Coveney]

At the start of this document, it excludes meglm as an estimation for which estat icc can be used.

I recommend re-reading that passage:

"estat icc is also for use after estimation with meglm in cases when the fitted model is a linear, logit, probit, ordered logit, ordered probit, or complementary log–log mixed-effects model." (italic emphasis added)