Testing equality of an ICC between independent samples

[Michael Hofler]

Dear all,

I wonder whether there is an easy, analytical way to compare the intraclass correlation coefficient (ICC) between independent samples (without using bootstrapping).

In two level data (person and time) ICC can be either computed (while allowing for missings at certain time poits) with
xtreg y , i(id) mle

or
mixed y || id:
estat icc

Now, I want to test equality of the ICC across independent samples, e.g. values of sex. For this, I need to compute all parameters jointly. The ICC is a funtion of the between-person and within-person variance. Does anyone know how to compute this jointly - e.g. for males and females?

Thank you and best,
Michael

[Joseph Coveney]

Try something like that below. It illustrates both the joint modeling and the likelihood ratio test of the difference of ICCs between sexes.

Start at the "Begin here" comment; the first part is just creating an illustrative dataset where the sexes have different ICCs.

.ÿversionÿ14.1

.ÿ
.ÿclearÿ*

.ÿsetÿmoreÿoff

.ÿsetÿseedÿ`=date("2015-11-26",ÿ"YMD")'

.ÿquietlyÿsetÿobsÿ250

.ÿ
.ÿgenerateÿbyteÿsexÿ=ÿmod(_n,ÿ2)

.ÿgenerateÿintÿidÿ=ÿ_n

.ÿgenerateÿdoubleÿuÿ=ÿrnormal()

.ÿ
.ÿforvaluesÿiÿ=ÿ1/5ÿ{
ÿÿ2.ÿÿÿÿÿÿÿÿÿgenerateÿdoubleÿy`i'ÿ=ÿuÿ+ÿ(sexÿ+ÿ1)ÿ*ÿrnormal()
ÿÿ3.ÿ}

.ÿquietlyÿreshapeÿlongÿy,ÿi(id)ÿj(rep)

.ÿ
.ÿxtregÿyÿifÿ!sex,ÿi(id)ÿmleÿnolog

Random-effectsÿMLÿregressionÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿobsÿÿÿÿÿ=ÿÿÿÿÿÿÿÿ625
Groupÿvariable:ÿidÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿgroupsÿÿ=ÿÿÿÿÿÿÿÿ125

Randomÿeffectsÿu_iÿ~ÿGaussianÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿObsÿperÿgroup:
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿminÿ=ÿÿÿÿÿÿÿÿÿÿ5
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿavgÿ=ÿÿÿÿÿÿÿÿ5.0
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿmaxÿ=ÿÿÿÿÿÿÿÿÿÿ5

ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿWaldÿchi2(0)ÿÿÿÿÿÿ=ÿÿÿÿÿÿÿ0.00
Logÿlikelihoodÿÿ=ÿ-986.07582ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿProbÿ>ÿchi2ÿÿÿÿÿÿÿ=ÿÿÿÿÿÿÿÿÿÿ.

------------------------------------------------------------------------------
ÿÿÿÿÿÿÿÿÿÿÿyÿ|ÿÿÿÿÿÿCoef.ÿÿÿStd.ÿErr.ÿÿÿÿÿÿzÿÿÿÿP>|z|ÿÿÿÿÿ[95%ÿConf.ÿInterval]
-------------+----------------------------------------------------------------
ÿÿÿÿÿÿÿ_consÿ|ÿÿÿ.1159562ÿÿÿ.0964055ÿÿÿÿÿ1.20ÿÿÿ0.229ÿÿÿÿ-.0729951ÿÿÿÿ.3049076
-------------+----------------------------------------------------------------
ÿÿÿÿ/sigma_uÿ|ÿÿÿ.9849608ÿÿÿ.0748506ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ.8486588ÿÿÿÿ1.143154
ÿÿÿÿ/sigma_eÿ|ÿÿÿ.9787858ÿÿÿ.0309511ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ.9199644ÿÿÿÿ1.041368
ÿÿÿÿÿÿÿÿÿrhoÿ|ÿÿÿ.5031444ÿÿÿ.0423361ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ.4206944ÿÿÿÿ.5854599
------------------------------------------------------------------------------
LRÿtestÿofÿsigma_u=0:ÿchibar2(01)ÿ=ÿ211.88ÿÿÿÿÿÿÿÿÿÿÿÿÿProbÿ>=ÿchibar2ÿ=ÿ0.000

.ÿxtregÿyÿifÿsex,ÿi(id)ÿmleÿnolog

Random-effectsÿMLÿregressionÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿobsÿÿÿÿÿ=ÿÿÿÿÿÿÿÿ625
Groupÿvariable:ÿidÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿgroupsÿÿ=ÿÿÿÿÿÿÿÿ125

Randomÿeffectsÿu_iÿ~ÿGaussianÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿObsÿperÿgroup:
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿminÿ=ÿÿÿÿÿÿÿÿÿÿ5
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿavgÿ=ÿÿÿÿÿÿÿÿ5.0
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿmaxÿ=ÿÿÿÿÿÿÿÿÿÿ5

ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿWaldÿchi2(0)ÿÿÿÿÿÿ=ÿÿÿÿÿÿÿ0.00
Logÿlikelihoodÿÿ=ÿ-1367.6554ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿProbÿ>ÿchi2ÿÿÿÿÿÿÿ=ÿÿÿÿÿÿÿÿÿÿ.

------------------------------------------------------------------------------
ÿÿÿÿÿÿÿÿÿÿÿyÿ|ÿÿÿÿÿÿCoef.ÿÿÿStd.ÿErr.ÿÿÿÿÿÿzÿÿÿÿP>|z|ÿÿÿÿÿ[95%ÿConf.ÿInterval]
-------------+----------------------------------------------------------------
ÿÿÿÿÿÿÿ_consÿ|ÿÿÿ.1063497ÿÿÿ.1126923ÿÿÿÿÿ0.94ÿÿÿ0.345ÿÿÿÿ-.1145233ÿÿÿÿ.3272226
-------------+----------------------------------------------------------------
ÿÿÿÿ/sigma_uÿ|ÿÿÿÿ.878649ÿÿÿ.1179729ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ.6753483ÿÿÿÿÿ1.14315
ÿÿÿÿ/sigma_eÿ|ÿÿÿ2.019182ÿÿÿ.0638516ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ1.897834ÿÿÿÿ2.148288
ÿÿÿÿÿÿÿÿÿrhoÿ|ÿÿÿ.1592091ÿÿÿ.0389257ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ.0947108ÿÿÿÿ.2472608
------------------------------------------------------------------------------
LRÿtestÿofÿsigma_u=0:ÿchibar2(01)ÿ=ÿ25.11ÿÿÿÿÿÿÿÿÿÿÿÿÿÿProbÿ>=ÿchibar2ÿ=ÿ0.000

.ÿ
.ÿ*
.ÿ*ÿBeginÿhere
.ÿ*
.ÿquietlyÿtabulateÿsex,ÿgenerate(sex_)

.ÿmixedÿyÿÿi.sex||ÿid:ÿsex_1ÿsex_2,ÿnoconstantÿresidual(independent,ÿby(sex))ÿnolrtestÿnolog

Mixed-effectsÿMLÿregressionÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿobsÿÿÿÿÿ=ÿÿÿÿÿÿ1,250
Groupÿvariable:ÿidÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿgroupsÿÿ=ÿÿÿÿÿÿÿÿ250

ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿObsÿperÿgroup:
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿminÿ=ÿÿÿÿÿÿÿÿÿÿ5
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿavgÿ=ÿÿÿÿÿÿÿÿ5.0
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿmaxÿ=ÿÿÿÿÿÿÿÿÿÿ5

ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿWaldÿchi2(1)ÿÿÿÿÿÿ=ÿÿÿÿÿÿÿ0.00
Logÿlikelihoodÿ=ÿ-2353.7313ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿProbÿ>ÿchi2ÿÿÿÿÿÿÿ=ÿÿÿÿÿ0.9484

------------------------------------------------------------------------------
ÿÿÿÿÿÿÿÿÿÿÿyÿ|ÿÿÿÿÿÿCoef.ÿÿÿStd.ÿErr.ÿÿÿÿÿÿzÿÿÿÿP>|z|ÿÿÿÿÿ[95%ÿConf.ÿInterval]
-------------+----------------------------------------------------------------
ÿÿÿÿÿÿÿ1.sexÿ|ÿÿ-.0096065ÿÿÿ.1483023ÿÿÿÿ-0.06ÿÿÿ0.948ÿÿÿÿ-.3002737ÿÿÿÿ.2810606
ÿÿÿÿÿÿÿ_consÿ|ÿÿÿ.1159562ÿÿÿ.0964055ÿÿÿÿÿ1.20ÿÿÿ0.229ÿÿÿÿ-.0729951ÿÿÿÿ.3049076
------------------------------------------------------------------------------

------------------------------------------------------------------------------
ÿÿRandom-effectsÿParametersÿÿ|ÿÿÿEstimateÿÿÿStd.ÿErr.ÿÿÿÿÿ[95%ÿConf.ÿInterval]
-----------------------------+------------------------------------------------
id:ÿIndependentÿÿÿÿÿÿÿÿÿÿÿÿÿÿ|
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvar(sex_1)ÿ|ÿÿÿ.9701487ÿÿÿ.1474503ÿÿÿÿÿÿ.7202221ÿÿÿÿ1.306803
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvar(sex_2)ÿ|ÿÿÿ.7720245ÿÿÿ.2073144ÿÿÿÿÿÿ.4560947ÿÿÿÿ1.306794
-----------------------------+------------------------------------------------
Residual:ÿIndependent,ÿÿÿÿÿÿÿ|
ÿÿÿÿbyÿsexÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ|
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ0:ÿvar(e)ÿ|ÿÿÿ.9580215ÿÿÿ.0605906ÿÿÿÿÿÿ.8463316ÿÿÿÿ1.084451
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ1:ÿvar(e)ÿ|ÿÿÿ4.077095ÿÿÿ.2578581ÿÿÿÿÿÿ3.601771ÿÿÿÿ4.615147
------------------------------------------------------------------------------

.ÿestimatesÿstoreÿFull

.ÿ
.ÿdisplayÿinÿsmclÿasÿtextÿ"rhoÿ0ÿ=ÿ"ÿexp(_b[lns1_1_1:_cons])^2ÿ/ÿ///
>ÿÿÿÿÿ(exp(_b[lns1_1_1:_cons])^2ÿ+ÿexp(_b[lnsig_e:_cons])^2)
rhoÿ0ÿ=ÿ.50314474

.ÿdisplayÿinÿsmclÿasÿtextÿ"rhoÿ1ÿ=ÿ"ÿexp(_b[lns1_1_2:_cons])^2ÿ/ÿ///
>ÿÿÿÿÿ(exp(_b[lns1_1_2:_cons])^2ÿ+ÿexp(_b[r_lns2ose:_cons]ÿ+ÿ_b[lnsig_e:_cons])^2)
rhoÿ1ÿ=ÿ.15920921

.ÿ
.ÿquietlyÿmixedÿyÿ||ÿid:ÿ,ÿnolrtestÿnolog

.ÿlocalÿlinesizeÿ`c(linesize)'

.ÿsetÿlinesizeÿ80

.ÿlrtestÿFull

Likelihood-ratioÿtestÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿLRÿchi2(3)ÿÿ=ÿÿÿÿ245.08
(Assumption:ÿ.ÿnestedÿinÿFull)ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿProbÿ>ÿchi2ÿ=ÿÿÿÿ0.0000

Note:ÿTheÿreportedÿdegreesÿofÿfreedomÿassumesÿtheÿnullÿhypothesisÿisÿnotÿon
ÿÿÿÿÿÿtheÿboundaryÿofÿtheÿparameterÿspace.ÿÿIfÿthisÿisÿnotÿtrue,ÿthenÿthe
ÿÿÿÿÿÿreportedÿtestÿisÿconservative.

.ÿsetÿlinesizeÿ`linesize'

.ÿ
.ÿexit

endÿofÿdo-file


.

[Joseph Coveney]

The comparison model ought to have had the same fixed effects term, that is,

quietly mixed y i.sex || id: , nolrtest nolog

In the illustrative example, the accidental omission doesn't have much of effect on the likelihood ratio test--LR chi2(2) = 245.07 instead of what is shown above--but be sure to keep the fixed effects part of the model the same when you go to do the likelihood ratio test with your data.

[Michael Hofler]

Many thanks, Joseph Coveney, for that very very helpful code!

[Joseph Coveney]

By the way, because, as you say, the two samples are independent, you don't really need to devise a complicated mixed model to fit data of both groups at once. Instead, you can just fit the two groups' simple xtreg models separately and sum their log likelihoods. Because the two groups are independent, the overall log likelihood is just the sum of their models' log likelihoods.

So, if you don't mind computing the likelihood-ratio chi-square test statistic manually (which is fairly simple to do), you can get the answer you want without having to spend time thinking about how to compute all the parameters jointly.


.ÿversionÿ14.1

.ÿ
.ÿclearÿ*

.ÿsetÿmoreÿoff

.ÿsetÿseedÿ`=date("2015-11-26",ÿ"YMD")'

.ÿquietlyÿsetÿobsÿ250

.ÿ
.ÿgenerateÿbyteÿsexÿ=ÿmod(_n,ÿ2)

.ÿgenerateÿintÿidÿ=ÿ_n

.ÿgenerateÿdoubleÿuÿ=ÿrnormal()

.ÿ
.ÿforvaluesÿiÿ=ÿ1/5ÿ{
ÿÿ2.ÿÿÿÿÿÿÿÿÿgenerateÿdoubleÿy`i'ÿ=ÿuÿ+ÿ(sexÿ+ÿ1)ÿ*ÿrnormal()
ÿÿ3.ÿ}

.ÿquietlyÿreshapeÿlongÿy,ÿi(id)ÿj(rep)

.ÿ
.ÿ*
.ÿ*ÿBeginÿhere
.ÿ*
.ÿquietlyÿxtregÿyÿifÿ!sex,ÿi(id)ÿmleÿnologÿ//ÿSeeÿpreviousÿpostÿforÿoutput

.ÿtempnameÿllÿdf

.ÿscalarÿdefineÿ`ll'ÿ=ÿe(ll)

.ÿscalarÿdefineÿ`df'ÿ=ÿe(rank)

.ÿ
.ÿquietlyÿxtregÿyÿifÿsex,ÿi(id)ÿmleÿnologÿ//ÿSeeÿpreviousÿpostÿforÿoutput

.ÿscalarÿdefineÿ`ll'ÿ=ÿ`ll'ÿ+ÿe(ll)

.ÿscalarÿdefineÿ`df'ÿ=ÿ`df'ÿ+ÿe(rank)

.ÿ
.ÿquietlyÿxtregÿyÿi.sex,ÿi(id)ÿmleÿnologÿ//ÿOr:ÿquietlyÿmixedÿyÿi.sexÿ||ÿid:ÿ,ÿnolrtestÿnolog

.ÿ
.ÿtempnameÿlr

.ÿscalarÿdefineÿ`lr'ÿ=ÿ2ÿ*ÿ(`ll'ÿ-ÿe(ll))

.ÿscalarÿdefineÿ`df'ÿ=ÿ`df'ÿ-ÿe(rank)

.ÿ
.ÿdisplayÿinÿsmclÿasÿtextÿ"LRÿchi2(`=`df'')ÿ=ÿ"ÿasÿresultÿ%6.2fÿ`lr'
LRÿchi2(2)ÿ=ÿ245.07

.ÿdisplayÿinÿsmclÿasÿtextÿ"Probÿ>ÿchi2ÿ=ÿ"ÿasÿresultÿ%06.4fÿchi2tail(`=`df'',ÿ`=`lr'')
Probÿ>ÿchi2ÿ=ÿ0.0000

.ÿ
.ÿexit

endÿofÿdo-file


.