-lrtest- after -mixed- with -vce(robust)- option

[Umito Takao]

Dear statalisters,

Hi, I have two multilevel model, one is normal model and the other is allowed coefficient to vary at higher level. For both model, I use -vce(robust)- option.
I'd like to test if the latter model fits better than the former, but using -lrtest- after -vce(robust)- would violate one of the assumptions of -lrtest- and thus problematic.
I have looked into that problem and some people suggest -test- rather than -lrtest-, but I think it is only valid for nested model (e.g. it is only valid after inclusion of an additional independent variable)
Is there any way to test the model fit after my case?

Kind regards,
Umito

[Joseph Coveney]

I have two multilevel model, one is normal model and the other is allowed coefficient to vary at higher level. For both model, I use -vce(robust)- option.
I'd like to test if the latter model fits better than the former

You could consider something like the following. Begin at the "Begin here" comment; the top part of the output shows just setting up the problem with a multilevel model analogous to what you describe that is fitted to a fictitious dataset for illustration.

.ÿ
.ÿversionÿ16.0

.ÿ
.ÿclearÿ*

.ÿ
.ÿsetÿseedÿ`=strreverse("1505367")'

.ÿ
.ÿquietlyÿsetÿobsÿ250

.ÿgenerateÿintÿpidÿ=ÿ_n

.ÿquietlyÿdrawnormÿpid_interceptÿpid_slope,ÿdoubleÿcorr(1ÿ-0.25ÿ\ÿ-0.25ÿ1)

.ÿ
.ÿquietlyÿexpandÿ10

.ÿbysortÿpid:ÿgenerateÿbyteÿtimÿ=ÿ_n

.ÿ
.ÿgenerateÿdoubleÿoutÿ=ÿ///
>ÿÿÿÿÿÿÿÿÿpid_interceptÿ+ÿ///
>ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ(timÿ-ÿ5.5)ÿ/ÿ9ÿ*ÿpid_slopeÿ+ÿ///
>ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿrnormal()

.ÿ
.ÿmixedÿoutÿ,ÿvce(robust)ÿ||ÿpid:ÿtim,ÿcovariance(unstructured)ÿnolog

Mixed-effectsÿregressionÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿobsÿÿÿÿÿ=ÿÿÿÿÿÿ2,500
Groupÿvariable:ÿpidÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿgroupsÿÿ=ÿÿÿÿÿÿÿÿ250

ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿObsÿperÿgroup:
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿminÿ=ÿÿÿÿÿÿÿÿÿ10
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿavgÿ=ÿÿÿÿÿÿÿ10.0
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿmaxÿ=ÿÿÿÿÿÿÿÿÿ10

ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿWaldÿchi2(0)ÿÿÿÿÿÿ=ÿÿÿÿÿÿÿÿÿÿ.
Logÿpseudolikelihoodÿ=ÿ-3915.2461ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿProbÿ>ÿchi2ÿÿÿÿÿÿÿ=ÿÿÿÿÿÿÿÿÿÿ.

ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ(Std.ÿErr.ÿadjustedÿforÿ250ÿclustersÿinÿpid)
------------------------------------------------------------------------------
ÿÿÿÿÿÿÿÿÿÿÿÿÿ|ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿRobust
ÿÿÿÿÿÿÿÿÿoutÿ|ÿÿÿÿÿÿCoef.ÿÿÿStd.ÿErr.ÿÿÿÿÿÿzÿÿÿÿP>|z|ÿÿÿÿÿ[95%ÿConf.ÿInterval]
-------------+----------------------------------------------------------------
ÿÿÿÿÿÿÿ_consÿ|ÿÿÿ.0212697ÿÿÿÿ.064105ÿÿÿÿÿ0.33ÿÿÿ0.740ÿÿÿÿ-.1043737ÿÿÿÿ.1469131
------------------------------------------------------------------------------

------------------------------------------------------------------------------
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ|ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿRobustÿÿÿÿÿÿÿÿÿÿÿ
ÿÿRandom-effectsÿParametersÿÿ|ÿÿÿEstimateÿÿÿStd.ÿErr.ÿÿÿÿÿ[95%ÿConf.ÿInterval]
-----------------------------+------------------------------------------------
pid:ÿUnstructuredÿÿÿÿÿÿÿÿÿÿÿÿ|
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvar(tim)ÿ|ÿÿÿ.0123036ÿÿÿ.0021884ÿÿÿÿÿÿ.0086823ÿÿÿÿ.0174354
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvar(_cons)ÿ|ÿÿÿ1.643174ÿÿÿ.2133307ÿÿÿÿÿÿ1.274012ÿÿÿÿ2.119307
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿcov(tim,_cons)ÿ|ÿÿ-.0961584ÿÿÿ.0196113ÿÿÿÿÿ-.1345958ÿÿÿÿ-.057721
-----------------------------+------------------------------------------------
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvar(Residual)ÿ|ÿÿÿ.9902753ÿÿÿ.0306616ÿÿÿÿÿÿ.9319668ÿÿÿÿ1.052232
------------------------------------------------------------------------------

.ÿ
.ÿ*
.ÿ*ÿBeginÿhere
.ÿ*
.ÿestatÿsd,ÿpost
------------------------------------------------------------------------------
ÿÿÿÿÿÿÿÿÿÿÿÿÿ|ÿÿÿÿÿÿCoef.ÿÿÿStd.ÿErr.ÿÿÿÿÿÿzÿÿÿÿP>|z|ÿÿÿÿÿ[95%ÿConf.ÿInterval]
-------------+----------------------------------------------------------------
outÿÿÿÿÿÿÿÿÿÿ|
ÿÿÿÿÿÿÿ_consÿ|ÿÿÿ.0212697ÿÿÿÿ.064105ÿÿÿÿÿ0.33ÿÿÿ0.740ÿÿÿÿ-.1043737ÿÿÿÿ.1469131
-------------+----------------------------------------------------------------
pidÿÿÿÿÿÿÿÿÿÿ|
ÿÿÿÿÿÿsd(tim)|ÿÿÿ.1109216ÿÿÿ.0098646ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ.0931786ÿÿÿÿ.1320431
ÿÿÿÿsd(_cons)|ÿÿÿ1.281864ÿÿÿ.0832112ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ1.128721ÿÿÿÿ1.455784
ÿÿÿÿcorr(tim,|
ÿÿÿÿÿÿÿ_cons)|ÿÿ-.6762846ÿÿÿ.0566606ÿÿÿ-11.94ÿÿÿ0.000ÿÿÿÿ-.7873374ÿÿÿ-.5652319
-------------+----------------------------------------------------------------
Residualÿÿÿÿÿ|
ÿÿÿÿÿÿÿÿsd(e)|ÿÿÿ.9951258ÿÿÿ.0154059ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ.9653843ÿÿÿÿ1.025784
------------------------------------------------------------------------------

.ÿtestÿ[pid]sd(tim)ÿ=ÿ0,ÿnotest

ÿ(ÿ1)ÿÿ[pid]sd(tim)ÿ=ÿ0

.ÿtestÿ[pid]corr(tim,_cons)ÿ=ÿ0,ÿaccumulate

ÿ(ÿ1)ÿÿ[pid]sd(tim)ÿ=ÿ0
ÿ(ÿ2)ÿÿ[pid]corr(tim,_cons)ÿ=ÿ0

ÿÿÿÿÿÿÿÿÿÿÿchi2(ÿÿ2)ÿ=ÿÿ179.91
ÿÿÿÿÿÿÿÿÿProbÿ>ÿchi2ÿ=ÿÿÿÿ0.0000

.ÿ
.ÿexit

endÿofÿdo-file


.

[Umito Takao]

Thank you so much for replying.
Would it be possible for you to explain what you did in the last two commands "test [pid]sd(tim)=0, notest" and "test [pid]corr(tim, _cons) = 0, accumulate"?
I would appreciate it if you could do that.

Regards,
Umito Takao

[Joseph Coveney]

It's explained in the help file for the command:

"accumulate allows a hypothesis to be tested jointly with the previously tested hypotheses."

and

"notest suppresses the output. This option is useful when you are interested only in the joint test of several hypotheses, specified in a subsequent call of test, accumulate."

help test

I'm not sure how well behaved Wald tests are on the random effects. The likelihood profiles (and posterior distributions) of variances that are in the neighborhood of zero are highly skew, and so Wald-type tests that rely on a symmetric "standard error of the estimate" might not have the frequentist properties that you'd like them to have. I'm not sure how feasible it is to check on their operating characteristics, because under the null hypothesis (zero variance for the random slope) -mixed- is liable not to converge in a not-insignificant percentage of repetitions in a -simulate- exercise.

If it were me, for deciding between models, I would be more apt to rely on graphical assessment of predictions and on subject matter considerations over some kind of null hypothesis statistical significance test. (Actually, if it were me, I wouldn't have used -vce(robust)- in the first place, and so this testing problem wouldn't have arisen.)