LR test for comparison of multilevel mixed effects models (xtmixed)

[jaewon kim]

Hi I am doing analysis 8-year panel data which has two level; individual(id) and household(hhid).


1. I am wondering how to decide lrtest in xtmixed about random intercept model and random slope model.

Here is my code and result.

================================================== ==

. . xtmixed d econ_d || id: ||hhid: , iter(1)
Performing EM optimization:
Performing gradient-based optimization:
Iteration 0: log likelihood = -129385.05
Iteration 1: log likelihood = -129385.05 (not concave)
convergence not achieved
Computing standard errors:
Mixed-effects ML regression Number of obs = 37424
-----------------------------------------------------------
| No. of Observations per Group
Group Variable | Groups Minimum Average Maximum
----------------+------------------------------------------
id | 4678 8 8.0 8
hhid | 4678 8 8.0 8
-----------------------------------------------------------
Wald chi2(1) = 249.44
Log likelihood = -129385.05 Prob > chi2 = 0.0000
------------------------------------------------------------------------------
d | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
econ_d | 2.154423 .1364098 15.79 0.000 1.887065 2.421782
_cons | 7.511848 .0872245 86.12 0.000 7.340891 7.682805
------------------------------------------------------------------------------
------------------------------------------------------------------------------
Random-effects Parameters | Estimate Std. Err. [95% Conf. Interval]
-----------------------------+------------------------------------------------
id: Identity |
sd(_cons) | 3.780967 67.04078 3.05e-15 4.68e+15
-----------------------------+------------------------------------------------
hhid: Identity |
sd(_cons) | 3.780968 67.04077 3.05e-15 4.68e+15
-----------------------------+------------------------------------------------
sd(Residual) | 6.876201 .026937 6.823608 6.9292
------------------------------------------------------------------------------
LR test vs. linear regression: chi2(2) = 8644.87 Prob > chi2 = 0.0000
Note: LR test is conservative and provided only for reference.
Warning: convergence not achieved; estimates are based on iterated EM
. estimates store rint
. . xtmixed d econ_d || id: ||hhid: econ_d
Performing EM optimization:
Performing gradient-based optimization:
Iteration 0: log likelihood = -129264.81
Iteration 1: log likelihood = -129253.19 (not concave)
Iteration 2: log likelihood = -129253.13 (backed up)
Iteration 3: log likelihood = -129253.09 (not concave)
Iteration 4: log likelihood = -129253.09 (backed up)
Computing standard errors:
Mixed-effects ML regression Number of obs = 37424
-----------------------------------------------------------
| No. of Observations per Group
Group Variable | Groups Minimum Average Maximum
----------------+------------------------------------------
id | 4678 8 8.0 8
hhid | 4678 8 8.0 8
-----------------------------------------------------------
Wald chi2(1) = 140.07
Log likelihood = -129253.09 Prob > chi2 = 0.0000
------------------------------------------------------------------------------
d | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
econ_d | 2.060142 .1740728 11.83 0.000 1.718966 2.401319
_cons | 7.472637 .0850595 87.85 0.000 7.305924 7.639351
------------------------------------------------------------------------------
------------------------------------------------------------------------------
Random-effects Parameters | Estimate Std. Err. [95% Conf. Interval]
-----------------------------+------------------------------------------------
id: Identity |
sd(_cons) | 3.670353 14.29806 .0017734 7596.428
-----------------------------+------------------------------------------------
hhid: Independent |
sd(econ_d) | 4.453652 .1933661 4.090339 4.849236
sd(_cons) | 3.670753 14.2965 .0017765 7584.644
-----------------------------+------------------------------------------------
sd(Residual) | 6.785597 .0271294 6.732632 6.838979
------------------------------------------------------------------------------
LR test vs. linear regression: chi2(3) = 8908.78 Prob > chi2 = 0.0000
Note: LR test is conservative and provided only for reference.

. estimates store rcoef
. lrtest rint rcoef
Likelihood-ratio test LR chi2(1) = 263.92
(Assumption: rint nested in rcoef) Prob > chi2 = 0.0000
Note: The reported degrees of freedom assumes the null hypothesis is not on the boundary of the parameter space. If this is not true, then the reported test is conservative.

================================================== =============================

As seen in red, p-value for the study is <.0005.
Then, in statistical perspective, which model is appropriate, between random intercept model and random slope model?


2. When I choose the appropriate model like above,
should I do with all control variables or with only the variable i am interested in?
In workshop material of Griffith Univ., which I refer, that is done like above.

[Oded Mcdossi]

The lrtest support the random slope model. You can also find large variation in the effect of the ecod_d across households. However, I think the decision to let the slope vary should be motivated by theoretical and content explanations. If you let the slope to vary between the units you should also try to explain this variation by cross level interactions between the econ_d and household level variables. Hope this helps.

[Clyde Schechter]

Both of your models failed to converge. The results are, in both cases, incorrect. Comparing them is pointless.

If you had achieved convergence with your models, the LR test is telling you that the random coefficient model is not equivalent to the random intercept model. If you were to choose your model based only on the LR test (which I do not recommend), you would choose the random intercept model. This is because the random intercept model, in effect, constrains the randomness of the slopes to zero, and the LR test rejects that null hypothesis.

Added: Crossed with Oded's response, which makes the same points and provides additional excellent advice.

[jaewon kim]

Oded Mcdossi Thank you. But how is it going if I prefer Random Int Model in theoretically and for research implication?
If I want to see the X variable have a meaning or not, then can I use Random Int Model regardless of the test result?

[jaewon kim]

Clyde Schechter
Thank yoy for kind answer.

I am not sure "not converge"...
You mean the message "not concave" or "backed up"?
Actually, I eagerly want to know what the meaning is " "not concave" or "backed up".
Those are not clear for me.

And.. someone said, in that situation, one of the solutions is to change my model.
But what if I want not to modify the model in this situation?
Can't I get proper result, statistically?
Its my thesis and have theoratical base, and I have plan to use model repeatedly, its difficult to modify it...
(I am working with same control variables set and several "X", like below.
y= control variables + x1
y= control variables + x2 .....)

I want to see the effect of "X"s, which measure similar characters of subjects.
But there is no consensus to combine "X"s, existing research also did analysis like this.

How can I solve the problem....??

[Oded Mcdossi]

This is a matter of scientific approach rather than pure statistical. Without much detail on your research aims I can just say that I wouldn't let the coefficient vary across the units, I'll constraint its randomness, if I can't explain this variation. This is generally done by using variables at level two unit using cross-level interactions. In my work I'm trying to explain the variance of the coefficients if I let them vary across units. In linear mixed model the change in the variance component part can tells you how much of the variance can be explained after adding cross-level interactions, or in other words how much you can explain the variation of the effect of one of your level-1 explanatory variables due to contextual or level-2 variables.
BTW-1, I'm not sure why you constrained your first model in #1 to converge after first iteration. In any case you should let the program to choose the number of iteration until convergence.
BTW-2, In the newer versions of stata the xtmixed is now just mixed.

[Clyde Schechter]

Note the lines in your output that I have emphasized:

. . xtmixed d econ_d || id: ||hhid: , iter(1)
Performing EM optimization:
Performing gradient-based optimization:
Iteration 0: log likelihood = -129385.05 
Iteration 1: log likelihood = -129385.05 (not concave)
convergence not achieved
Computing standard errors:
Mixed-effects ML regression Number of obs = 37424
-----------------------------------------------------------
| No. of Observations per Group
Group Variable | Groups Minimum Average Maximum
----------------+------------------------------------------
id | 4678 8 8.0 8
hhid | 4678 8 8.0 8
-----------------------------------------------------------
Wald chi2(1) = 249.44
Log likelihood = -129385.05 Prob > chi2 = 0.0000
------------------------------------------------------------------------------
d | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
econ_d | 2.154423 .1364098 15.79 0.000 1.887065 2.421782
_cons | 7.511848 .0872245 86.12 0.000 7.340891 7.682805
------------------------------------------------------------------------------
------------------------------------------------------------------------------
Random-effects Parameters | Estimate Std. Err. [95% Conf. Interval]
-----------------------------+------------------------------------------------
id: Identity |
sd(_cons) | 3.780967 67.04078 3.05e-15 4.68e+15
-----------------------------+------------------------------------------------
hhid: Identity |
sd(_cons) | 3.780968 67.04077 3.05e-15 4.68e+15
-----------------------------+------------------------------------------------
sd(Residual) | 6.876201 .026937 6.823608 6.9292
------------------------------------------------------------------------------
LR test vs. linear regression: chi2(2) = 8644.87 Prob > chi2 = 0.0000
Note: LR test is conservative and provided only for reference.
Warning: convergence not achieved; estimates are based on iterated EM
. estimates store rint
. . xtmixed d econ_d || id: ||hhid: econ_d
Performing EM optimization:
Performing gradient-based optimization:
Iteration 0: log likelihood = -129264.81 
Iteration 1: log likelihood = -129253.19 (not concave)
Iteration 2: log likelihood = -129253.13 (backed up)
Iteration 3: log likelihood = -129253.09 (not concave)
Iteration 4: log likelihood = -129253.09 (backed up)
Computing standard errors:
Mixed-effects ML regression Number of obs = 37424
-----------------------------------------------------------
| No. of Observations per Group
Group Variable | Groups Minimum Average Maximum
----------------+------------------------------------------
id | 4678 8 8.0 8
hhid | 4678 8 8.0 8
-----------------------------------------------------------
Wald chi2(1) = 140.07
Log likelihood = -129253.09 Prob > chi2 = 0.0000
------------------------------------------------------------------------------
d | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
econ_d | 2.060142 .1740728 11.83 0.000 1.718966 2.401319
_cons | 7.472637 .0850595 87.85 0.000 7.305924 7.639351
------------------------------------------------------------------------------
------------------------------------------------------------------------------
Random-effects Parameters | Estimate Std. Err. [95% Conf. Interval]
-----------------------------+------------------------------------------------
id: Identity |
sd(_cons) | 3.670353 14.29806 .0017734 7596.428
-----------------------------+------------------------------------------------
hhid: Independent |
sd(econ_d) | 4.453652 .1933661 4.090339 4.849236
sd(_cons) | 3.670753 14.2965 .0017765 7584.644
-----------------------------+------------------------------------------------
sd(Residual) | 6.785597 .0271294 6.732632 6.838979
------------------------------------------------------------------------------
LR test vs. linear regression: chi2(3) = 8908.78 Prob > chi2 = 0.0000
Note: LR test is conservative and provided only for reference.

In the first case, it appears that convergence failed because you specified the -iter(1)- option (as Oded noted in his resopnse), thereby aborting estimation after one iteration and not giving it a chance to reach convergence (though, of course, I cannot predict whether it will actually converge if you let it run to completion). In the second case, your final estimates show "(backed up)" which does signal that your final results are not valid. The "(not concave)" and "(backed up)" messages are warnings from Stata that it is having difficulty finding the maximizing parameter values for this likelihood. If you don't have these messages in the final iteration, then you are OK, but when they do appear in the final iteration, as here, then the results cannot be accepted as correct. Other than saying that they are Stata's way of telling you that this is a difficult calculation that is not heading to a successful conclusion, it is too complicated to describe what they mean in a post here.

Here is one suggestion for changing your model. The way you have coded it, you are saying that id is nested within hhid. I'll go out on a limb here and guess that id identifies individual people and hhid identifies households. So you have told Stata that each individual is found in several different households. I suspect the reality is the other way around and households contain several individuals. If I'm right about this, you need to reverse the order of your random effects so that Stata will treat individuals as nested in households. That alone may be sufficient to get your models to converge: a model with the order backwards will frequently have this kind of convergence problem.

[jaewon kim]

Oded Mcdossi Thank you for reply.
I think constraining randomness of slope in my case, because I am not interested in doing that and the purpose of research is just to see whether there is an effect or not.(Variation in population doesn't matter in my research).

And the reason I constrained # of iteration is to see the result.
Because of "not concave" and "backed up" problems, it is difficult to get the result.
(I didn't know "not concave"/ "backed up" imply possibility of wrong result statistically, in fact.
And after changing my model as Clyde Schechter suggests, it looks not having those problems anymore.)

Lasty, thank you for letting me know about command in newer version!

[jaewon kim]

Clyde Schechter Thanks for explaing the meanings!
Can you let me know some further texts or materials about the warnings?
I am not native, so it is difficult to realize the meanings and that misunderstanding makes me confused.

And as I said in previous post, the constrainment of iteration is to force to get the result, actually.
(It doesn't look the right way, come to think of it.)

As your suggestion I reverse the order of level variables,
(because your explanation about my level variables is right)
then it looks to work properly.

Thank you again, and I will do it all right again^^

[Clyde Schechter]

Try looking at the -maximize- section of the online manuals (p. 1481 if you are running the most recent version of Stata). It gives some concrete explanation of how these messages arise in the course of Stata's calculations of maximum likelihood estimates and some practical ways of dealing with it. For a more conceptual understanding you really would need to go to a textbook on numerical methods of maximum likelihood estimation and learn the basics there. I'm a long way removed from that and I don't have specific references for you.

[jaewon kim]

Clyde Schechter Oh. What I mean is about "not concave" or "backed up". It is somewhat technical part as I think, so...
About MLE, I also need more references, but I think I can get help from others
Thanks for your kind consideration!