xtmixed vs. xtreg - results are not identical

[Mitja Kleczka]

Dear Stata Community,

my boss has asked me to check a dataset for a multilevel nature.
The data is nested in 2 categories, but he suspects the level 2 effect to be very small and wants to ignore the grouping effect by only conducting a single-level analysis. My task is to check whether this is alright, or if it is necessary to choose a more complicated multi-level approach.
I have done this a couple of times before and I usually begin by checking on the ICC using one of those two regressions:

xtmixed (Dependent Variable) || (Grouping Variable):, mle
xtreg (Dependent Variable), mle i(Grouping Variable)

Now, here comes my problem.
In college I have learned that those two regressions should yield the same results. I also did that kind of investigation on other datasets in the past, and in fact always got the same coefficients, standard deviations and sigmas. But not this time.
For the present dataset, xtmixed and xtreg deliver very different results. The sizes of the coefficients and the standard deviations significantly vary between the models. This goes even more for the ICC: while the xtmixed command delivers comprehensible sizes for sigma_u and sigma_e, leading to an ICC of about 0.1-0.15 for each dependent variable, the xtreg always delivers sigma_u=0, hence leading to an ICC of 0. Hence, according to xtmixed, there is evidence for a level 2 effect, while according to xtreg, it seems that there is nothing like that at all.

How can it be that the results from those two models are so far off in this case? And how do I know which result I can trust - are there any tests I could run? In the past, both of them have always led to the same results, and I can't explain those big differences which I got now.

Thanks to you in advance - any advice will be appreciated!

Best,
Mitja Kleczka

[Jeph Herrin]

Which version of Stata are you using? I think it is not Stata 13.1.

[Richard Williams]

Please provide code and output, as well as the version of Stata used. I can't replicate, and you say you haven't had problems in the past. That implies that there may be a problem with your syntax, or maybe there is something weird about your data. A replicable example, if possible, would also be good.

xtmixed has been renamed mixed in Stata 13. xtmixed still works. I notice that mixed is providing variances rather than standard deviations, but other than that I am not noticing any obvious differences in the output for the simple examples I ran.

[Alfonso Sánchez-Peñalver]

Mitja,

why are you specifying i(Grouping Variable) in your xtreg command? With xtset Grouping Variable, you don't need to specify the i() option. In any case, I agree with Richard, provide the code so that we can analyze what you're doing, it's impossible to know otherwise.

[Jeph Herrin]

The i() option was why I assumed that this was an older version of Stata - in version 13.x, -xtreg- will not accept the i() option.

[M. Cleves]

Mitja, you must have something wrong in your syntax. Here is an example using Stata's nlswork dataset
. webuse nlswork
. xtset idcode
. xtreg ln_w , mle

Random-effects ML regression Number of obs = 28534
Group variable: idcode Number of groups = 4711

Random effects u_i ~ Gaussian Obs per group: min = 1
avg = 6.1
max = 15

Wald chi2(0) = 0.00
Log likelihood = -12868.942 Prob > chi2 = .

------------------------------------------------------------------------------
ln_wage | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
_cons | 1.656629 .0059588 278.01 0.000 1.64495 1.668308
-------------+----------------------------------------------------------------
/sigma_u | .3743305 .0046763 .3652764 .383609
/sigma_e | .3210726 .0014746 .3181953 .3239759
rho | .5761391 .0066717 .5630233 .5891711
------------------------------------------------------------------------------
Likelihood-ratio test of sigma_u=0: chibar2(01)= 1.3e+04 Prob>=chibar2 = 0.000

. mixed ln_w ||idcode:, mle nolog

Mixed-effects ML regression Number of obs = 28534
Group variable: idcode Number of groups = 4711

Obs per group: min = 1
avg = 6.1
max = 15


Wald chi2(0) = .
Log likelihood = -12868.942 Prob > chi2 = .

------------------------------------------------------------------------------
ln_wage | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
_cons | 1.656629 .0059578 278.06 0.000 1.644952 1.668306
------------------------------------------------------------------------------

------------------------------------------------------------------------------
Random-effects Parameters | Estimate Std. Err. [95% Conf. Interval]
-----------------------------+------------------------------------------------
idcode: Identity |
var(_cons) | .1401233 .003501 .1334268 .1471559
-----------------------------+------------------------------------------------
var(Residual) | .1030876 .0009469 .1012482 .1049604
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) = 13123.82 Prob >= chibar2 = 0.0000


As you can see, Stata yields the same results.

Regards,
Mario

[Richard Williams]

The i() option was why I assumed that this was an older version of Stata - in version 13.x, -xtreg- will not accept the i() option.

Yes it will.

webuse nlswork, clear
xtreg tenure, mle i(idcode)
xtreg tenure, mle

There is no need to use it if you have xtset the data, but it doesn't seem to hurt. But yes, the same thought crossed my mind: maybe the original poster is using an old version of Stata.

One other piece of advice: Whenever you have a weird problem, it is good to make sure your Stata is up to date, e.g. -update all-. If there really was a problem, maybe it has been fixed. There is also the possibility of a corrupted installation. But once we see the code and output, the cause of the problem may become obviousof

[Jeph Herrin]

Yes it will.

Interesting, I tried it before I posted the above because I had this problem recently - but now I realize I in both instances tried -id()-, not -i().

[Richard Williams]

The use of i() works but does not appear to be documented. It looks like it was last documented in Stata 9.2. According to -help whatsnew- xtset was added to Stata 9 on May 17, 2006.

My theory, by the way, is that Mitja accidentally specified different panel id codes in his commands. But we will see.

[Martijn Hogerbrugge]

Dear all,

As this topic is very close to my question, I thought it would be best to post it in here instead of making a new topic.
While reading the 2nd edition of the Rabe-Hesketh & Skrondal book on Multilevel and Longitudinal modeling, I found a strange inconsistency between the SEs of a simple model that is specified using either xtmixed or xtreg. I'm specifically referring to page 77 of the book.

The stata code:
use http://www.stata-press.com/data/mlmus2/pefr, clear
reshape long wp wm, i(id) j(occasion)
gen occ2 = occasion==2
xtmixed wm occ2 || id:, mle
xtreg wm occ2, i(id) mle

xtmixed returns a coefficient for occ2 of 2.882353 with a SE of 6.793485 (similar to what is printed in the book)
xtreg returns a coefficient for occ2 of 2.882353 with a SE of 14.84805

Thus, while the coefficients are the same, the SEs are clearly different.
When using a GMM/GLS estimation instead of the ML estimation for xtreg, the SE (7.002564) is closer to the xtmixed estimated SE.

Likewise, the intra-class correlation in both models is also quite different:
for the xtmixed estimation: 107.0561^2 / (19.80624^2 + 107.0561^2) = .9669048
for the xtreg estimation: 103.5383^2 / (43.28914^2 + 103.5383^2) = .8512045

Can somebody please inform me and others the reasons for these inconsistencies?
Is it merely a result of different estimation procedures? If so, which one should I rely on?

Best,
~Martijn

(As a side note / interestingly: xtreg using the re option [=GLS estimation] yields estimates that are closer to the xtmixed estimates:
xtreg wm occ2, i(id) re
returns a coefficient for occ2 of 2.882353 with a SE of 7.002564)

[Roman Mostazir]

Hi Martijn (#10), I think you should have started a new thread. However, this is interesting. I have no clue. Clearly, the standard errors are different. Hope someone can spot the reason. I am putting both the results for anyone may be interested to diagnose:

use http://www.stata-press.com/data/mlmus2/pefr.dta,clear

reshape long wp wm, i(id) j(occasion)
gen occ2 = ==2

/*xtmixed*/

xtmixed wm occ2 || id:, mle

------------------------------------------------------------------------------
          wm |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        occ2 |   2.882353   6.793483     0.42   0.671    -10.43263    16.19734
       _cons |   452.4706   26.40555    17.14   0.000     400.7167    504.2245
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
id: Identity                 |
                   sd(_cons) |   107.0561   18.67684       76.0523    150.6991
-----------------------------+------------------------------------------------
                sd(Residual) |   19.80624   3.396741      14.15214    27.71927
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) =    46.44 Prob >= chibar2 = 0.0000


/*xtreg*/

xtset id
xtreg wm occ2, mle

------------------------------------------------------------------------------
          wm |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        occ2 |   2.882353   14.84805     0.19   0.846    -26.21929      31.984
       _cons |   452.4706   27.21821    16.62   0.000     399.1239    505.8173
-------------+----------------------------------------------------------------
    /sigma_u |   103.5383   19.30866                       71.8393    149.2244
    /sigma_e |   43.28914          .                             .           .
         rho |   .8512045          .                             .           .
------------------------------------------------------------------------------
Likelihood-ratio test of sigma_u=0: chibar2(01)=   33.30 Prob>=chibar2 = 0.000

[Marcos Almeida]

I gahter it is something related to a specific characterist of THIS dataset, I mean, the "mlmus2", and not "hidden" theoretical differences between models.

Apart from the estimations with the nlswork dataset (already displayed by M. Cleves), I checked out with other datasets (for example, with "gymdata" below), and the results are quite consistent:

. use http://www.stata-press.com/data/r13/nlswork.dta
(National Longitudinal Survey.  Young Women 14-26 years of age in 1968)

. clear

. use http://www.stata-press.com/data/r13/gymdata.dta

. xtset id
       panel variable:  id (unbalanced)

. xtreg wt month, mle

Fitting constant-only model:
Iteration 0:   log likelihood = -1228.5288
Iteration 1:   log likelihood = -1133.7106
Iteration 2:   log likelihood = -1099.9118
Iteration 3:   log likelihood = -1092.2464
Iteration 4:   log likelihood = -1091.5765
Iteration 5:   log likelihood = -1091.5685

Fitting full model:
Iteration 0:   log likelihood = -868.94938
Iteration 1:   log likelihood = -844.76498
Iteration 2:   log likelihood = -843.59543
Iteration 3:   log likelihood = -843.54137
Iteration 4:   log likelihood = -843.54131

Random-effects ML regression                    Number of obs      =       296
Group variable: id                              Number of groups   =        25

Random effects u_i ~ Gaussian                   Obs per group: min =        11
                                                               avg =      11.8
                                                               max =        12

                                                LR chi2(1)         =    496.05
Log likelihood  = -843.54131                    Prob > chi2        =    0.0000

------------------------------------------------------------------------------
          wt |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       month |  -1.945327    .051635   -37.67   0.000     -2.04653   -1.844125
       _cons |   174.9252   6.999704    24.99   0.000      161.206    188.6444
-------------+----------------------------------------------------------------
    /sigma_u |   34.94724   4.945511                      26.48234     46.1179
    /sigma_e |   3.067955   .1317798                      2.820245    3.337423
         rho |   .9923522   .0022449                      .9866701    .9957825
------------------------------------------------------------------------------
Likelihood-ratio test of sigma_u=0: chibar2(01)= 1258.72 Prob>=chibar2 = 0.000

. mixed wt month || id:, mle

Performing EM optimization: 

Performing gradient-based optimization: 

Iteration 0:   log likelihood = -843.54131  
Iteration 1:   log likelihood = -843.54131  

Computing standard errors:

Mixed-effects ML regression                     Number of obs      =       296
Group variable: id                              Number of groups   =        25

                                                Obs per group: min =        11
                                                               avg =      11.8
                                                               max =        12


                                                Wald chi2(1)       =   1419.37
Log likelihood = -843.54131                     Prob > chi2        =    0.0000

------------------------------------------------------------------------------
          wt |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       month |  -1.945327   .0516351   -37.67   0.000     -2.04653   -1.844125
       _cons |   174.9252   6.999706    24.99   0.000      161.206    188.6444
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
id: Identity                 |
                  var(_cons) |    1221.31   345.6641      701.3145    2126.862
-----------------------------+------------------------------------------------
               var(Residual) |   9.412349   .8085901       7.95378    11.13839
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) =  1258.72 Prob >= chibar2 = 0.0000

Best,


Marcos

[Marcos Almeida]

Using the same dataset (mlmus2), and checking a few models (with one continous independent variable = wm; only the dependent variables.; and adding the binary occ2 plus wm):

. use http://www.stata-press.com/data/mlmus2/pefr, clear

. reshape long wp wm, i(id) j(occasion)
(note: j = 1 2)

Data                               wide   ->   long
-----------------------------------------------------------------------------
Number of obs.                       17   ->      34
Number of variables                   5   ->       4
j variable (2 values)                     ->   occasion
xij variables:
                                wp1 wp2   ->   wp
                                wm1 wm2   ->   wm
-----------------------------------------------------------------------------

. gen occ2 = occasion==2

. xtset id
       panel variable:  id (balanced)

. xtreg wm wp, mle

Fitting constant-only model:
Iteration 0:   log likelihood = -262.41263
Iteration 1:   log likelihood = -213.88374
Iteration 2:   log likelihood = -196.32088
Iteration 3:   log likelihood = -185.45064
Iteration 4:   log likelihood = -184.66283
Iteration 5:   log likelihood = -184.57991
Iteration 6:   log likelihood = -184.57839
Iteration 7:   log likelihood = -184.57839

Fitting full model:
Iteration 0:   log likelihood = -166.58137
Iteration 1:   log likelihood = -166.37488
Iteration 2:   log likelihood = -166.33176
Iteration 3:   log likelihood = -166.33122

Random-effects ML regression                    Number of obs      =        34
Group variable: id                              Number of groups   =        17

Random effects u_i ~ Gaussian                   Obs per group: min =         2
                                                               avg =       2.0
                                                               max =         2

                                                LR chi2(1)         =     36.49
Log likelihood  = -166.33122                    Prob > chi2        =    0.0000

------------------------------------------------------------------------------
          wm |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
          wp |   .8830986   .0654239    13.50   0.000     .7548701    1.011327
       _cons |   58.38749   30.22932     1.93   0.053    -.8608982    117.6359
-------------+----------------------------------------------------------------
    /sigma_u |   25.50371   6.693394                      15.24781     42.6579
    /sigma_e |   23.99221    4.15157                      17.09149    33.67911
         rho |   .5305095   .1774381                      .2125098    .8291583
------------------------------------------------------------------------------
Likelihood-ratio test of sigma_u=0: chibar2(01)=    5.50 Prob>=chibar2 = 0.009

. xtmixed wm wp || id:, mle

Performing EM optimization: 

Performing gradient-based optimization: 

Iteration 0:   log likelihood = -166.33123  
Iteration 1:   log likelihood = -166.33122  

Computing standard errors:

Mixed-effects ML regression                     Number of obs      =        34
Group variable: id                              Number of groups   =        17

                                                Obs per group: min =         2
                                                               avg =       2.0
                                                               max =         2


                                                Wald chi2(1)       =    188.92
Log likelihood = -166.33122                     Prob > chi2        =    0.0000

------------------------------------------------------------------------------
          wm |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
          wp |   .8830986   .0642493    13.74   0.000     .7571724    1.009025
       _cons |   58.38747   29.71962     1.96   0.049     .1380813    116.6369
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
id: Identity                 |
                   sd(_cons) |   25.50369   6.693703      15.24743    42.65889
-----------------------------+------------------------------------------------
                sd(Residual) |   23.99222   4.152399      17.09034     33.6814
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) =     5.50 Prob >= chibar2 = 0.0095

. xtreg wm , mle
Iteration 0:   log likelihood = -187.89003
Iteration 1:   log likelihood = -184.95979
Iteration 2:   log likelihood = -184.76189
Iteration 3:   log likelihood =  -184.5855
Iteration 4:   log likelihood =  -184.5784
Iteration 5:   log likelihood = -184.57839

Random-effects ML regression                    Number of obs      =        34
Group variable: id                              Number of groups   =        17

Random effects u_i ~ Gaussian                   Obs per group: min =         2
                                                               avg =       2.0
                                                               max =         2

                                                Wald chi2(0)       =      0.00
Log likelihood  = -184.57839                    Prob > chi2        =         .

------------------------------------------------------------------------------
          wm |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       _cons |   453.9118   26.18616    17.33   0.000     402.5878    505.2357
-------------+----------------------------------------------------------------
    /sigma_u |   107.0464   18.67858                       76.0406    150.6949
    /sigma_e |   19.91083   3.414659                       14.2269     27.8656
         rho |   .9665602   .0159494                      .9210943    .9878545
------------------------------------------------------------------------------
Likelihood-ratio test of sigma_u=0: chibar2(01)=   46.27 Prob>=chibar2 = 0.000

. xtmixed wm  || id:, mle

Performing EM optimization: 

Performing gradient-based optimization: 

Iteration 0:   log likelihood = -184.57839  
Iteration 1:   log likelihood = -184.57839  

Computing standard errors:

Mixed-effects ML regression                     Number of obs      =        34
Group variable: id                              Number of groups   =        17

                                                Obs per group: min =         2
                                                               avg =       2.0
                                                               max =         2


                                                Wald chi2(0)       =         .
Log likelihood = -184.57839                     Prob > chi2        =         .

------------------------------------------------------------------------------
          wm |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       _cons |   453.9118   26.18617    17.33   0.000     402.5878    505.2357
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
id: Identity                 |
                   sd(_cons) |   107.0464   18.67858      76.04062     150.695
-----------------------------+------------------------------------------------
                sd(Residual) |   19.91083   3.414678      14.22687    27.86564
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) =    46.27 Prob >= chibar2 = 0.0000

. xtreg wm wp occ2, mle

Fitting constant-only model:
Iteration 0:   log likelihood = -262.71894
Iteration 1:   log likelihood = -212.95258
Iteration 2:   log likelihood = -196.00974
Iteration 3:   log likelihood = -195.26541
Iteration 4:   log likelihood = -195.26454
Iteration 5:   log likelihood = -195.26454

Fitting full model:
Iteration 0:   log likelihood = -166.43675
Iteration 1:   log likelihood = -166.30547
Iteration 2:   log likelihood = -165.95803
Iteration 3:   log likelihood = -165.93448
Iteration 4:   log likelihood = -165.93431

Random-effects ML regression                    Number of obs      =        34
Group variable: id                              Number of groups   =        17

Random effects u_i ~ Gaussian                   Obs per group: min =         2
                                                               avg =       2.0
                                                               max =         2

                                                LR chi2(2)         =     58.66
Log likelihood  = -165.93431                    Prob > chi2        =    0.0000

------------------------------------------------------------------------------
          wm |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
          wp |   .8842291   .0653032    13.54   0.000     .7562371    1.012221
        occ2 |   7.251485   8.048785     0.90   0.368    -8.523843    23.02681
       _cons |   54.25543   30.59798     1.77   0.076    -5.715507    114.2264
-------------+----------------------------------------------------------------
    /sigma_u |   25.74131    6.59273                      15.58208    42.52418
    /sigma_e |   23.44777   4.054678                      16.70738    32.90747
         rho |    .546526   .1729227                      .2301094    .8345405
------------------------------------------------------------------------------
Likelihood-ratio test of sigma_u=0: chibar2(01)=    5.92 Prob>=chibar2 = 0.007

. xtmixed wm wp occ2 || id:, mle

Performing EM optimization: 

Performing gradient-based optimization: 

Iteration 0:   log likelihood = -165.93431  
Iteration 1:   log likelihood = -165.93431  

Computing standard errors:

Mixed-effects ML regression                     Number of obs      =        34
Group variable: id                              Number of groups   =        17

                                                Obs per group: min =         2
                                                               avg =       2.0
                                                               max =         2


                                                Wald chi2(2)       =    189.64
Log likelihood = -165.93431                     Prob > chi2        =    0.0000

------------------------------------------------------------------------------
          wm |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
          wp |   .8842291   .0642316    13.77   0.000     .7583374    1.010121
        occ2 |   7.251485   8.048779     0.90   0.368    -8.523832     23.0268
       _cons |   54.25541   30.13444     1.80   0.072    -4.807007    113.3178
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
id: Identity                 |
                   sd(_cons) |    25.7413   6.592819      15.58196    42.52446
-----------------------------+------------------------------------------------
                sd(Residual) |   23.44777   4.054937      16.70703    32.90819
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) =     5.92 Prob >= chibar2 = 0.0075

Only when there is just the binary occ2 in the model (and this variable only provides the "time" of the measurements, and, by the way, that is somewhat already given by "id" settings) there is a difference between sds.

Would the phenomenon be due to the, say, the "superfluity" of the information regarding occ2?

Best,

Marcos

[Martijn Hogerbrugge]

Dear Marcos and Roman,

Thank you for your answers. It might be that the (large) differences between the SEs from the xtreg and xtmixed commands is indeed only apparent in this very particular dataset.
However, as the original starter of this thread (Mitja) ran into a similar problem with a different (real-life) dataset, I believe something more is "going on".
Actually, I wouldn't have posed this question on the mlmus2 dataset if I hadn't ran into a similar issue in the European Social Survey data. This issue led me to go back to the Rabe-Hesketh & Skrondal book on Multilevel and Longitudinal modelling, which then triggered my question on the different results from xtreg and xtmixed.

@Marcos: in your last post, there is actually a difference in the SEs of the continuous independent variable as well; In your very first model (estimated using xtreg) the SE for wp is .0654239, in the second model (estimated using xtmixed) the SE for wp is .0642493. Likewise, the SEs for the constant are different between the two models as well.

In the mean time, I have emailed the authors of the book, Sophia Rabe-Hesketh and Andres Skrondal about my observation. As soon as I have an answer from them -and their permission to- I will post their reply here.

Best,
~Martijn

[Hugo Soto]

Hi Martin

I am struggling with exactly the same problem with my data. Did you get an answer on this issue?
thank you
Hugo