How to calculate the explained variance by the addition of a random intercept in mixed

[Patrick Abi Nader]

Hello All,

I have the two models below, which are only different by the addition of the random intercept "|| first_school:" in the 2nd model. I want to know how much variance does the random intercept add to my model. I came across the following UCLA link but it talks about estimating effect size from "meglm." I am not sure how "mixed" and "meglm" differ but I want to keep using "mixed." Does anyone have any suggestion and can I apply the same methods that you recommend to models that use "xtmepoisson"?

Thank you for your time
Best wishes
Patrick

********Model 1
mixed AvPAwk person_ev1 person_circum1 extrinsic_ev1 extrinsic_circum1 cycle cycle2 ///
> if girl==1 || ID: cycle, covariance(uns)

Performing EM optimization: 

Performing gradient-based optimization: 

Iteration 0:   log likelihood = -8666.8432  
Iteration 1:   log likelihood = -8666.8431  

Computing standard errors:

Mixed-effects ML regression                     Number of obs     =      4,943
Group variable: ID                              Number of groups  =        509

                                                Obs per group:
                                                              min =          1
                                                              avg =        9.7
                                                              max =         16

                                                Wald chi2(6)      =     116.80
Log likelihood = -8666.8431                     Prob > chi2       =     0.0000

-----------------------------------------------------------------------------------
           AvPAwk |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
------------------+----------------------------------------------------------------
       person_ev1 |   .0283224   .0288354     0.98   0.326    -.0281939    .0848387
   person_circum1 |  -.1669628   .0283415    -5.89   0.000    -.2225112   -.1114145
    extrinsic_ev1 |   .0417367   .0295919     1.41   0.158    -.0162625    .0997358
extrinsic_circum1 |  -.0437448   .0274655    -1.59   0.111    -.0975763    .0100867
            cycle |   .1340743    .018817     7.13   0.000     .0971937     .170955
           cycle2 |  -.0081732   .0010466    -7.81   0.000    -.0102245    -.006122
            _cons |   4.130385   .0985388    41.92   0.000     3.937252    4.323517
-----------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
ID: Unstructured             |
                  var(cycle) |   .0141806   .0016476      .0112927    .0178071
                  var(_cons) |   2.231859   .1961679      1.878671    2.651444
            cov(cycle,_cons) |  -.1042847   .0152304     -.1341357   -.0744337
-----------------------------+------------------------------------------------
               var(Residual) |   1.418693   .0316948      1.357913    1.482194
------------------------------------------------------------------------------
LR test vs. linear model: chi2(3) = 2175.37               Prob > chi2 = 0.0000

Note: LR test is conservative and provided only for reference.

*************Model 2
. mixed AvPAwk person_ev1 person_circum1 extrinsic_ev1 extrinsic_circum1 cycle cycle2 ///
> if girl==1 || first_school: || ID: cycle, covariance(uns)

Performing EM optimization: 

Performing gradient-based optimization: 

Iteration 0:   log likelihood =  -8664.574  
Iteration 1:   log likelihood = -8664.5739  

Computing standard errors:

Mixed-effects ML regression                     Number of obs     =      4,943

-------------------------------------------------------------
                |     No. of       Observations per Group
 Group Variable |     Groups    Minimum    Average    Maximum
----------------+--------------------------------------------
   first_school |         23          7      214.9        537
             ID |        509          1        9.7         16
-------------------------------------------------------------

                                                Wald chi2(6)      =     116.56
Log likelihood = -8664.5739                     Prob > chi2       =     0.0000

-----------------------------------------------------------------------------------
           AvPAwk |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
------------------+----------------------------------------------------------------
       person_ev1 |   .0268595   .0288265     0.93   0.351    -.0296394    .0833585
   person_circum1 |  -.1678182    .028333    -5.92   0.000    -.2233498   -.1122865
    extrinsic_ev1 |   .0416861   .0295803     1.41   0.159    -.0162902    .0996624
extrinsic_circum1 |  -.0421132   .0274558    -1.53   0.125    -.0959255    .0116992
            cycle |   .1337544   .0188523     7.09   0.000     .0968046    .1707041
           cycle2 |  -.0081533   .0010474    -7.78   0.000    -.0102063   -.0061003
            _cons |   4.104399   .1151006    35.66   0.000     3.878806    4.329992
-----------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
first_school: Identity       |
                  var(_cons) |   .0660652   .0470587      .0163552    .2668639
-----------------------------+------------------------------------------------
ID: Unstructured             |
                  var(cycle) |   .0141876   .0016478      .0112992    .0178145
                  var(_cons) |   2.159549   .1940144      1.810885    2.575344
            cov(cycle,_cons) |  -.1033947   .0151213     -.1330319   -.0737575
-----------------------------+------------------------------------------------
               var(Residual) |   1.418517   .0316904      1.357745    1.482009
------------------------------------------------------------------------------
LR test vs. linear model: chi2(4) = 2179.91               Prob > chi2 = 0.0000

Note: LR test is conservative and provided only for reference.

[Marcos Almeida]

It seems the residual variance didn't change much with the addition of first_school.

[Patrick Abi Nader]

Hello Marcos,

As far as I understand you are talking about the values 1.418693 and 1.418517. They did not change much I agree. But is this something that I can use to justify not using the models that include the first_school random intercept. By trying to get the explained variance I am hoping to be able to provide support for the claim that the addition of first_school takes up degrees of freedom for no real benefit and that using the more simplistic model was more appropriate. I have a reviewer on a paper that is challenging the non-inclusion of first_school.

Best wishes,
Patrick

[Marcos Almeida]

Maybe the proportional reduction in error would help you to inform that the second model (i.e., the inclusion of the school level) didn't improve much:

 
. di  (1.418693 -1.41851)/ 1.418693
.00012899

This being said, the first model itself may not necessarily be the most appropriate one, for it may have problems related to mispecification, who knows.

[Patrick Abi Nader]

Hello Marcos,

Thank you for your response.

1) Is there anything specific that leads you to make this comment? If you noticed a mistake please let me know.

This being said, the first model itself may not necessarily be the most appropriate one, for it may have problems related to mispecification, who knows.

2) Do you have a suggestion on how I may be able to compare the two xtmepoisson models below? The output does not show residual variance.

. ***************************************************************
. xtmepoisson upa1 person_ev1 person_circum1 extrinsic_ev1 extrinsic_circum1 cycle i.income ///
> if girl==0 || ID: cycle, covariance(uns) irr

Refining starting values: 

Iteration 0:   log likelihood = -8670.9032  (not concave)
Iteration 1:   log likelihood = -8444.8192  (not concave)
Iteration 2:   log likelihood = -8228.7162  

Performing gradient-based optimization: 

Iteration 0:   log likelihood = -8228.7162  (not concave)
Iteration 1:   log likelihood =   -8038.37  
Iteration 2:   log likelihood = -8011.3102  
Iteration 3:   log likelihood = -7966.8205  
Iteration 4:   log likelihood = -7952.2324  
Iteration 5:   log likelihood = -7951.7285  
Iteration 6:   log likelihood = -7951.7267  
Iteration 7:   log likelihood = -7951.7267  

Mixed-effects Poisson regression                Number of obs     =      3,699
Group variable: ID                              Number of groups  =        391

                                                Obs per group:
                                                              min =          1
                                                              avg =        9.5
                                                              max =         16

Integration points =   7                        Wald chi2(7)      =     124.92
Log likelihood = -7951.7267                     Prob > chi2       =     0.0000

-----------------------------------------------------------------------------------
             upa1 |        IRR   Std. Err.      z    P>|z|     [95% Conf. Interval]
------------------+----------------------------------------------------------------
       person_ev1 |   1.005079   .0155278     0.33   0.743     .9751013    1.035978
   person_circum1 |   .9956365   .0177582    -0.25   0.806     .9614323    1.031057
    extrinsic_ev1 |   1.044291   .0165552     2.73   0.006     1.012343    1.077248
extrinsic_circum1 |   1.034818   .0162453     2.18   0.029     1.003463    1.067153
            cycle |   .9485198   .0053694    -9.34   0.000     .9380541    .9591023
                  |
           income |
               2  |   .9111861   .0863383    -0.98   0.326     .7567502    1.097139
               3  |    .874876   .0886445    -1.32   0.187     .7172999    1.067068
                  |
            _cons |   3.512321   .2656373    16.61   0.000     3.028431    4.073527
-----------------------------------------------------------------------------------
Note: _cons estimates baseline incidence rate (conditional on zero random effects).

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
ID: Unstructured             |
                   sd(cycle) |   .0780514   .0051462      .0685896    .0888184
                   sd(_cons) |   .8528442   .0470554       .765429    .9502425
           corr(cycle,_cons) |  -.4214816   .0668651     -.5433213   -.2822532
------------------------------------------------------------------------------
LR test vs. Poisson model: chi2(3) = 4979.10              Prob > chi2 = 0.0000

Note: LR test is conservative and provided only for reference.

. xtmepoisson upa1 person_ev1 person_circum1 extrinsic_ev1 extrinsic_circum1 cycle i.income ///
> if girl==0 || first_school: || ID: cycle, covariance(uns) irr

Refining starting values: 

Iteration 0:   log likelihood = -8684.8635  (not concave)
Iteration 1:   log likelihood = -8059.4895  (not concave)
Iteration 2:   log likelihood = -8010.8601  

Performing gradient-based optimization: 

Iteration 0:   log likelihood = -8010.8601  
Iteration 1:   log likelihood = -8001.8079  (not concave)
Iteration 2:   log likelihood = -7991.9404  (not concave)
Iteration 3:   log likelihood = -7967.3883  (not concave)
Iteration 4:   log likelihood = -7952.4959  
Iteration 5:   log likelihood =  -7948.988  
Iteration 6:   log likelihood = -7948.2745  
Iteration 7:   log likelihood = -7948.2731  
Iteration 8:   log likelihood = -7948.2731  

Mixed-effects Poisson regression                Number of obs     =      3,699

----------------------------------------------------------------------------
                |     No. of       Observations per Group       Integration
 Group Variable |     Groups    Minimum    Average    Maximum      Points
----------------+-----------------------------------------------------------
   first_school |         21          5      176.1        380           7
             ID |        391          1        9.5         16           7
----------------------------------------------------------------------------

                                                Wald chi2(7)      =     127.56
Log likelihood = -7948.2731                     Prob > chi2       =     0.0000

-----------------------------------------------------------------------------------
             upa1 |        IRR   Std. Err.      z    P>|z|     [95% Conf. Interval]
------------------+----------------------------------------------------------------
       person_ev1 |   1.005066   .0155176     0.33   0.743     .9751079    1.035945
   person_circum1 |   .9954913   .0177569    -0.25   0.800     .9612898     1.03091
    extrinsic_ev1 |    1.04506   .0165694     2.78   0.005     1.013084    1.078045
extrinsic_circum1 |    1.03469    .016242     2.17   0.030     1.003341    1.067018
            cycle |   .9477253    .005402    -9.42   0.000     .9371966    .9583723
                  |
           income |
               2  |   .8991973    .091543    -1.04   0.297     .7365432    1.097771
               3  |   .8231023   .0966675    -1.66   0.097     .6538623    1.036147
                  |
            _cons |   3.620082   .3653715    12.75   0.000     2.970349    4.411938
-----------------------------------------------------------------------------------
Note: _cons estimates baseline incidence rate (conditional on zero random effects).

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
first_school: Identity       |
                   sd(_cons) |   .2318126   .0759054      .1220161    .4404098
-----------------------------+------------------------------------------------
ID: Unstructured             |
                   sd(cycle) |   .0780331    .005133      .0685942    .0887708
                   sd(_cons) |   .8366688   .0469676      .7494974    .9339788
           corr(cycle,_cons) |   -.442216   .0669629     -.5636262   -.3021033
------------------------------------------------------------------------------
LR test vs. Poisson model: chi2(4) = 4986.00              Prob > chi2 = 0.0000

Note: LR test is conservative and provided only for reference.