Possible error in stored results when using bootstrapped mixed-effects models

[Martin Dalgaard]

Hello all

I have been making mixed-effects models for a project that I am working on, and as a sensitivity analysis I have furthermore been using bootstrapping on the mixed-effects models.

A rather weird problem, however, seems to be that the results from bootstrapped mixed-effects models shown in the output do not match with the results stored in r(table). Instead, the results stored in r(table) match those from the non-bootstrapped mixed-effects models.

As an example, I have included code for making a simple mixed-effects model on the pig dataset (similar to example 1 on page 487 on https://www.stata.com/manuals/me.pdf) and a bootstrapped version of the same mixed-effects model with 20 repetitions.

webuse pig, clear

mixed weight week || id:
mat list r(table)

bootstrap, reps(20) strata(week) seed(314): mixed weight week || id:
mat list r(table)

The output and the results stored in r(table) for each of the two models are shown on the two pictures below.

As the second picture shows, the bootstrapped version of the mixed-effects model gives slightly wider confidence intervals (due to the slightly higher standard error), but the results stored in r(table) are identical to those from the simple mixed-effects model.

It has turned out that this problem only seems to occur in Stata/MP 17.0 for Windows, which I am using through a server. The problem does not occur on my PC, where I am using Stata/SE 16.1. Using version # does not change whether the problem occurs or not on neither the server nor on my PC. Furthermore, from what I can see, the problem does not occur with other types of bootstrapped models. Finally, the problem does not seem to affect bootstrap postestimation commands.

Does anyone have any idea why it is happening? I do not know a lot about servers, but I somehow doubt that it is specific to the server mentioned above. Thanks in advance.

Yours sincerely
Martin Kamp Dalgaard

[Leonardo Guizzetti]

Thank you for the data example. It would seem that this is a bug to me, as I would expect that -r(table)- always reflect the presented estimation results. It also appears that this issue was introduced in Stata 17 and persists in Stata 18.

I would contact Stata Technical Services and confirm it with them.

Note that the returned V matrix shows the expected results after bootstrap.

webuse pig, clear

mixed weight week || id:
mat list r(table)

bootstrap, reps(20) strata(week) seed(314): mixed weight week || id:
mat list r(table)
di sqrt(e(V)[1,1])

Selected result

. di sqrt(e(V)[1,1])
.04474968

On another note: I don't think your use of bootstrapping here is statistically valid, though I appreciate this might have been a simple example to illustrate your question. A cluster-bootstrap would be appropriate since there are repeated measures over individual pigs. A simple bootstrap would over-represent some weeks for some individuals and not for others, destroying within-pig correlation structure.

[Jeff Pitblado (StataCorp)]

In the Stata 17 update on 08mar2023, mixed was changed to work better with collect and etable. The intent was to get mixed to populate r(table) with the values reported in the estimation table, namely the variance components, where prior to this change r(table) was populated with values in the estimation metric. We missed a coding detail when prefix bootstrap or jackknife is used. In addition, estat sd is reporting the model-based standard errors instead of the bootstrap, but this is an old bug not due to the change on 08mar2023. We hope to have all of this fixed soon.

[Martin Dalgaard]

Hello

Thank you for your answers.

Especially thanks to you, Leonardo, for the suggestion to use cluster-bootstrap, which I had not considered. Although this was just an example, it is similar to the project that I am actually working on.

For anyone wondering, it does not seem that this bug has been fixed yet.

Yours sincerely
Martin Kamp Dalgaard