Meaning of variance increase/reduction in repeated measures multilevel models

[Andrea Arancio]

Hi everybody!

I'm building a multilevel model where I have repeated measures of students' performance and some predictors (gender, age, studyhours, etc..). Gender does not vary over time, hours of study do vary. Level 1 is repeated measures, Level 2 is students.
My question is about changes in variance when comparing the empty model with the models where I add predictors.
Can you help me to interpret the following cases?

1. I add gender (time invariant predictor) which is significant and not only reduces the variance at level 2, but also at level 1. Why?

2. Studyhours in the model is significant, is time variant, but not only reduces variance at level 1, it also reduces it at level 2. Why also at level 2?

3. Another time variant variable (significant in the models) reduces variance at level 1, as expected, but increases it at level 2. Why?
Or in another case it reduces variance at level 2 and not at level 1!

4. Last case a level 2 predictor, significant in the models that reduces variance at level 2, but increases it at level one. Why?

Is there a general rule to intepret variance changes with respect to the empty model?

I'm using Stata 13.0, command mixed.

Thanks a lot for your help and patience!
Andrea.

[Carlo Lazzaro]

Andrea:
pasting what you typed and what Stata gave you back in between code delimiters (#-icon among the options of the A-icon Advanced editor) will increase your chances of getting helpful replies.

[Andrea Arancio]

Thanks Carlo. I'm describing the output of many models (pasting everything would make a huge post), but in general I'm using

mixed performance age gender studyhours || StudentID:

.
When I say empty model, I refer to the same model without predictors, i.e.

mixed performance || StudentID:

.

My question is about variance and in the Stata output I refer to the line identified by: var(_cons) and var(Residual).

PS
I'm only testing fixed effects.

[Clyde Schechter]

It is difficult, at best, to give concrete answers to your specific questions. But from a more general perspective, look at it this way. There are many sources of variation in your outcome measure. Some of them you have measured; others are unobserved. Whether these sources of variation are explicitly represented in a model or not, they exert their influence on the outcome variable. When they are not explicitly represented, that variation will still show up somewhere. Whether it shows up as an overestimate of variance on a particular level, or in the residual, or as a bias in the coefficient of one of the fixed effects depends on the details of its distribution at each level and its degree of association with other variables (including the unit indicators at each level). It is, in general, complicated to tease out the details, and would require direct access to the particular data set. But, in general, I would not be surprised at any of the phenomena you have described in your post.

That said, there is one other possibility that you should look into, which would be a problem: when variables are added to a model, any observation with a missing value for that variable gets excluded from the analysis. If your different models are not all estimated on the same sample of observations, then you could be seeing this kind of artifact. Before reaching conclusions, you should make sure that all of your models are using exactly the same set of observations.

[Andrea Arancio]

Thanks Clyde. I previously checked for that and I'm always using the same set of observations. So in this case this is not the issue.

My concern was if the 4 cases described give me results that I could publish or not. I mean If I add a predictor at one level (I find it significant), it reduces the variance at its level, but it increases the variance at the other level, can I say this is a good predictor? Or not?

And how do I justify the case where adding a predictor reduces the variance not only at its level but at both levels?

[Carlo Lazzaro]

Andrea:
as an aside to Clyde's excellent advice, I would consider what other researches did in the past on your very same research topic (eg: did they second a random intercept model only or consider to add a random slope, too?).