Melogit: variance of random slope looks insignficant, but LRtest says it is?

[Joe Brown]

Can someone please help me with a multilevel modeling question? I've gotten conflicting advice from colleagues, so I thought I'd ask the experts here...

I'm estimating a mixed effects logit model with several variables, trying to determine which should have random slopes. The command I'm using has the form:

melogit Y X1 X2 || group: X1 X2

X1 appears to have a significant effect on Y, but X2 does not.

The random effects portion of my output looks like this:


| Coef. Std. Err. [95% Conf. Interval]
-----------------+----------------------------------------------------------------
group |
var(X1) | .031886 .0129037 .0144256 .07048
var(X2) | 1.981478 1.375139 .5084618 7.721827
var(_cons) | 1.588888 .6971545 .6723764 3.754689
----------------------------------------------------------------------------------
LR test vs. logistic model: chi2(3) = 461.64 Prob > chi2 = 0.0000

One colleague tells me that the random slope on X2 is not necessary, since the variance looks insignificant (and the variable's effect on Y is insignificant). Another colleague tells me a likelihood ratio test is actually necessary for random slopes. So I tested the model with X1 and X2 random slopes against a smaller model with a random slope on X1 only. (FWIW, X2's effect on Y is insignificant in the simpler model too.)

The LR test result (Prob > chi2 = 0.0003) suggests that the random slope on X2 is preferred, even though the variance LOOKED insignificant based on the larger model's output shown above.

Whose advice do I believe? Do I report the model with random slopes on X1 and X2 or the model with a random slope on X1 only? (And is there a source I can cite to explain this decision to reviewers?)

Thanks!

Joe

[Clyde Schechter]

One colleague tells me that the random slope on X2 is not necessary, since the variance looks insignificant (and the variable's effect on Y is insignificant).

Either you have misunderstood what that colleague told you or they don't know what they're talking about. The statistical significance of X2 is irrelevant to whether the random slope on X2 is a necessary component of the model.

even though the variance LOOKED insignificant based on the larger model's output shown above.

Moreover, the variance does not look insignificant based on the output you show. It's much larger, for example, than the variance of the X1 random slopes, and the lower end of the 95% CI isn't even close to 0. So if you're going to play the significance game in the first place, the rules of that game in no way knock this one out.

Personally, I do not like deciding what to include in a model based on any statistical test. But, again, if you are going to do it that way, rely on the LR test. It's the least bad way to use a statistical test for model selection.

Do read https://www.nature.com/articles/d41586-019-00857-9 for a pep talk on why we all ought to banish "statistical significance" from our vocabulary.

[Joe Brown]

Thank you for your quick response, Clyde! This is very helpful. I'll report the results with random slopes on X1 and X2 and give this Nature article a read.