Interpreting Rho in Random-effects logistic model

[roxy kallsen]

Hello statalist users,

I am interpreting results from my random effects logistic model. The F model test is significant and I ran a Hausman test on individual imputation data sets prior to deciding whether or not to use FE or RE, they failed to reject the null hypothesis. So what does rho here mean? And is a 5.15 within the range of rho's or is there something wrong with these results?

Thank you in advance,

[Clyde Schechter]

Your value of rho is not 5.15. It is 5.15e-07 which is how Stata outputs 5.15 x 10^-7. So this is within the range of values for rho, which is from 0 to 1. The interpretation of this is that rho is, for practical purposes zero, and that there is, for practical purposes, very little residual outcome variation at the ID level--nearly all of the variation is occurring within IDs.

[roxy kallsen]

Thank you, Clyde Schechter.

On another note, is there a way to get model fit statistics for this logistic regression with imputed data? Or do I ignore this information and focus on the RE model statistics only?

[Clyde Schechter]

Post-estimation statistics are rather under-developed with multiple imputation. As far as I know the evaluations of performance of logistic regression models that I typically use, area under the ROC curve and a Hosmer-Lemeshow table have not been implemented for use with multiply imputed data. Actually, given the way these are computed, I don't think that Rubin's rules would be applicable anyway (but I have not looked into this in any depth.)

daniel klein is far more knowledgeable about this kind of thing than I am, and I hope he will see this thread and respond.

[roxy kallsen]

Thank you, Clyde.

[Rich Goldstein]

re: Clyde Schechter ,#4,the following article discusses for which statisics Rubin's rules can and cannot be used:

Marshall, A, Altman, DG, Holder, RL and Royston, P (2009), "Combining estimates of interest in prognostic modeling studies after multiple imputation: current practice and guidelines," BMC Medical Research Methodology, 9: 57, doi:10.1186/1471-2288-9-57

I note that the area under the ROC curve, in this article, is listed as "robust methods" not Rubin's rules; however, I used both on about 3-4 (thus not strong evidence) cases and found the result differed in only about the 4th decimal; note that I have previously posted code for getting the area under the ROC curve via Rubin's rules; see #6 in https://www.statalist.org/forums/for...ng-mi-estimate

[daniel klein]

White et al. (2011) explicitly list the area under the ROC curve as an example to which Rubin's rules apply. StataCorp. use this as an example in their FAQ on this topic.

In general, the parameters that you want to combine should be population parameters and they should be approximately normal; if the latter is not true, one can try and find an appropriate transformation (see the first example of combining R-squared in the FAQ cited above).

Best
Daniel


White, I. R., Royston, P., Wood, M. A. 2011. Multiple imputation using chained equations: Issues and guidance for practice. Statistics in Medicine 30:377-399.

[Clyde Schechter]

Thanks, Rich and Daniel!

[roxy kallsen]

Thank you, all.

[roxy kallsen]

Hi Daniel,

Thank you again for pointing out the STATA example and the White et al., (2011) paper. I found them extremely helpful. A couple of questions.

First, I tried running the syntax presented in the FAQ link you provided in #7.

mi xeq 0: roctab infractions contact 
tempname b V
mat `b' = r(area)
mat `V' = r(se)^2
local N = r(N)
mat colnames `b' = AUC
mat colnames `V' = AUC
mat rownames `V' = AUC     
ereturn post `b' `V', obs(`N')
ereturn local cmd "eroctab"
ereturn local title "ROC area" 
mi estimate, cmdok: eroctab infractions contact

I was able to get the following output (please see attachment). When I tried saving the estimate and its variance in temporary matrices I received the following error: "non e-class program may not set e()"

The data was originally in wide format, then I converted it to flong and reshaped it to long to set it to xt and run the final model

mi estimate, or : xtlogit infractions contact pwas therapy age miles length race, re i(ID)

I ran the model prior to the running the under the ROC command. I am not sure what I am missing here. Does it have to do with the fact that I am working with xt data?

Second, in the command for under the ROC how many variables are we expected to put? I first had only infractions and I received an error: "too few variables specified." So then I added all the variables in the model and received the error: "too many variables specified." So I simply included the DV and the main predictor (contact). Am I correct in doing this?

Thank you in advance for your time and assistance.

Roxy

[daniel klein]

The syntax you cite does not match what is proposed in the FAQ. I copy the relevant code from the FAQ here.

cap program drop eroctab
program eroctab, eclass properties(mi)
        version 12.0

        /* Step 1: perform ROC analysis */
        args refvar classvar
        roctab `refvar' `classvar'

        /* Step 2: save estimate and its variance in temporary matrices*/
        tempname b V
        mat `b' = r(area)
        mat `V' = r(se)^2
        local N = r(N)

        /* Step 3: make column names and row names consistent*/
        mat colnames `b' = AUC
        mat colnames `V' = AUC
        mat rownames `V' = AUC

        /*Step 4: post results to e()*/
        ereturn post `b' `V', obs(`N')
        ereturn local cmd "eroctab"
        ereturn local title "ROC area"
end

For your second question, please read

help roctab

I am not using this command; others can advise much better.

Best
Daniel

[roxy kallsen]

Thank you, Daniel. I will try this code.