using mi estimate: with svy: for multilevel anylsis

[Fiona Barthmes]

Hello! I have spent a lot of time trying to get my imputation model to work and managed to impute my data now. However I continue running into issues when using the mi estimate: prefix for multilevel logistic analysis.
The code I would like/need to run with mi estimate is:
svy melogit any_ipv || EA_ID:
...and this including many more independent variables and potentially some random slopes.
Setting svy works, as does running simpler models such as: mi estimate: svy: regress any_ipv i.FQ_age_cat i.school_cat
The problem seems to start when including the random effect. Errors I receive are "mi estimate: command not supported" or "xtlogit is not supported by svy with vce(linearized)" even though xtlogit is specified as a command compatible with mi estimate and my data is vce(linearized).

I have seen people here having used code like mi estimate: svy linearized: melogit var1 var2 || cluster:
But this does not work for me and it seems as if stata comes up with a new error message for everything I try. I would highly appreciate any support or advice!

[Andrew Musau]

From the documentation, you have the -cmdok- option for unsupported commands. Consider:

webuse mheart1s20, clear
mi estimate: melogit attack smokes age|| fe:
mi estimate, cmdok: melogit attack smokes age|| fe:

Res.:

. webuse mheart1s20, clear
(Fictional heart attack data; BMI missing)

. 
. mi estimate: melogit attack smokes age|| fe:
mi estimate: command not supported
    melogit is not officially supported by mi estimate; see mi estimation for a list of Stata estimation commands that are supported by
    mi estimate.  You can use option cmdok to allow estimation anyway.
r(198);

. 
. mi estimate, cmdok: melogit attack smokes age|| fe:

Multiple-imputation estimates                   Imputations       =         20
Mixed-effects logistic regression               Number of obs     =        154
                                                Average RVI       =     0.0000
                                                Largest FMI       =     0.0000
DF adjustment:   Large sample                   DF:     min       =  1.35e+130
                                                        avg       =  1.35e+130
                                                        max       =          .
Model F test:       Equal FMI                   F(   2,      .)   =       7.38
Within VCE type:          OIM                   Prob > F          =     0.0006

------------------------------------------------------------------------------------
            attack | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
-------------------+----------------------------------------------------------------
            smokes |   1.143541   .3458387     3.31   0.001     .4657093    1.821372
               age |   .0313176   .0148138     2.11   0.035      .002283    .0603521
             _cons |  -2.470375    .879083    -2.81   0.005    -4.193346   -.7474042
-------------------+----------------------------------------------------------------
 var(_cons[female])|   1.65e-34   3.67e-18                             .           .
------------------------------------------------------------------------------------

[Fiona Barthmes]

Hello Andrew! Thank you so, so much for pointing out cmdok! I am extremely relieved that it works! I checked the 4 conditions that need to be met in order for cmdok to work properly, which are specified here, but with which you are probably familiar already: https://www.stata.com/support/faqs/s...ns%20are%20met.

I just wanted to make sure I understand these correctly. When I run the full model with mi estimate and cmdok, e(b) and e(V) are not properly stored. However when I check the conditions after running the command on only one imputation:
mi xeq 1: svy: melogit ipv_emotional i.FQ_age_cat i.school_cat i.school_partner_cat i.pregnant i.prop_private_matter_cat || EA_ID:

...and check with ereturn list directly after (as indicated in the FAQ linked above), stata correctly stores command name in e(cmd) (shows meglm), coefficients in e(b) matrix, full variance-covariance matrix estimate in e(V) matrix and residual degrees of freedom in the e(df_r) scalar >> all 4 conditions met. Could you tell me if this is sufficient proof that stata properly understands the commands and I can use the -cmdok option?

Thank you a lot in advance, I very much appreciate your support!

[Andrew Musau]

Correct. melogit is an official estimation command, so it does all that. In any case, it would've errored out if it didn't.

[Andrew Musau]

Also, see #7 of https://www.statalist.org/forums/for...it-is-it-legit.

[Fiona Barthmes]

Thank you so much! So from what I understood the results from -cmdok work well, except for the CI for the variance component which I will need to transform using the code you linked?!

I have one other question. I am using data obtained via a complex survey design, meaning I need to set svyset and use the svy: prefix. I am unsure how to compare my model fit for the multilevel mixed-effects models I am running. Some argue that assumptions underlying the AIC are violated and that it thus cannot be used as a measure of model fit for complex survey data. However I have also read that it is common to run the model without the svy prefix but including the individual survey weight just for the purpose of being able to run estat ic after to calculate the model AIC. Could you tell me whether this is a sound approach and in general what the best measure of model fit is for svy-set mixed-effects models? I was thinking to use the deviance and AIC as some authors have done before but wanted to ensure this is sound to do...

Thank you again!

[Andrew Musau]

Likelihood-based statistics (e.g., AIC) are not valid with complex survey data. You can find a discussion of why this is the case here: https://www.stata.com/support/faqs/s...od-ratio-test/. If you are comparing nested models, Wald tests are still valid. You could test the joint significance of the added variables, which tells you whether they improve the model and thus justify their inclusion. Personally, I don’t rely heavily on tests to determine which models are best—a test might suggest that a set of variables is unnecessary, even when theory argues otherwise. Otherwise, I recommend posting this in a new thread to get input from a broader audience.

[Andrew Musau]

Sorry, I missed to reply to this:

Thank you so much! So from what I understood the results from -cmdok work well, except for the CI for the variance component which I will need to transform using the code you linked?!

Correct. If you are only interested in the coefficients, no need to do anything.

[Fiona Barthmes]

Thank you for your detailed responses! I read more into AIC and dAIC which unfortunately seems to not yet be included as an option in STATA compared to other programmes and came across several people who recommend using the Wald tests. I very much appreciate that you took the time to respond, your comments were very helpful!