Alternatives to backward stepwise elimination for variable selection in the case of multiply imputed datasets

[Federico Tedeschi]

Dear Statalisters,

I'm using Stata 19.

Here:
https://www.statalist.org/forums/for...ple-imputation
and here:
https://www.statalist.org/forums/for...e-imputed-data
users asking how to perform stepwise regression for multiple imputation receive the usual caveat that it is an outdated approach, proved to often select the set of predictors that is not the best one. However,

1. it seems to me mim does not work with other commands for model selection,

2. bmaregress doesn’t accept either multiple imputation (returing the error:

matrix e(b) is not set
matrix e(V) is not set

) or importance weights (only fweights are allowed);

3. lasso doesn’t accept multiple imputation (returning the error:

matrix e(V) is not set

) and doesn’t give importance weights the interpretation they have with “regress” (10 observations with weight 0.1 bringing the same evidence of 1 observation with weight 1, so that you can give importance weight 1/m to each observation and just use imputed data to avoid confidence intervals being moved toward 0 by multiple imputation, as in the approach W1 here:
https://pubmed.ncbi.nlm.nih.gov/18203127/
stating: "These weights scale the log likelihood for the stacked data to the equivalent of a data set of length n but ignore the degree of missing information."
).
As for the user-written commands:

4. miinc just performs multiple imputation regression (as if “mi estimate: regress” were called), and it seems to me to return an error if weights or “if” are used in “regress” (while I see it accepts the options listed after a comma), thus I can neither use importance weights nor limit the sample to imputed data. In particular I get the message:

The model
[...]
resulted in an error.
Check to see if the estimation command is missing a comma to separate the command from the options.
A comma is necessary to allow constraints() to work properly within miinc me.

5. vselect either (with "best") gives me the same error as "bmaregress"/performs regression ignoring the users request to select predictors (I don't know why I get different behaviours in different moments, however both insatisfactory) or requires the choice of predictors to be the same across multiply imputed dataset, that is clearly something that can’t be guaranteed (otherwise one could apply the command on any single imputed dataset, knowing they would get the same result with another dataset). In particular, with "backward" or "forward" I get the error:

mi estimate: omitted terms vary
The set of omitted variables or categories is not consistent between m=1 and m=2; this is not allowed. To identify varying sets, you can use mi xeq to run the command
on individual imputations or you can reissue the command with mi estimate, noisily
r(498);

Also, it allows for fweights, aweights and pweights, but not for iweights.

All in all, I have the impression that the stepwise procedure for variable selection is used for lack of simple alternatives. I also found this paper (alongside the one linked above):
https://pmc.ncbi.nlm.nih.gov/articles/PMC7665277/
using backward stepwise elimination (in particular, with the 0.157 p-value as threshold, given its equivalence with the AIC).

Am I overlooking something, or the alternative is between preparing an ad-hoc syntax and using stepwise regression despite its known flaws?

[Felix Bittmann]

Technically, you can use any estimation command with imputed data. Simply estimate your command, separately for each of the imputations, collect point estimates and standard errors and apply Rubin's rules. This works, the question is whether it gives valid results. Personally, I would use this approach for lasso. If predictions differ between imputations (say, M1 selects V1 and V2, but M2 selects V2 and V3), I would include them both in my final model (V1, V2, V3). If anyone else has additional input I am happy to learn about models that work with MI out of the box. Any methods that rely on bootstrapping or resampling (also Bayesian methods) add another layer of complexity, so these I would try to avoid with imputed data.

[Carlo Lazzaro]

Federico:
I would recommend -lasso- too.