Run seemingly unrelated regression after multi-level models: Combining meglm / mixed with suest?

[Johannes Muller]

Dear all,

I am running a (logit) multi-level model on two sub-samples and afterwards would like to test whether the coefficients for one variable (AA) are significantly different from each other in the two samples.

I know the alternative is to use a single-level logit, and afterwards in the suest cluster the standard errors. The code below works just fine:

logit DepVar $indivcntrls2 AA  if M ==1
est sto M
logit DepVar $indivcntrls2 AA  if F==1
est sto F
suest M F ,  cluster(town)
test [M_DepVar]AA=[F_DepVar]AA

However, I am not really fond of this solution and I would really like to run the seemingly unrelated regression after the multi-level models.
To this end, I am looking for a way to run suest after meglm / mixed or any other multi-level command.

Hypothetical command:

melogit DepVar $indivcntrls2 AA if M ==1 || town:
est sto M
melogit DepVar $indivcntrls2 AA if F==1 || town:
est sto F
suest M F  
test [M_DepVar]AA=[F_DepVar]AA

Is there any way this can be implemented?

I am super grateful for any help and suggestions!

Best

[Joseph Coveney]

Have you considered something like the following?

melogit DepVar $indivcntrls2 i.AA##i.sex || town:

[Johannes Muller]

Dear Joseph,

Thank you for your suggestion. Yes, I have, but as far as I understand, for this to be valid, the other controls should also have the same effect across groups.

Given that this is not the case, I want to split the sample into male and female and then compare the estimated coefficients afterward; essentially, the parallel lines assumption doesn't hold for many controls which is why I test separately.

Is there any elegant way to compare the estimates from two different subsamples after the multi-level model?

Best

PD. Apologies for using a global for the controls, I know quite some people are strongly opposed to their usage...

[Weiwen Ng]

Have you considered something like the following?

melogit DepVar $indivcntrls2 i.AA##i.sex || town:

I endorse this. Recently, my colleagues and I ran a Chow test using the -suest- solution. We were testing if the coefficients on the same variable in one state were different from another state.

A reviewer asked us why we didn't fit a fully interacted model, which in the context of the question would be:

melogit DepVar i.sex##(i.AA $indivcntrls2) || town:

The coefficients produced were ultimately identical. The p-values indicating if a coefficients had a significant difference between states also appeared identical. We were using -regress- in both cases. Nonetheless, I think Johannes is justified in either using Joseph's solution or interacting everything as I presented.

[Joseph Coveney]

the parallel lines assumption doesn't hold for many controls which is why I test separately.

Is there any elegant way to compare the estimates from two different subsamples after the multi-level model?

Yes: use Weiwen's approach. The following proviso holds, though: either all of the variables in $indvcntrls2 are continuous or, if categorical, are of only two categories and are coded 0/1. If you have categorical variables with more than two categories, or if two and they're coded for example 1/2, then separate them out and use the i. factor variable notation for their interaction term.

[Johannes Muller]

Dear Jospeh, dear Weiwen,

Thank you for your suggestions, the fully-interacted model worked like a charm and I implemented, many thanks!

There still is a question I am not sure about, though, because while this approach only works in the context above, the underlying problem of comparing coefficients across (nested-data) samples remains, in particular for these use-cases:

1. Subsample estimations (f.e. restricting sample by education or year) where it would still be conceivable to work with fully interacted models, but it all becomes somewhat tricky to explain / report increasingly complex results.
   It seems to me that sample restrictions are somewhat more intuitive than having more and more interaction effects, especially, if one would want to restrict the sample on several parameters; tripple and n-interactions become very difficult and warrant graphical representations, and this all just becomes too much for actually simple robustness checks.
2. More problematically, if I restrict the sample based on the values of dependent variable then the fully interacted approach cannot be applied.
   How could I test for differences in coefficients after estimating the same MLM (logit) regression on two different sample, where the difference stems from how I define the DV?
   This might sound like I am contructing a selection problem myself, which would warrant f.e. Heckman estimations; that's not the case, the restrictions are theoretically and empirically warranted.

For example:
A = 1; 2; 3
DepVar = 0 if A = 1
DepVar = 1 if A =2  | A =3

meologit DepVar $indivcntrls2 AA  || town:
est sto M1 

*change definition of dependent variable
DepVar2 = 0 if A = 2
DepVar2 = 1 if A =3 // hence omitting A =1 from sample

meologit DepVar2 $indivcntrls2 AA  || town:
est sto M2

I observe differences in coefficients of my independent variable AA but I cannot really be sure that they are different, can I?
How can I formally test for this?

Thanks so much in advance!

Best

[Anna Manzoni]

Johannes, I wonder if you ever found a solution for how to run suest-like command to test coefficients across models with "mixed" (besides the interaction model option). If you did, I'd like to hear about it. thanks

[Johannes Muller]

Dear Anna, in short, I am afraid not. If You find a workable solution please let me know.

[Gurdil Gill]

Dear Anna and Johannes,

I, too, have been (unsuccessfully) trying to run the suest command to test coefficients across mixed-effects ordered logistic models (meologit).

I did find a link (attached) that suggested to run a GSEM model by allowing the error terms to covary, and then test the coefficients. However, I was not able to specify the error terms.

Please have a look and maybe it would be useful for you, if workable.

Thanks