Multi-level Linear Mixed Model - Correcting for multiple comparisons

[Barry Horgan]

Hi, a first-time Statalist forum post for me!
As part of my PhD studies, I've ran a Multi-level Linear Mixed Model (LMM) for my statistical analysis, using the xtmixed command. I've used a random cross-over study design with n=18, 4 treatment groups, and 6 time-points. I'm interested in the time * treatment interaction effect.

Here is a sample of the code that I've ran below:
* Original model, treatment (i.trt) and time as categorical variable (i.time); listing 'c.covariate1##i.covariate2' as covariate fixed effects, one of which is a continuous variable (c.) and the other which is a categorical variable (i.)
xtmixed `x' i.trt##i.time c.covariate1##i.covariate2 || id: || _all: R.time, reml nolog

* Test of simple effects (within columns or rows), and interaction effects (treatment at each time)
contrast trt@time, effect
* as above, using Scheffe's method for multiple pairwise comparisons
contrast trt@time, effect mcompare(scheffe)

1.) My first query relates to the use of a correction method for multiple pairwise comparisons. I have used Scheffé's method, however, I find this to be very conservative. Using LMM's (xtmixed) command, how important is it to correct for multiple pairwise comparisons i.e. as one would typically do in ANOVA style tests? If/Assuming it is still important, is Scheffe's method suitable choice for my study design and choice of statistical analysis? Below are two images showing the p-values without no correction (top image) and using Scheffe's method (bottom) to correct for multiple pairwise comparisons. The Scheffe's p-values look extremely conservative to me.





2.) My second question is in relation to interpreting the effect of the covariates (c.sum7skinfolds##i.beliefcategory) on the model. The i.beliefcategory is a time-varying covariate that is either '1 = believer' or '2 = non-believer' depending on the treatment group they were in at the time. The other c.sum7skinfolds is a continuous non-varying continuous variable. See images of a sample output below. Happy if you can help me with interpreting these please.

Appreciate your time and expertise here. Any further questions, please ask. Thanks, Barry

[Barry Horgan]

Hi All,
Just a gentle follow-up to see if anybody can assist re above?
Thank-you for your time and consideration.
Thanks,
Barry

[Joseph Coveney]

I have used Scheffé's method, however, I find this to be very conservative.

Sheffé's method is post hoc, and so is intended to be conservative. Scanning through your regression output, I'm not sure whether it makes any difference in your case.

Using LMM's (xtmixed) command, how important is it to correct for multiple pairwise comparisons i.e. as one would typically do in ANOVA style tests?

Why would it make any difference whether you fit a mixed model with ANOVA or by means of an iterative maximum-likelihood method? To me, the decision to correct for pairwise comparisons is independent of the decision about the model-fitting algorithm.

My second question is in relation to interpreting the effect of the covariates (c.sum7skinfolds##i.beliefcategory) on the model.

I'm not sure what you're looking for here, but, again, scanning through the output, I'm not sure whether there is much there to interpret.

As an aside, your model might be misspecified. It would be a cross-classified (study participant × time) random effects model if all study participants' time T were the same date and time-of-day. Such a design implies that you would have 4 treatments × 6 time points each = 24 (and only 24) unique time points at each of which all 18 participants are observed simultaneously. From the fixed-effects portion of the regression output, it doesn't seem to be the case.

[Barry Horgan]

As an aside, your model might be misspecified. It would be a cross-classified (study participant × time) random effects model if all study participants' time T were the same date and time-of-day. Such a design implies that you would have 4 treatments × 6 time points each = 24 (and only 24) unique time points at each of which all 18 participants are observed simultaneously. From the fixed-effects portion of the regression output, it doesn't seem to be the case.

Hi Joseph, thank-you very much for the reply above. This is greatly appreciated.

To clarify re above, 18 participants were randomly allocated and rotated between treatment groups (in a different random order) over the duration of the study i.e. participants completed all treatments at the same time of day, however, participants completed treatments on different dates, as per the random allocation to these groups. Please refer to attached image for visual representation of study design, as well as how I've considered the nesting for the model. Based on these images, do you believe that the model is adequately specified?

Thanks, Barry

[Joseph Coveney]

do you believe that the model is adequately specified?

Naw, I don't.

I recommend that you locate a classical ANOVA-based text (say, B. J. Winer, Statistical Principles in Experimental Design or something similar—I used the 1971 edition way back when, but there are newer editions now) and go through the chapters on nesting versus crossing of factors. I would not consider the relationship to be nesting, as you have in your attachment.

Another thing that I recommend is to include in your regression model the sequence (your "different random order") to which study participants were randomly assigned. Also, you might want to consider including what in the bioequivalence literature is called the period, which in your case is the temporal ordinal number (1 through 4) of the block of six observations. This is distinct from, and not collinear with, the four treatment groups. If you haven't already, then you might want to take a gander at the entries for the pk suite of commands in the Stata user's manual for more on how crossover studies are set up and analyzed.

[Barry Horgan]

Thank-you very much for your time and advice Joseph.