Using repeated measures ANOVA vs. linear mixed effects model to evaluate changes in scores after training

[Sam Honer]

Hello,

I have a dataset that contains survey responses from participants attending a training workshop. Surveys were administered prior to training, just after training, and 3 months post training. Moreover, there are a total of 5 trainings were carried out over a period of 2 years; however, there were no major changes in training content or the way the trainings were administered (although the trainers might have changed over time). I am interested in investigating the change in scores from pre to post (to evaluate improvements in score from time 1 to time 2 and retention of improvements from time 2 to time 3). Finally, I'm also interested in determining the effect of certain potential confounders (age, sex, and previous training) on the change in scores.


I am unsure about whether to run a mixed repeated measures ANOVA here with the between subjects variables being year of training (or training batch), age, and sex. Or if it would be more suitable to run a linear two-level mixed effects model with, with time as a fixed-effects predictor and participants as a level 1 cluster. Please advise on what would be more suitable or how to determine the most suitable approach in this case.

Moreover, seeing as the trainings did not change (in terms of content) would it be reasonable to not consider it year or batch of training as a covariate?

Thank you,
Sam

[Joseph Coveney]

I am unsure about whether to run a mixed repeated measures ANOVA here with the between subjects variables being year of training (or training batch), age, and sex. Or if it would be more suitable to run a linear two-level mixed effects model with, with time as a fixed-effects predictor and participants as a level 1 cluster. Please advise on what would be more suitable or how to determine the most suitable approach in this case.

There's a wrinkle in your case in how to specify a continuous between-subjects predictor (age) in a repeated measures ANOVA model. But otherwise, with a small-sample adjustment (e.g., Kenward-Roger) in the iteratively fit mixed-effects linear regression model, shouldn't the latter give you the same result as the former in a balanced dataset?

Moreover, seeing as the trainings did not change (in terms of content) would it be reasonable to not consider it year or batch of training as a covariate?

That's more of a substantive-knowledge question, coupled with your audience's taste in what passes for reasonableness. You could explore, or adjust for, any differences between training batches, if you wanted to, just by adding training batch to the model.

[Sam Honer]

Ah I see, I just wasn't sure what the difference between two models (RM ANOVA and Linear Mixed Effects Model) was because as far as I am aware the RM ANOVA also takes into account the clustering within individuals and allows for random effects. I'm guessing that in case sample sizes vary substantially (however that may be determined) across the three occasions it might be best to used a mixed effects model?

I guess using a mixed effects model might also be useful in case we think that that those who start out with better scores show less improvement over time - in which case this can be modelled using a random slope model?

[Joseph Coveney]

the RM ANOVA also takes into account the clustering within individuals and allows for random effects.

They both do. Again, with a balanced dataset,

anova outcome group / participant|group time group#time

and

mixed outcome i.group##i.time || participant: , reml dfmethod(kroger)
contrast group time group#time, small

will give you identical results.

I'm guessing that in case sample sizes vary substantially (however that may be determined) across the three occasions it might be best to used a mixed effects model?

Yes, ANOVA assumes that you'll have equal numbers of time observations between participants. Nevertheless, -mixed- is no cure for informative missingness (dropout).

I guess using a mixed effects model might also be useful in case we think that that those who start out with better scores show less improvement over time - in which case this can be modelled using a random slope model?

I'm not sure that including a random slope alone will fix that up for you better than, say, ANCOVA.