fitting mixed model to factorial survey data

[James McGovern]

Hi,

I am currently conducting an online survey in a factorial setting ("vignette study"). I have 8 vignettes in total, varying in three dimensions (let us call them Dimension A, Dimension B and Dimension C), each of which has two attributes (so basically a 2x2x2 universe). The participants (university students) rate all 8 vignettes (different seminar descriptions); the vignettes are shown in a random order.

examples:

- vignette 1: "The seminar is taught by a lecturer who has limited experience in research in this field. During the sessions, students mainly listen to the instructor’s presentation. The assessment procedures and grading criteria are not explained in detail”

- vignette 2: "The seminar is taught by a lecturer who has much experience in research in this field. During the sessions, students often take part in discussions. The assessment procedures and grading criteria are explained in advance, and students receive feedback on their performance."

So the three dimensions in the vignettes are: “experience” (low vs. high degree), “participation” (low vs. high degree) and “transparency of grading” (low vs. high degree).Then participants score all vignettes on these three different statements (5-point likert scale; ranging from “not agree at all” to “fully agree”): - “This seminar deviates from seminars I am used to in my studies”. - “I find this seminar appealing” - “I think that the university administration would view this seminar as an example of high teaching quality.” // I do not average these ratings, but either want to include these these scorings as three dependent variables in one model or would like to fit three models (with one dependent variable) to these data.



I want to fit a mixed effect model to the data, with respondent ID as a random effect, and various fixed effects. For the fixed effects: In addition to the three dimension variables (see above), I want to include these respondent-specific independent variables:

• gender,
• field of study (nominal),
• semester (numerical),
• 5 personality factors (numerical data, based upon 5-point likert-scale on personality questions)
• and attitudes towards studying at university (numerical data, based upon 5-point likert-scale).

As a dependent variable, I want to include participants´ ratings of the vignettes. There were three ratings for each vignette (each of which measured with a 5-point likert scale). The rating represent participant´s evaluations of the vignettes.

The number of participants will be (approx.) 170.

I wanted to use the lme4 package in rstudio to model this. However, it seems that it can only be used for one dependent variable, not for more than one dependent variable?

Then, I ask myself how I transform the data into long format. Thus far my columns are:

• participant ID;
• gender;
• field of study;
• semester;
• personality factor 1;
• personality factor 2;
• personality factor 3;
• personality factor 4;
• personality factor 5;
• attitude to studying;
• dimension 1 of vignette;
• dimension 2 of vignette;
• dimension 3 of vignette.

- Do I then have to add three separate columns for each rating of the vignette? However, this means that several cells in the table will be empty. Can the lme4 package in rstudio handle this?

- Long format then means that I will have 4080 rows (in case of 170 participants), is this correct? (170 participants x 8 vignettes x 3 ratings)


Here some exemplary data (In Table 1 (two participants, only 3 vignettes included here) I included the three dependent variable in one row. In Table 2 (just one participant) I have them separate in different rows (which is why some cells are empty "NA"). For the likert scale I assume that I can give numbers (e.g. 1 to "not at all agree" and 5 to "fully agree") . In both Tables I excluded some respondent-specific independent variables (for the sake of illustration):

[Joseph Coveney]

I wanted to use the lme4 package in rstudio to model this. However, it seems that it can only be used for one dependent variable, not for more than one dependent variable? . . . Do I then have to add three separate columns for each rating of the vignette? However, this means that several cells in the table will be empty. Can the lme4 package in rstudio handle this?

You'd probably be better off asking those questions on a forum for R users.

You could fit a model to those data using Stata in a couple of different ways, for example, using mixed, especially if, as it appears from your illustrations, the five-level ordered-categorical outcome variables can take half-values so that there are a total of 10 levels. You'd have to examine the distribution of the 10 potential categories, though, in order to determine whether a sufficient subset of them, say, at least seven, are represented among the realized data; otherwise, you could look into fitting an ordered-categorical generalized linear mixed model with meoprobit or meologit.

Long format then means that I will have 4080 rows (in case of 170 participants), is this correct? (170 participants x 8 vignettes x 3 ratings)

It appears so. If you wish to keep the outcome variables in wide format, then (in Stata) you could use sem in lieu of mixed or gsem in lieu of meoprobit or meologit.

The number of participants will be (approx.) 170.

You have a relatively large number of predictors and a somewhat involved study design. One suggestion would be to use simulation with anticipated distributions of the predictor values and outcome values to verify that 170 participants will be sufficient to yield your desired level of precision for the estimates of the parameters that you're interested in.