mixed model and ICC

[Chris Mooney]

Hello --

I am trying to fit a mixed effect linear model that examines the effect of an educational intervention (exp) on students (id) score (score). Each student is observed by two raters (rater). Hence, students are nested in raters.

Specifically, I have the following two questions:

1. Does score (score) differ by level of intervention (exp), accounting for rater differences?

2. What is the inter-rater reliability (ICC) for the two raters?

I have proposed the following syntax:

mixed score i.exp || rater:


Below is the output:


Mixed-effects ML regression Number of obs = 126
Group variable: rater Number of groups = 2

Obs per group:
min = 63
avg = 63.0
max = 63

Wald chi2(1) = 8.62
Log likelihood = -268.17273 Prob > chi2 = 0.0033

------------------------------------------------------------------------------
score | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
1.exp | 1.056061 .359606 2.94 0.003 .3512459 1.760875
_cons | 16.05 .3568169 44.98 0.000 15.35065 16.74935
------------------------------------------------------------------------------

------------------------------------------------------------------------------
Random-effects Parameters | Estimate Std. Err. [95% Conf. Interval]
-----------------------------+------------------------------------------------
rater: Identity |
var(_cons) | .1191623 .1838568 .0057918 2.451701
-----------------------------+------------------------------------------------
var(Residual) | 4.064231 .5161579 3.168661 5.212921
------------------------------------------------------------------------------
LR test vs. linear model: chibar2(01) = 1.55 Prob >= chibar2 = 0.1067


Here, this suggests that score differs across the groups, accounting for the random effect of raters, correct?

Second, how might I go about examining the ICC?

Last, if I wanted to examine whether scores differ by rater, should I add i.rater as a fixed effect, such as:

mixed score i.exp i.rater || rater:

Thanks for the help. I am new to this type of modeling.

Chris

[Clyde Schechter]

Each student is observed by two raters (rater). Hence, students are nested in raters.

Wait, not so fast. If there are a total of two raters and both of them rated every student, then it is raters who are nested in students. The code would then be

mixed score i.exp || id: // id = student identifier
estat icc // GIVES YOU THE INTRA-CLASS CORRELATION
//  WHICH ESTIMATES THE INTER-RATER RELIABILITY

//    AND IF YOU WANT TO ESTIMATE RATER DIFFERENCE
mixed score i.exp i.rater || id:

[Chris Mooney]

Thanks for the response, Clyde. I confess, I am not sure if its residents nested in students, or vice versa...but defer to your expertise as this is new to me. It seems like both could be correct. The syntax worked fine, however I neglected to mention that students have multiple observations (noteid) scored by raters. So, noteid is nested in raters nested in students.

In that case, would it be the following code:

mixed score i.exp || id: ||noteid:
estat icc

or

mixed score i.exp i.rater || id: ||noteid:
estat icc

Thanks for the help...wish I could shake your hand in person.

[Clyde Schechter]

mixed score i.exp || id: ||noteid:
estat icc

[Chris Mooney]

again, thank you