Two way mixed effects Intraclass correlation coefficient - how to prepare data

[Kate Dutton-Regester]

Hi.

I apologise if this is a simple question answered elsewhere - but I cant seem to find out how to do this-

How do I put my data in stata for ICC analysis, as in what is my dependent variable, what is my target variable and what is my rater variable.

I am using the two way mixed effects ICC.

My scenario - I have a farm worker (FW) and a farm vet (FV). They have rated cows for locomotion on a scale of 1-5 (1 being not lame and 5 being extremely lame).

I want to assess agreement between the FW and FV.

So for each cow the FW has given a rating, and the FV has given a rating.

Kind regards,

Kate

[Marcos Almeida]

Hello Kate,

Welcome to the Forum.

Please see if this excerpt from the manual (http://www.stata.com/manuals13/ricc.pdf), basically the example 3 in page 10, fits your needs.

Best,

Marcos

[Cole Hooley]

Hi Marcos,

Thank you for your response. I read through example 3 on page 10. The syntax was useful, but I still don't understand how the data should be set up.

Question: How should I format the data for a two-way, mixed effect ICC?

Background: I am using Stata version 14.2. I am trying to run an intraclass correlation, two-way mixed effects model. The data consists of fidelity ratings from two observers of 10 therapy sessions. Each rater observed each therapy session and assigned a fidelity score of 0 to 5 for the ten components of the therapy protocol. I have pasted an example of the data below. The two columns are the fidelity scores assigned by Rater1 and Rater2, the rows are the components of the fidelity rating protocol of the intervention for session #1. There are ten components of the fidelity rating protocol, and 10 sessions so there are 100 rows in the full data set.

[CODE]
* Example generated by -dataex-. To install: ssc install dataex
clear
input float(Rater1 Rater2)
0 0
0 1
4 4
5 4
5 4
5 5
2 2
2 2
2 2
2 2

With the data in this format I ran the code:

icc Rater1 Rater2, mixed

When I run that syntax I receive this error message:

. icc Rater1 Rater2, mixed
you must specify the rater variable with mixed-effects models
r(198);

I looked through the

help icc

, and reviewed example 3 on page 10 of the manual. The manual outlines that the syntax requires I enter a dependent variable, target variable and rater variable. In my data set, I only have two variables, the fidelity scores from the two raters. How should I format my data to use the ICC two-way, mixed methods command?

Thanks, in advance!

[daniel klein]

Starting from

input float(Rater1 Rater2)
0 0
0 1
4 4
5 4
5 4
5 5
2 2
2 2
2 2
2 2
end

you want

generate session = _n
rename Rater# score#
reshape long rating , i(session) j(rater)

icc score session rater , mixed

Given the predetermined (ordered) categorical nature of your data, the ICC might not be the most appropriate method for assessing inter-rater reliability (which is I suppose you are after). Have a look at agreement coefficients, e.g. Cohen's kappa (see help kappa), possibly weighted. Also see kappaetc (SCC) for this. These are designed to work with the data in the original setup.

Best
Daniel

[Cole Hooley]

Thank you so much Daniel, I really appreciate your time and help!

Thanks for pointing out the kappa option. I was able to run that program before I attempted the ICC, and seems like I should just stick with those results.

I think that I may have made an error in executing the code you provided.

When I used:

generate session = _n

rename Rater# score#

The data looked like this:

* Example generated by -dataex-. To install: ssc install dataex
clear
input float(score1 score2 session)
0 0  1
0 1  2
4 4  3
5 4  4
5 4  5
5 5  6
2 2  7
2 2  8
2 2  9
2 2 10
end

Then when I executed:

 
 reshape long rating , i(session) j(rater)

I received the following error message:

reshape long rating , i(session) j(rater)
no xij variables found
    You typed something like reshape wide a b, i(i) j(j).
    reshape looked for existing variables named a# and b# but could not find any.
    Remember this picture:

         long                                wide
        +---------------+                   +------------------+
        | i   j   a   b |                   | i   a1 a2  b1 b2 |
        |---------------| <--- reshape ---> |------------------|
        | 1   1   1   2 |                   | 1   1   3   2  4 |
        | 1   2   3   4 |                   | 2   5   7   6  8 |
        | 2   1   5   6 |                   +------------------+
        | 2   2   7   8 |
        +---------------+

        long to wide: reshape wide a b, i(i) j(j)    (j existing variable)
        wide to long: reshape long a b, i(i) j(j)    (j    new   variable)
r(111);

end of do-file

r(111);

Could you point out where I have gone wrong?

Thanks, again. Figuring this out will help me in future iterations.

Cole

[Red Owl]

The variable rating does not appear in the data.

Try this.

reshape long score ,  i(session) j(rater)

Good luck.

Red Owl

[Cole Hooley]

Thanks Red Owl! That did it.