Within group interrater agreement - rwg / ICC / Kappa

[Valentina Schneider]

Hi all,

I have been trying to figure out how to calculate the interrater agreement within the work teams in my dataset. I only find advice for all subjects but not group-specific calculations. I have data shaped as follows:

IDindiv	IDgroup	item1	item2	item3	item4	AVGindiv	AVGgroup
1	1	2	2	2	2	2	2.66
2	1	2	4	2	4	3	2.66
3	1	2	3	4	3	3	2.66
4	2	1	1	1	1	1	1.5
5	2	1	3	1	3	2	1.5
6	3	2	2	2	2	2	3
7	3	3	3	3	3	3	3
8	3	2	4	3	3	3	3
9	3	3	5	4	4	4	3

IDindiv is every inidividual in my data set.
IDgroup is the group they have been randomly assigned to (in a classroom exercise).
item1-4 are the outcomes they were measured on, all items that were synthesized to a scale.
AVGindiv is the average of each individual's answers to each item (so, the avg of the 4 scores).
AVGgroup is the average of all individuals' average scores collated to a group average (the same for all members of the same group). Groups varied in size, each individual is only part of one specific group.

How can I calculate the interrater agreement within the groups, i.e. see whether there is enough agreement between individuals from one group to synthesize their individual scores to a group average? I honestly already had difficulties for the reshape long command, as I don't know how to do that within the groups, either. Would be thankful for any advice on the ira package I tried to use, or any other way this can be calculated (Rwg, ICC, or others).

Thank you!

[Camilla Marabini]

Dear Valentina,

you should be able to run ira by doing:

ssc install ira 
reshape long item, i(IDIndiv) j(itemno) // now every individual has 4 rows each containing the response to 1 item
ira IDIndiv item, group(IDGroup) options() item(itemno)

where in options you fill in the number of response items.