Averaged kappa in longitudinal data

[Jorge Ruiz Moreno]

Dear Support Service,
I have an ordinal scale (values 1, 2, 3) and 2 raters rating individuals (ind) over 4 occasions (time). I woukd like to obtain intra rater and interrater agreement. Due to longitudinal structure I don't know how to compute the average kappa.
Here and invented example of data:

ind	time	rater1	rater2
1	1	3	2
1	2	3	3
1	3	2	1
1	4	2	2
2	1	1	2
2	2	2	3
2	3	3	3
2	4	2	2

Could you be so kind to indicate me some method or literature?
Thank you in advance,
Jorge

[Eric Makela]

Hello Jorge, are you sure the longitudinal structure changes your analysis of the kappa? I'd refer you to the .pdf documentation for this series of analyses.

The 'kappa' command is sub settable, so you can do the analysis by time period, or by another categorical variable; the phrase 'average kappa' implies the average of all observations in your dataset, but it doesn't sound like that's what you mean. What is your hypothesis? Any commands you put to your data should work towards answering your research question or creating its supporting documentation.

[Jorge Ruiz Moreno]

Dear Eric,
Thank you for your answer.
With averaged kappa I meant weighted kappa, the special kappa for ordinal variable (sorry for the bad expression). Do you mean that I can use kapci for each time point?. That is,

kapci rater1 rater2 if time == 1, w(w) r(1000)
kapci rater1 rater2 if time == 2, w(w) r(1000)
kapci rater1 rater2 if time == 3, w(w) r(1000)

What about replications inside raters?
Thank you again,
Jorge

[daniel klein]

With the data structure in #1

kap rater1 rater2

provides a measure of inter-rater agreement.

For intra-rater agreement, the data structure needs to be changed to something like

generate `c(obs_t)' id = _n
quietly reshape long rater , i(id)
rename rater rating
rename _j    rater
drop id
quietly reshape wide rating , i(rater ind) j(time)

Then,

kap rating1 rating2 rating3 rating4 , wgt(w)

kapci is community-contributed software, probably from SJ. The confidence intervals might not be appropriate because they are based on bootstrapping observations; given the nested data, bootstrapping clusters might be more appropriate.

[Jorge Ruiz Moreno]

Thank you very much Daniel.

From the following data

ind	time	rater1	rater2
1	1	2	2
1	2	3	3
1	3	3	3
1	4	3	2
2	1	4	4
2	2	3	3
2	3	2	2
2	4	3	4
3	1	2	3
3	2	2	3
3	3	3	3
3	4	3	3
4	1	3	3
4	2	2	2
4	3	3	3
4	4	1	1
5	1	1	1
5	2	1	2
5	3	2	2
5	4	2	3

I change the structure to wide and obtain


generate `c(obs_t)' id = _n quietly reshape long rater , i(id) rename rater rating rename _j rater drop id quietly reshape wide rating , i(rater ind) j(time)

rater	ind	rating1	rating2	rating3	rating4
1	1	2	3	3	3
1	2	4	3	2	3
1	3	2	2	3	3
1	4	3	2	3	1
1	5	1	1	2	2
2	1	2	3	3	2
2	2	4	3	3	4
2	3	3	3	3	3
2	4	3	2	3	1
2	5	1	2	2	3

and

kap rating1 rating2 rating3 rating4 , wgt(w) Unfortunately we obtain option opts not allowed r(198); Any idea?

Thank you again,
Jorge

[daniel klein]

I forgot that, for some reason, Stata's kap does not allow weights for disagreement for more than two raters.* You can download kappaetc from SSC:

ssc install kappaetc

The syntax is the same

kappaetc rating1 rating2 rating3 rating4 , wgt(w)

but see

help kappaetc

and Klein (2018) for more details.


* The error message is a bug, by the way. It should probably be: "option wgt() not allowed"


Klein, D. 2018. Implementing a general framework for assessing interrater agreement in Stata. The Stata Journal, 18(4): pp. 871–-901.

[Jorge Ruiz Moreno]

Thank you Daniel!