Kappaetc provides very different values for Krippendorf alpha/Gwet's AC

[Apphia Tan]

Hi there,

I have 30 participants rating 29 questions on an ordinal 4-point likert scale. I am trying to look at the overall agreement of the participants for this survey round of my Delphi study and used the kappaetc command. However i noticed the values are quite different and am having trouble deciding which statistic is best to represent the data?

I have input kappaetc rater1 rater2 rater3 rater4 rater5 rater6 rater7 rater8 rater9 rater10 rater11 rat
> er12 rater13 rater14 rater15 rater16 rater17 rater18 rater19 rater20 rater21 rater22 rater23
> rater24 rater25 rater26 rater27 rater28 rater29 rater30, categories(1/4) wgt(ordinal)

Any help and advice would be much appreciated!!

Cheers
Apphia

[daniel klein]

I discuss such differences between agreement coefficients in Klein (2018: pp. 876--878; 890-894.).


Klein, D. (2018). Implementing a General Framework for Assessing Interrater Agreement in Stata. The Stata Journal, 18(4), 871–901. https://doi.org/10.1177/1536867X1801800408

[Apphia Tan]

Dear Daniel,

Thank you so much! The article is very helpful and i managed to figure out which stats to present and how to justify. Many thanks again.

Cheers
Apphia