Agreement Analysis

[Janet Hill]

I have an agreement analysis where 7 assessors rate 100 images on an ordinal scale of 1~10, there are no missing scale values. There is a tendency to higher values so I decided to use Gwet's AC (kappaetc from SSC) as my test statistic. This is some of the data:

* Example generated by -dataex-. For more info, type help dataex
clear
input byte(image v1 v2 v3 v4 v5 v6 v7)
 1 8 9 10 10 10 7 10
 2 5 7  6  8  7 4  3
 3 2 3  3  5  6 2  1
 4 8 9  7 10  8 7  7
 5 6 5  6  5  5 4  5
 6 4 3  5  4  5 6  6
 7 8 4  6  7  7 5  2
 8 8 7  8 10  8 8  8
 9 7 8  6  5  7 3  4
10 9 9  5  7  6 6  9
11 7 6  3  6  5 3  3
12 4 3  3  3  4 3  5
13 8 5  6  7  7 7  7
14 6 4  7  7  6 6  2
15 9 7  4  9  5 3  4
16 8 4  5  4  5 5  6
17 4 8  8  4  7 4  3
18 3 5  4  2  4 3  5
19 8 8  7  8  7 4  6
20 8 8  8  8  7 6  8
end

. kappaetc v*,wgt(ordinal)

Interrater agreement                             Number of subjects =      20
(weighted analysis)                             Ratings per subject =       7
                                        Number of rating categories =      10
------------------------------------------------------------------------------
                     |   Coef.  Std. Err.    t    P>|t|   [95% Conf. Interval]
---------------------+--------------------------------------------------------
   Percent Agreement |  0.9299    0.0093 100.50   0.000     0.9105     0.9493
Brennan and Prediger |  0.6813    0.0421  16.20   0.000     0.5933     0.7694
Cohen/Conger's Kappa |  0.4450    0.0953   4.67   0.000     0.2457     0.6444
 Scott/Fleiss' Kappa |  0.4377    0.0989   4.43   0.000     0.2307     0.6446
           Gwet's AC |  0.7174    0.0405  17.73   0.000     0.6327     0.8020
Krippendorff's Alpha |  0.4417    0.0989   4.47   0.000     0.2348     0.6486
------------------------------------------------------------------------------

The assessors fall into two groups v1-v5 & v6-v7 and I have now been asked to compare these groups and I am uncertain how to do this. Is it sufficient to inspect the confidence interval and use a Wald test to compare the coefficients or is there a better method?

Thank you.
Janet

[daniel klein]

Because the agreement coefficients are computed on the same set of subjects, they are correlated. For Gwet’s AC, the author proposed a method for testing the difference of correlated agreement coefficients for statistical significance (Gwet 2016). The procedure reduces to a paired t test and is implemented in kappaetc.

Here is an example using your data excerpt:

. kappaetc v1-v5 , wgt(ordinal) categories(1/10) store(group1)

(output omitted)

. kappaetc v6 v7 , wgt(ordinal) categories(1/10) store(group2)

(output omitted)

. kappaetc group1==group2

Paired t tests of agreement coefficients         Number of subjects =      20
                                                 Differences (group1)-(group2)
------------------------------------------------------------------------------
                     |   Diff.  Std. Err.    t    P>|t|   [95% Conf. Interval]
---------------------+--------------------------------------------------------
   Percent Agreement | -0.0019    0.0162  -0.12   0.909    -0.0359     0.0321
Brennan and Prediger | -0.0086    0.0738  -0.12   0.909    -0.1630     0.1458
Cohen/Conger's Kappa | -0.0574    0.1375  -0.42   0.681    -0.3452     0.2303
    Scott/Fleiss' Pi | -0.0554    0.1403  -0.39   0.697    -0.3490     0.2382
           Gwet's AC | -0.0037    0.0692  -0.05   0.958    -0.1485     0.1412
Krippendorff's Alpha | -0.0611    0.1377  -0.44   0.662    -0.3493     0.2271
------------------------------------------------------------------------------

Note that I include the categories() option to ensure that the weighted agreement coefficients in both groups are computed using the same set of weights.


Gwet, K. L. (2016). Testing the Difference of Correlated Agreement Coefficients for Statistical Significance. Educational and Psychological Measurement, 76, 609-637.

[Janet Hill]

Thank you very much, I missed that when looking at the help.
Thank you for a most useful program.
Janet