weighted kappa 95%CI help

[Liana Varrone]

When calculating 95%CIs: how do I dictate that my data are 8x8 when some ratings have not been used in my dataset?

ie. page 7: https://www.stata.com/manuals13/rkappa.pdf

I've calculated weighted kappa statistics but the "absolute" command doesn't work for kapci.

I have 2 raters, 8 possible ratings, and custom weights assigned. I'm also working with reps(1000) bootstrapping.

Any help would be greatly appreciated!

[daniel klein]

Some details are missing here. kapci is (probably) from SJ; absolute is not a command itself but an option of Stata's kap command.

If you want to stick with the bootstrap approach, you can do this yourself with Stata's kap (see example below). For various reasons, I suggest considering kappaetc (SSC). The latter does not yet support bootstrap standard errors but the default standard errors are close to bootstrap's for reasonable sample sizes.

The example code

// example data
clear
set seed 42
set obs 50
generate rater1 = floor(8*runiform() + 1)
generate rater2 = floor(8*runiform() + 1)

// rating category 3 is not observed
replace rater1 = 4 if (rater1 == 3)
replace rater2 = 4 if (rater2 == 3)

// customized weights
kapwgt myw 1 \                       ///
           0.8 1 \                   ///
           0.7 0.8 1 \               ///
           0.5 0.7 0.8 1 \           ///
           0.4 0.5 0.7 0.8 1 \       ///
           0.1 0.4 0.5 0.7 0.8 1 \   ///
           0 0.1 0.4 0.5 0.7 0.8 1 \ ///
           0 0 0.1 0.4 0.5 0.7 0.8 1
    
kapwgt myw

// the bootstrap
bootstrap kappa = r(kappa) , reps(1000) : ///
    kap rater1 rater2 , wgt(myw) absolute

estat bootstrap , all
    
// kappaetc (SSC)
kappaetc rater1 rater2 , wgt(myw) categories(1/8)

and results

. // example data
. clear

...

. kapwgt myw

 1.0000
 0.8000 1.0000
 0.7000 0.8000 1.0000
 0.5000 0.7000 0.8000 1.0000
 0.4000 0.5000 0.7000 0.8000 1.0000
 0.1000 0.4000 0.5000 0.7000 0.8000 1.0000
 0.0000 0.1000 0.4000 0.5000 0.7000 0.8000 1.0000
 0.0000 0.0000 0.1000 0.4000 0.5000 0.7000 0.8000 1.0000

.
. // the bootstrap
. bootstrap kappa = r(kappa) , reps(1000) : ///
>     kap rater1 rater2 , wgt(myw) absolute
(running kap on estimation sample)

...

Bootstrap results                               Number of obs      =        50
                                                Replications       =      1000

      command:  kap rater1 rater2, wgt(myw) absolute
        kappa:  r(kappa)

------------------------------------------------------------------------------
             |   Observed   Bootstrap                         Normal-based
             |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       kappa |  -.0018401   .0836843    -0.02   0.982    -.1658583     .162178
------------------------------------------------------------------------------

.
. estat bootstrap , all

Bootstrap results                               Number of obs      =        50
                                                Replications       =      1000

      command:  kap rater1 rater2, wgt(myw) absolute
        kappa:  r(kappa)

------------------------------------------------------------------------------
             |    Observed               Bootstrap
             |       Coef.       Bias    Std. Err.  [95% Conf. Interval]
-------------+----------------------------------------------------------------
       kappa |  -.00184011   .0025799   .08368427   -.1658583    .162178   (N)
             |                                      -.1618779   .1651446   (P)
             |                                      -.1660355   .1631599  (BC)
------------------------------------------------------------------------------
(N)    normal confidence interval
(P)    percentile confidence interval
(BC)   bias-corrected confidence interval

.        
. // kappaetc (SSC)
. kappaetc rater1 rater2 , wgt(myw) categories(1/8)

Interrater agreement                             Number of subjects =      50
(weighted analysis)                             Ratings per subject =       2
                                        Number of rating categories =       8
------------------------------------------------------------------------------
                     |   Coef.  Std. Err.    t    P>|t|   [95% Conf. Interval]
---------------------+--------------------------------------------------------
   Percent Agreement |  0.6080    0.0426  14.29   0.000     0.5225     0.6935
Brennan and Prediger |  0.0910    0.0987   0.92   0.361    -0.1073     0.2893
Cohen/Conger's Kappa | -0.0018    0.0839  -0.02   0.983    -0.1705     0.1668
    Scott/Fleiss' Pi | -0.0047    0.0840  -0.06   0.956    -0.1736     0.1642
           Gwet's AC |  0.1504    0.0991   1.52   0.135    -0.0486     0.3495
Krippendorff's Alpha |  0.0054    0.0840   0.06   0.949    -0.1635     0.1743
------------------------------------------------------------------------------

Best
Daniel

[Liana Varrone]

Thanks Daniel - your methods gave me slightly wider confidence intervals for the datasets that Stata identified as 8x8, so I've redone my whole analyses using your syntax.

I'm quite new to Stata so am still learning, but you've been very helpful!

Kind regards,
Liana

[daniel klein]

The slightly wider confidence intervals might partly be due to using the t-distribution; bootstrap uses the normal approximation. You can get the latter with kappaetc when you specify the largesample option, which should get you pretty close to bootstrap's normal-based CI. However, I would be more concerned about (larger) differences in the estimated coefficients if there are any.

By the way, I realized that I have omitted a detail myself: I ran the example above using Stata version 12.1; results for more recent versions of Stata (14 or higher) will differ because recent Stata versions have a different (better) random number generator.

Best
Daniel

[Liana Varrone]

Just an FYI:

I'm using Stata version 15.1, and the coefficients I calculated were exactly the same using your method and the method I had used previously.

The first code I used was:

kap rater1 rater2, wgt(myw) absolute

Then for the 95%CIs:

kapci rater1 rater2, wgt(myw) reps(1000)

This kapci is the code that won't work if the data don't appear 8x8 to the software.

Thanks again!

Kind regards,
Liana