[Joe Tuckles]

Yes I am planning to add the bland altman plots. I am just unfamiliar with Kappa statistic, I submitted a paper looking at the correlation between a questionnaire and accelerometer for measuring physical activity. The accelerometer is expensive and a burden to use, whereas a questionnaire is quick and easy, so if it can be shown as valid against the accelerometer that is what I am trying to assess. The peer reviewer has asked me to add in the Kappa statistics and Bland Altman plots so I am not sure out of all the above data what they will specifically want, or what is standard for a paper.

[Bruce Weaver]

A few years ago, the team I am part of collected data from two different versions of Fan & Posner's Attention Network Test (ANT). We included scatter-plots with X = our newer & shorter version of the ANT and Y = the current "gold standard" version. The plots also showed the following:

• The OLS regression line (a solid line)
• The Y=X line (a dashed line)
• Pearson r with 95% CI
• ICC (mixed design, IIRC) with 95% CI
• Mean difference(with 95% CI

We reckoned that this would provide most readers with all of the information they might conceivably want. I've attached an example where the variable of interest was median response time over all trials.

HTH.

[Joe Tuckles]

Thanks that is very helpful! I really like how it is presented. I have found the accompanying paper online but you don't refer to the Kappa statistic. I've looked at other papers too and they seem to just present the coefficient and the confidence intervals and refer to this as 'Quadratically weighted kappa'. I presume they are reporting the 'Percent Agreement' coefficient? I have uploaded a paper as an example. The confusing thing is my Percent Agreement coefficients that I have run all seem very high like >0.9 even though the spearman rho is more like 0.40. So I am not sure I am looking at the right thing?

[daniel klein]

I've looked at other papers too and they seem to just present the coefficient and the confidence intervals and refer to this as 'Quadratically weighted kappa'. I presume they are reporting the 'Percent Agreement' coefficient?

Most likely, these papers do not report the "Percent Agreement"; they report the "Quadratucally weighted kappa"; this is labeled "Cohen/Conger's Kappa" in the output in #14.

[Joe Tuckles]

Excellent thank you :-)

[Joe Tuckles]

Hi all,

Is anyone able to understand why the quadratically weighted Kappa and ICC are quite widely different to spearman rho? (earlier examples were for walking this is for minutes spent running as measured by accelerometer versus questionnaire)

. kappaetc objective_running subjective_running, wgt(quadratic)

Interrater agreement                             Number of subjects =      28
(weighted analysis)                             Ratings per subject =       2
                                        Number of rating categories =      50
------------------------------------------------------------------------------
                     |   Coef.  Std. Err.    t    P>|t|   [95% Conf. Interval]
---------------------+--------------------------------------------------------
   Percent Agreement |  0.9647    0.0286  33.68   0.000     0.9059     1.0000
Brennan and Prediger |  0.1636    0.6781   0.24   0.811    -1.0000     1.0000
Cohen/Conger's Kappa |  0.1380    0.0322   4.28   0.000     0.0719     0.2041
    Scott/Fleiss' Pi |  0.0721    0.0449   1.61   0.120    -0.0200     0.1641
           Gwet's AC |  0.2278    0.6366   0.36   0.723    -1.0000     1.0000
Krippendorff's Alpha |  0.0887    0.0449   1.98   0.058    -0.0034     0.1807
------------------------------------------------------------------------------
Confidence intervals are clipped at the lower and upper limits.

. kappaetc objective_running subjective_running, icc(mixed)

Interrater reliability                           Number of subjects =      28
Two-way mixed-effects model                     Ratings per subject =       2
------------------------------------------------------------------------------
               |   Coef.     F     df1     df2      P>F   [95% Conf. Interval]
---------------+--------------------------------------------------------------
      ICC(3,1) |  0.1609   1.38    27.00   27.00   0.202    0.0000     0.4987
---------------+--------------------------------------------------------------
       sigma_s | 42.1905
       sigma_e | 96.3518
------------------------------------------------------------------------------
Confidence interval is clipped at the lower limit.

. spearman objective_running subjective_running , star(.05)

 Number of obs =      28
Spearman's rho =       0.4488

Test of Ho: objective_running and subjective_running are independent
    Prob > |t| =       0.0166

[daniel klein]

Spearman's rho measures association; kappa and ICC measure agreement; these are two very different concepts.

From a more substantive perspective, I think your Spearman's rho is pretty low, given that the accelerometer and questionnaire measure the very same thing in the very same units. We might be able to say more if you show summary statistics and a scatter plot of the two variables.

[Joe Tuckles]

Thanks that makes sense! Here are some summary statistics and scatter plot

. sum objective_running, detail

                           objective_running
-------------------------------------------------------------
      Percentiles      Smallest
 1%     1.714286       1.714286
 5%            6              6
10%     7.428571       7.428571       Obs                  28
25%     13.85714       8.285714       Sum of Wgt.          28

50%     30.57143                      Mean           36.39286
                        Largest       Std. Dev.      28.56817
75%         52.5             81
90%     83.85715       83.85715       Variance       816.1401
95%     88.14286       88.14286       Skewness       .8302545
99%     101.4286       101.4286       Kurtosis        2.53195

. sum subjective_running, detail

                         subjective_running
-------------------------------------------------------------
      Percentiles      Smallest
 1%            1              1
 5%         4.29           4.29
10%     17.14286       17.14286       Obs                  28
25%     32.14286             20       Sum of Wgt.          28

50%           60                      Mean            95.8474
                        Largest       Std. Dev.      145.9838
75%     100.7143            190
90%          210            210       Variance       21311.27
95%       214.29         214.29       Skewness       3.872715
99%          780            780       Kurtosis       18.59636

. kappaetc objective_running subjective_running, loa

. twoway (scatter objective_running subjective_running)