kappa2: storing Kappa scores from a series of tests

[Michael McCulloch]

Hello,
I will be running 10 iterations of Kappa2, on a list of 10 different variables, with data from 11 raters, using the command structure:

kappa2 r1-r11, jack

Since this experiment will be performed repeatedly in a workplace context, I wish to capture and store the results so they can be displayed efficiently as one table. Is that possible?

Additionally, how would one interpret the 95% CI on the following kappa2 result:

Po          Pe          K        PJ(K)        SE(K)     [95% Conf Interval]
.6038159   .4726951   .2486623   .2740824   .1150249   .0088346  .5393302

[daniel klein]

kappa2 (probably from SSC, as you are asked to explain) seems to store its results in r(). Maybe even more useful, with the jackknife option it seems the table is stored in r(results).

It is not completely clear from your description what exactly you are doing, because variables (in the Stata sense) record the ratings from each of the 11 raters. We cannot know how the variables (your sense of the word, probably indicating subjects in kappa's terminology) are identified so suggesting specific code is tricky.

Concerning the CIs, they are interpreted the usual way: 95 out of 100 confidence intervals are assumed to contain the "true" kappa value, so we could be be confident that the "true" value lies in the range between 0.00 and 0.59. There are different way to construct confidence intervals for the kappa statistic; besides the jackknife replication method you could use bootstrapping (implemented in kapci, SJ; estimates Fleiss kappa, not Cohen's for more than two raters) or a design-based approach implemented in kappaetc (SSC).

Best
Daniel

[Michael McCulloch]

Thank you Daniel. Yes, SSC was the source of kappa2. M

y dataset contains patient observations in rows (variable Patient identifies ID #, numbering 1-9). And there are 11 variables providing rater scoring, numbering r1 through r11.
Invoking

list

returns the attached screenshot table, which is in -wide- format.
When I invoke

return list

the following is displayed:

scalars:
    r(kappa)    =    .2486623375343656
    r(prop_o)    =    .6038159371492705
    r(prop_e)    =    .4726950575716007
    r(N)    =    9

[daniel klein]

Better to post examples using dataex (see FAQs), which is nowadays shipped with Stata. Sorry, I am still having trouble to understand where the

10 iterations of Kappa2, on a list of 10 different variables

come in. I can see 9 observations and 11 variables.

You have found the values in r(). You can store those values in matrices or variables or even export them to some other software. To be more specific, I would still need more details of what exactly you want to achieve. What do you have in mind when you ask for the results to

be displayed efficiently as one table

Do you want this in Stata? In the results window? In a log file? In another format?

Best
Daniel.

[Michael McCulloch]

Thank you for reframing the question, as I understand better what is needed. I wish to combine in one Stata table, from a series of repeated experiments of subsets of the data, the stored values of Kappa, prop_o and prop_e, in this manner:

TEST Kappa prop_o prop_e
Subset 1
Subset 2
Subset 3...
Subset Z

Once I get them in Stata, I'll know how to export to other formats.

[daniel klein]

Sorry Michael, still not precise enough for customized syntax. What is a "Stata table"? Do you mean a matrix? Are "TEST Kappa prop_o prop_e" supposed to be column names? Are these supposed to be variables in a (new?) dataset? How are the subsets 1, 2, ..., Z identified?

You are probably looking for some variation on

// setup a Z by 3 matrix
matrix K = J(Z, 3, .)
matrix colnames K = Kappa prop_o prop_e

// loop thru subsets
local count = 0
foreach | forvalues x ... {
    local count = `count' + 1
    kappa2 ... if subset == x
    matrix K[`count', 1] = r(kappa)
    matrix K[`count', 2] = r(prop_o)
    matrix K[`count', 3] = r(prop_e)
    // build rownames on-the-fly
    local rownames `rownames' subset`count'
}

// attach rownames
matrix rownames K = `rownames'

// look at the matrix
matlist K

where you obviously need to select one of foreach or forvalues and replace all elements that are typed in italics.
Best
Daniel

[Michael McCulloch]

Ah yes, I meant to say a Stata dataset that's been separately saved. I will first have a go at this, and let you know my progress.

[Michael McCulloch]

Well, so far, I can see that in the first section "// setup a Z by 3 matrix" the first command

matrix K = J(3, 3, 11)

would specify the creation of three matrices which each have three columns send 11 rows.

In the second section "// loop thru subsets"
The first line

local count = 0

Says start with an empty matrix, which will be populated in the command lines that follow.

I'm not sure what the "x" in the next line calls for:

foreach | forvalues x ... {

Subsets I've identified them now as 1, 2 and 3.

What I'm intending is for creation of a separately saved Stata dataset.