How can I calculate a kappa statistic for several variables at the same time?

[Guest]

Hello,

I'm having a tiny issue with Stata 12. Is it possible to calculate a kappa statistic for several variables at the same time? I have a scale with 8 labels/variable, évaluated by 2 raters. How do i go from there to know the total agreement rate and the agreement rate per domain?

thank you!
Y

[Steve Samuels]

This is a poorly posed problem-what do you mean by "total agreement" and "agreement per domain"?

More than that, "cute" names are frowned upon in Statalist; the long-time etiquette is to use full-real names (see the FAQ). Over the years this practice has promoted professionalism, mutual respect, and friendship on the list. I urge you to change your user-name to your real name, first and last. You can request this change via the "Contact Us" button at the bottom right of the page.

[Guest]

Good morning,

Thank you for the nice and warm welcome. One thing to be sure, I wasn't expecting that kind of message. The "friendship" part in your heart warming statement dosent seem that obvious. I will request a deregistration and not only a change of screen name.

Y.Yordanov

[Carlo Lazzaro]

Dear Y.Yordanov,
I'm afraid that you take Steve's right advice too personally. He was only pointing you out to one of the most reminded FAQ contents, that you were assumed to read before joinning the Forum.
Besides, I would pay more attention at the first part of his reply: please, pose queries effectively, trying your best to put yourself from readers' standpoint, who often cannot know what you're after unless you describe it clearly. That's why, as per FAQ again, posters are requested to report exactly what they typed and what Stata gave them back.
About your seeming complaint concerning a supposed lack of friendship across the list, obviously speaking for myself I can say that after an 8-year experience I still find lots of helpful advices from other more experienced listers and I am every day positively impressed that all guidances are issued by busy people who devoted their time to the list asking nothing in return.
That said, I would take the liberty to ask you to reconsider your decision to resign from the list if it is driven by a gut-feeling only.

Kind regards,
Carlo

[Steve Samuels]

I do apologize for the tone of the message. Please chalk it up to a bad night.

However I didn't know what you meant by "total agreement" and "agreement per domain". None of the Stata commands for kappa (search kappa, all) use those terms, and I searched for almost an hour before I posted and found no reference to "total agreement". A single online calculator claimed to calculate kappa with multiple variables (https://mlnl.net/jg/software/ira/ , but it gave no references. This morning I came up with the article below, which I haven't direct access to. Is this the method you refer to?


De Vries, Han, Marc N. Elliott, David E. Kanouse, and Stephanie S. Teleki. "Using pooled kappa to summarize interrater agreement across many items."
Field Methods August 2008 vol. 20 no. 3 272-282

[Phil Bromiley]

Let me echo that Statalist is an incredibly helpful set of folks if you have the kind of problem the list is set up to handle and you deal with it appropriately. These folks often take time from their days to program and test solutions to the problems of other users. However, you are wasting their time if you pose a question that is ambiguous since they really cannot answer such questions.

There are several procedures available under the kappa documentation (see kappa documentation from Stata manuals). If you don't find what you want there, you can enter findit kappa in Stata. This brings up a large number of user-written procedures related to kappa. So, the first thing to do is to see if any of these will handle your problem. Look at the help files for each of the user written programs - if what you want is not the primary objective of the author, the program may do what you want but not mention it in the summary that appears from findit.

If you can't see how to get what you want with these resources, then do come back to Statalist. Part of the problem is that the terminology you use is not necessarily the same terminology others use. We come from various backgrounds and statistical terminology varies substantially across disciplines. So, try to explain the problem much more specifically. (See FAQ on asking questions which has additional detail on how to pose questions.) Tell us the structure of the data, exactly what agreements among what variables you want to calculate, etc. From your description, I'm not sure if you just want the program to do several statistics at once to save programming, or if there is a statistical reason to do the statistics simultaneously. If the former, then just program it.


Phil

[Steve Samuels]

I too hope that M. Yordanov returns. Here's a program that computes the pooled kappa for multiple variables in the DeVries article mentioned above and that calculates a bootstrapped confidence interval The data is in the format below; i.e. I just repeat the data needed for a single run of kappa. "rada" and "radb" are the ratings for the given variable from raters "a" and "b". Not many data sets will be in this format, so some reshaping will be necessary.

vname id rada radb
x          1   2      3
x          2   4      4
...
y          2    1     1
y          2    3     2
....

capture program drop _all
cap macro drop _all
cap scalar drop _all
cap matrix drop _all

webuse rate2,clear
sum
/* Arrange Data with a separate data set for each variable */

gen id = _n  /* subject ID */
gen vname = "x"
tempfile t1
save `t1'
/* Create slightly different data for second variable */
replace rada = 2 in 1
replace rada = 4 in 1
replace radb = 4 in 2
replace vname = "y"
append using `t1'

tab vname
sum

program kpool, rclass
    qui levelsof vname, local(vlist)
    local nvars: word count `vlist'
    tempname m mx
    foreach v of local vlist {
      qui kap rada radb if vname=="`v'"
      matrix `mx'= (r(prop_o), r(prop_e))
      matrix `m' = (nullmat(`m') \ `mx')
      cap matrix drop `mx'
    }

    mata: nv =strtoreal(st_local("nvars"))
    mata: one = J(nv,1,1)
    mata: m = st_matrix("`m'")
    mata: num = m[.,1]- m[.,2]
    mata: denom = one -m[.,2]
    mata: pkap = sum(num)/sum(denom)
    mata: st_numscalar("pkap",pkap)
    return scalar pkap= pkap
end

/* Try single run */
kpool

/* Now bootstrap */
bootstrap pkap= r(pkap), cluster(id) reps(1000): kpool
estat bootstrap

[Steve Samuels]

I should have noted that the pooled kappa is the sum of the numerators from the individual kappas divided by the sum of the denominators. This summary was more precise in the DeVries et al. simulations than the arithmetic average of the individual kappas.

[Joseph Coveney]

Steve,

Does the Devries et al. method generalize to weighted kappa? I couldn't tell from the description in the original post whether Y. Yordanov is working with nominal choices, as you'd expect with conventional kappa, or is working with ordered-categorical scores. He (or she) asks about kappa, unqualified, which would imply nominal data. But then uses terms like "scale" and "domain", which imply some kind of Likert-like sum-score of a multi-item questionnaire whose responses are ordered categories.

I, too, found the request ambiguous and difficult to respond to, and I wonder whether the Devries et al. method works with both weighted as well as unweighted kappa. (I don't have access to the article and so cannot find out for myself.)

[Steve Samuels]

Joseph,

The paper itself was confined to 2 x 2 tables and says nothing about other dimensions. The simulation was confined to generating tables for 100 variables from each simulated configuration-so that the true kappa was identical for all variables. That said, with weights for > 2 categories, the kappa command generates weighted observed and expected proportions. There's no practical barrier, therefore, to estimating the pooled summary for weighted kappa.

It is interesting to note that this pooled summary is equivalent to a weighted average of the variable-specific kappa values. In the i-th table, with

$$
\kappa_i= \frac{ p_{o,i}-p_{e,i}}{1-p_{e,i}},
$$
The pooled summary
$$
\kappa_p= \frac{ \sum (p_{o,i}-p_{e,i})}{\sum (1-p_{e,i})} \\
$$
can be viewed as a weighted average of the individual \(\kappa_i\):

$$
\kappa_p= \sum w_i \kappa_i
$$

where \(w_i\) is proportional to \(1-p_{e,i}\). Thus variables with the least chance agreement are weighted most heavily.

[Roman Mostazir]

I apologize for not making a comment directly relevant to the post but yet not far from relevance given the conversations above. I have been a stalker of this forum since quite a while by now. It is just the recent move of this forum when I thought to engage a bit more with the experts here who are providing the ''cutting age'' solutions, thus I registered. I am enormously indebted to the contributors here. And many thanks to ........ too many you are to name. Since this forum started, two things that have repeatedly been implored by these contributors are: i) Registration with the full name ii) Reading the FAQ on how to post. Now after few months since the new forum has launched, I can sense that their voice is gradually dimming due to frequent pseudo registrations. Some giving it up by ignoring it and some still continuing with apparent frustration. No offense, but both of these situations lead deviation to the believed standard of this forum. On the other hand, the repeated requests to follow the instructions are not working for new users. Perhaps, we (considering myself a part of this forum) need to shift the paradigm of thinking. Perhaps it is the time to ask ourselves why most of the newcomers are missing this important etiquette. Is there anything in the home page we can do that will ''eye catch'' the newcomers and will help the process. I do not wish to through a dice on the floor and ask you to pick it up without providing any efforts of mine. Others may have different opinion and with due respect to those, two suggestions from my part to handle this problem: i) The FAQ in a tab is well styled with concurrent layouts but for this particular problem, this is not serving much. Could we put something instructional in the home page in sticky form: i.e. "Please read the FAQ before you post" / " It is important that you read FAQ before you post" type sticky ii) INSIDE the registration page "Please note we encourage everyone to register with their full name that will help us in the long run to build trust and sound professional relationship" sort of statement. It is evident that we need to do something. If we don't, we better stay away from the believed standard of this forum.

Best,

[Steve Samuels]

I've a correction to the artificial data layout for the post that calculated the pooled kappa. I mistakenly omitted id 1 for variable "y" and repeated id 2. The layout should have been:

vname id rada radb
x 1 2 3
x 2 4 4
...
y 1 1 1
y 2 3 2
....

[Nick Cox]

Roman: In #11 you raise several key points. Ad although it's clear why you posted in this thread the issues belong in "Using the forum". I'll copy the bulk of your post there and make a few comments.

[ashar ata]

Steve-
Would the pooled kappa program apply to the following scenario.

I have two raters who were to score each patient on 9 different criteria. There were a total of 150 patients.
We made both the raters score the same patient on all 9 criteria for 15 patients.

The remaining 135 patients were randomly scored by only one of the raters.

We want to use the 15 that were scored by both the raters to estimate kappa. The 9 different criteria range from yes/no to counts to estimation of time elapsed.

Similar to creating 2 categories for -vname- If I created categories for -vname- by appending the 9 criteria one below the other would the results of kpool represent a valid pooled kappa?
Thanks

Ashar

Albany Medical College
Albany, NY

[Steve Samuels]

The kappas in the pooled formula are for 2x2 agreement tables, not for counts or continuous measures. So adding values for such variables will not produce valid agreement measures.