sample size for kappa

[Renee Kam]

I am familiar with the following codes for estimating sample size for kappa (kapssi, sskdlg and sskapp). However, all of these are limited to binary outcomes. My research involves an ordinal outcome with 3 or 4 possible categories and there are 2 raters. Does anyone know of a code using Stata I could use to estimate sample size for kappa in this scenario please? Thank you!

[Joseph Coveney]

Your scenario is incomplete for sample size estimation, for example, is it your intention to posit a null hypothesis of absolutely no agreement at all between the two raters? What are the prevalences of the ratings in the population of targets? What weighting scheme are you planning to use to weight disagreements?

[Renee Kam]

Thank you for your response Joseph. If using the codes I mentioned in stata to determine the sample size calculation for the kappa-statistic measure of interrater agreement then, I need to include the proportion (prevalence) of ratings in two populations p1 and p2. However, in my research I'm not using a binary variable for two raters to rate on. I have an ordinal variable with 3-4 levels and so recoding the prevalence in only two populations won't work. For example, to use sskapp I need, p1() p2() diff() kapp(). As indicated providing p1() and p(2) doesn't make sense for my research. The diff() is the width of the confidence interval which I plan choosing 0.2 for and for kapp() which is the kappa I want the sample size for I plan on using (0.8). Does this make sense? Thank you

[Joseph Coveney]

I had never heard of those commands that you referred to, don't know anything about them, and it wasn't my intention that you collapse categories and use them. I was considering that if there isn't any canned program available for power estimation or sample size estimation for ordered-categorical kappa, then you could use simulation. And prevalence affects kappa, so for simulation you'd need to consider it.