Seeking help on Statistical Test for Concordance Evaluation

[Francesco Ditonno]

Dear Statalist members,

I am currently working on evaluating the concordance between the results of preoperative biopsy and final histology in patients undergoing radical nephroureterectomy. Both variables, biopsy_grade and ptgrade_path, are binary (low grade: 0; high grade: 1).

I am uncertain whether comparing percentages of high-grade preoperative biopsies and high-grade surgical specimens and tabulating them with a chi-squared test is sufficient.

I've tried to calculate the Cohen's kappa coefficient but I understand this is usually applied to inter-reader agreement on the same specimen and not to different specimens examined by different readers.

Here is the command I've used:

. tabulate grade_bio

Grade | Freq. Percent Cum.
------------+-----------------------------------
Low Grade | 283 28.30 28.30
High Grade | 717 71.70 100.00
------------+-----------------------------------
Total | 1,000 100.00

. tabulate ptgrade_path

T Grade | Freq. Percent Cum.
------------+-----------------------------------
Low Grade | 372 19.20 19.20
High Grade | 1,566 80.80 100.00
------------+-----------------------------------
Total | 1,938 100.00

. kap grade_bio ptgrade_path

Expected
Agreement agreement Kappa Std. err. Z Prob>Z
-----------------------------------------------------------------
81.38% 63.34% 0.4921 0.0317 15.54 0.0000

Could you please suggest if this is the correct method and, if not, the correct statistical method to verify the diagnostic concordance between the two methods?

Thank you in advance for your help.

Best regards,

Francesco Ditonno

[Clyde Schechter]

Simply comparing the prevalence of high-grade in each type of specimen is not an adequate way to assess this.

The kappa statistic is used to compare the agreement of any two measurements made on the same specimens. Whether those are two different raters or ratings of the same lesion at different times does not matter. It is an acceptable way of doing this for your data, but just barely so.

The real question here is not a matter of mere agreement between the two. The pre-operative biopsy is, if you will, a "screening test" that is used to perhaps guide surgery and to make decisions about adjuvant treatments. But the operative pathology is the reference criterion (in medicine often referred to as the "gold standard," a term I intensely dislike); it is "truth," or as close as we ever get to truth. So it is really not helpful to simply measure the agreement between the two. You really need to assess the validity of the pre-operative biopsy as a predictor of the operative pathology. So, with the operative pathology as our reference criterion, and taking high grade as the "disease" you really need to assess the sensitivity and specificity of the pre-operative biopsy. That is actually quite simple to do: -tab grade_bio ptgrade_path, col-. The column percentages on the main diagonal will provide these statistics for you. The one in the "high grade" column of the table is the sensitivity, and the one in the "low grade" column gives you the specificity. This kind of validity cannot be assessed in a single number--you need both the sensitivity and specificity for a complete picture.

[Francesco Ditonno]

Dear Clyde Schechter ,

Thank you for the exhaustive response.

I believe that in addition to reporting specificity and sensitivity, including positive predictive value and negative predictive value would contribute to the completeness of the analysis.

Is there a specific command to calculate these values for the preoperative biopsy?

Thank you in advance,
Francesco

[Clyde Schechter]

For the positive and negative predictive values, do -tab grade_bio ptgrade_path, row-. The main diagonal row percentages will give you those. The one in the "high grade" row is the positive predictive value, and the one in the "low grade" row is the negative predictive value.

[Francesco Ditonno]

Thank you very much Clyde Schechter