Incorrect P-value in Area under the ROC curve comparison instruction

[Sara Tous]

Hello,
I have been using the roccomp (or rocgold) instruction to compare two AUCROC but I cannot trust the result I get, specially in the following example (I show you the output). I have done the same analysis using Epidat software and the conclusion is that there are no differences between the 2 curves (Chi2=0.4241; df=1; P=0.515) and I am more confident with this result.
Do you think there is any problem with STATA calculations?
Thank you very much.
Sara

. roccomp hpv_mrna hpv_p16 p16_tota, graph summary

ROC -Asymptotic Normal--
Obs Area Std. Err. [95% Conf. Interval]

hpv_p16 786 0.9328 0.0210 0.89170 0.97397
p16_tota 786 0.9134 0.0213 0.87163 0.95510

Ho: area(hpv_p16) = area(p16_tota)
chi2(1) = 29.09 Prob>chi2 = 0.0000

. rocgold hpv_mrna hpv_p16 p16_tota, graph summary


ROC Bonferroni
Area Std. Err. chi2 df Pr>chi2 Pr>chi2

hpv_p16 (standard) 0.9328 0.0210
p16_tota 0.9134 0.0213 29.0941 1 0.0000 0.0000

note: 2 observations ignored because of missing values.

[Joseph Coveney]

Do you think there is any problem with STATA calculations?

Anything is possible, but in my experience it's been an alternative explanation. You wouldn't happen to be able to upload the dataset as an attachment to another post by any chance, would you? (Change its file extension from .dta to .txt, if the forum software objects to a Stata dataset file format.)

[Steve Samuels]

Your tables are hard to read because columns don't line up. Please read FAQ 12 and in future posts show commands and results between code delimiters.

The reason for the difference you observe is simple: roccgold considers the first modality, hpv_p16, to be a known gold standard. The ROC for the gold standard is assume to be known exactly, whereas the other ROC(s) are estimates. With roccomp, the ROC for hpv_p16 is not considered to be known in advance: it is an estimate. Thus the p-value for comparison to the other ROCs will have a larger p-value.


Compare Examples 1 and 3 in the manual entry for roccomp. In Example 1, the roccomp p-value for comparing mod3 to mod1 is .0642. In Example 3,, mod1 is considered to be a gold standard and the rocgold p-value for comparing mod3 is .0241