AUC and ordinal tests

[Javier Roberti]

I want to validate a predictive model of overall survival of prostate cancer in our sample of cancer patients. This test yields a result in 1 of 3 ordinal categories (low, intermediate or high probability of surviving). I think the correct approach would be a ROC and AUC against survival in my sample. Now, should I take the survival time and divide it into 3 on account of the 3 categories of the model? Is this correct way? If it were a 2 category model (high risk vs low risk), should I take the OS median to divide OS? My second question is how do I work with a patient who for example survived 6 months but was lost to follow-up? Thanks so much!

[Clyde Schechter]

With a three-level result, you cannot do an ordinary ROC analysis. And the losses to follow-up add another complication. Fortunately, both of these problems are easily resolved by doing a Cox proportional hazards model, with the predictive index as your predictor variable, and then run -estat concordance- after that. Be sure to familiarize yourself with the -stset- command (-help stset-) if you don't already know it. You will need it to tell -stcox- which patients were lost to follow-up and which weren't. The output of -estat concordance- will be the Harrell C statistic, which is the survival analysis analog of the area under the ROC curve.

If you need more specific help with coding, be sure to give example data when you post back. And example data should always be shown by using the -dataex- command. If you are running version 17, 16 or a fully updated version 15.1 or 14.2, -dataex- is already part of your official Stata installation. If not, run -ssc install dataex- to get it. Either way, run -help dataex- to read the simple instructions for using it. -dataex- will save you time; it is easier and quicker than typing out tables. It includes complete information about aspects of the data that are often critical to answering your question but cannot be seen from tabular displays or screenshots. It also makes it possible for those who want to help you to create a faithful representation of your example to try out their code, which in turn makes it more likely that their answer will actually work in your data.

When asking for help with code, always show example data. When showing example data, always use -dataex-.

[Javier Roberti]

Thanks so much for your reply. I really appreciate it. Your answer has been most helpful!!
For the ROC curve is the same method than for any model?

* Example generated by -dataex-. To install: ssc install dataex
clear
input long(modelo2 modelo3) float survival byte vivo1muerto0
2 3  28.60274 1
1 1  5.424657 0
2 2  9.041096 0
2 3 18.936987 0
1 2  5.753425 0
1 1  6.706849 0
1 1  2.728767 0
2 2 16.734247 1
2 2  4.964384 1
2 3  14.20274 0
2 3  7.463014 0
1 1   7.29863 0
2 3 12.887671 0
1 1 1.9068493 0
2 3 12.558904 0
1 1  5.063014 0
2 3 13.676712 0
2 3 28.273973 0
1 1  8.284931 0
2 3  4.931507 1
2 3 13.676712 1
2 3  7.364384 0
1 1  7.364384 1
2 3  9.073973 0
2 3  7.594521 0
1 1  4.964384 0
2 3  7.660274 1
1 2 12.920547 0
1 1  6.509589 0
1 1  6.443836 0
1 2 .06575342 0
2 3  9.665753 0
2 3  17.39178 0
1 1  4.109589 0
2 2  2.169863 1
2 3 4.2739725 0
1 2  2.432877 0
2 3 11.934247 0
2 3 16.832876 1
1 1  5.884932 1
1 2  .2630137 0
2 3  25.31507 1
2 3 14.728767 0
2 2 11.243835 0
2 3 2.1369863 1
1 2 3.4849315 0
2 3  6.706849 0
2 3 14.334247 1
2 3  9.863013 1
2 3  5.654795 1
1 1 2.0712328 1
1 1   2.30137 0
1 2  14.00548 0
2 2  8.021917 1
1 1   7.10137 0
2 3 28.832876 1
1 1  3.583562 0
1 1  5.687671 1
2 3  22.29041 1
2 3 26.071234 0
1 1 3.0246575 1
end
label values modelo2 modelo2
label def modelo2 1 "High", modify
label def modelo2 2 "Low", modify
label values modelo3 modelo3v
label def modelo3v 1 "High", modify
label def modelo3v 2 "Intermediate", modify
label def modelo3v 3 "Low", modify

[Clyde Schechter]

I'm confused now. You said in #1 that your models predict probabilities of surviving. But for both models 2 and 3, in the example data the survival times are shorter for people with a High result than for with a Low result. So it seems that your model actually predicts probabilities of dying.

Here's how I would do these analyses

gen byte died = !vivo1muerto0

stset survival, failure(died)

stcox i.modelo2
estat concordance

stcox i.modelo3
estat concordance

If you want to do an ROC curve for model 2 (you can't do it for model 3), you can do that, but to get results that make sense you have to turn around the coding. You have coded the results of model 2 with 1 = High and 2 = Low, but the ROC programs expect it the other way around. So:

//  IF YOU WANT AN ROC CURVE FOR MODELO 2:
recode modelo2 (2 = 1) (1 = 2), gen(modelo2_rev)

roctab died modelo2_rev

[Leandro Blas]

Thanks Clyde for the valuable answers (I am working with Javier Roberti).
I tried to calculate the ROC using that data in Stata version 14.2. The line roctab died modelo2_rev yields a AUC ( and not the curve). How may I do the curve?
Using that data, C-Harrel for modelo2 is 0.73 but ROC is 0.64.Shouldn`t the results be equals?
Thanks again in advance

[Clyde Schechter]

To graph the ROC, add the -graph- option to the -roctab- command. -roctab died modelo2_rev, graph-.

No, the Harrell C is an analog of the ROC area, but it is not the same thing. First of all, the Harrell C is based on the survival time, so it is a more fine-grained measure, whereas the ROC area is based simply on whether the person lived or died.

When I say the Harrell C is the analog of the ROC area, I mean this. One interpretation of the ROC area is this: if I randomly pick two people from the relevant population, one of whom lives and the other of whom dies, the ROC area equals the probability that the one who died will be the one for which the model predicts a higher probability of death. This is sometimes called the two-point forced-choice probability. The Harrell C statistic is similar in that it is equal to the probability that if I pick any two people at random from the relevant population, the one who has the longer survival time predicted by the model is the one who actually lives longer. Both statistics give you the probability that the model makes the right ordinal prediction of the outcome from a random pair.

Added: I misspoke when I said that you could not do an ROC area for a model with a three-level prediction. You can, and the command is just the same, replacing modelo2 by modelo3 (again, I would recode modelo3 so that it gives the probability of death rather than the probablity of not dying.) But the Harrell C statistic is a much more refined measure of the model's discrimination because it can distinguish longer from shorter survival times, not just whether the person died while under observation.

[Leandro Blas]

Thanks again Clyde for the valuable answers.
The resulting ROC is based on very few point, is it possible to soft it? )(attached)
The Harrel C output is without IC95%, I cannot use the "somersd" command because it does not run in Stata 14.2. Just to confirm, the Harrel C doesn`t have a graph associated?

Thanks again in advance.

[Clyde Schechter]

Any ROC curve is based on a number of points equal to the number of distinct values of the predictor. For modelo2 and modelo3 those are 2 and 3 respectively. There is no way out of that. Of course you could capture the coordinates of those points and fit some kind of smooth curve through there, but that would just be covering up the problem, not solving it in any way.

No, there are no graphs associated with Harrell's C as far as I know.

[Leandro Blas]

Dear Clyde,
Thank you for your answer. I have a doubt in how inform differences between models. When I use stat concordance command the resulting C-Harrel is different than the invhr when I use lincom. So 2 C-harrel that are very similar can get a p<0.05 even when CI interval overlap. The dataex only left me include 100 items so the following is not complete.
Is it ok to compare the C Harrel between two models in different number of patients using the lincom command?
In this extract the C-Harrels using stat concordance are 0.634 (CI 95% 0.594- 0.674) and C-Harrel 0.610 (CI 95% 0.545-0.676) for the two models, but when compare the invhr of the fisrt model changes to 0.52, the second remains the same. Which one should I inform? I write the line and commands.

* Example generated by -dataex-. For more info, type help dataex
clear
input int A str12 JUOGriskcategory byte C double survivalmonths byte(Death0 F) int G str12 H byte I double J byte K
246 "Intermediate" 1  .5333333333333333 1 . 551 "Intermediate" 1               51.5 1
188 "Favorable"    0  33.43333333333333 0 . 107 "Intermediate" 1  69.33333333333333 1
343 "Favorable"    0 17.766666666666666 0 . 395 "Intermediate" 1               62.5 1
571 "Favorable"    0 1.1333333333333333 1 . 306 "Poor"         2               15.7 1
270 "Intermediate" 1  62.13333333333333 0 . 280 "Intermediate" 1 3.8333333333333335 0
556 "Intermediate" 1 31.633333333333333 1 . 271 "Poor"         2 23.866666666666667 0
212 "Intermediate" 1                8.9 0 . 254 "Intermediate" 1  3.533333333333333 1
197 "Favorable"    0               46.6 0 . 289 "Intermediate" 1 31.966666666666665 1
485 "Poor"         2 3.2666666666666666 0 . 457 "Intermediate" 1  70.33333333333333 1
152 "Poor"         2  6.233333333333333 0 . 451 "Intermediate" 1  70.23333333333333 1
331 "Intermediate" 1 35.266666666666666 0 . 537 "Favorable"    0                 44 0
337 "Intermediate" 1 37.833333333333336 0 . 389 "Intermediate" 1  37.36666666666667 0
220 "Favorable"    0               39.1 0 . 321 "Poor"         2 10.766666666666668 0
413 "Poor"         2  9.066666666666666 0 . 292 "Intermediate" 1 20.233333333333334 0
501 "Favorable"    0               70.2 0 . 286 "Intermediate" 1  33.86666666666667 0
 40 "Poor"         2 1.5666666666666667 0 . 125 "Intermediate" 1 26.566666666666666 0
 23 "Poor"         2 28.233333333333334 0 . 177 "Favorable"    0               69.9 1
253 "Intermediate" 1  65.03333333333333 0 . 473 "Poor"         2  51.06666666666667 0
103 "Poor"         2  37.46666666666667 0 .  96 "Poor"         2                6.4 0
211 "Intermediate" 1               40.7 0 . 311 "Favorable"    0                5.8 0
196 "Favorable"    0  94.46666666666667 0 . 503 "Poor"         2                5.2 0
309 "Favorable"    0              105.3 1 .  90 "Intermediate" 1  56.96666666666667 0
273 "Intermediate" 1 62.733333333333334 0 . 575 "Poor"         2                 30 1
138 "Poor"         2               16.5 0 .  63 "Intermediate" 1  69.16666666666667 1
 43 "Poor"         2  3.533333333333333 0 . 589 "Intermediate" 1 23.666666666666668 0
202 "Favorable"    0 122.03333333333333 1 . 439 "Intermediate" 1 61.833333333333336 1
 19 "Intermediate" 1 20.866666666666667 0 .  76 "Intermediate" 1  68.73333333333333 1
178 "Favorable"    0               88.3 0 . 364 "Intermediate" 1               18.9 0
 36 "Poor"         2               13.6 0 . 340 "Favorable"    0  65.06666666666666 1
283 "Intermediate" 1              124.4 1 . 275 "Poor"         2                8.7 0
588 "Poor"         2 16.066666666666666 0 . 500 "Poor"         2                4.5 0
109 "Intermediate" 1               26.3 0 . 258 "Intermediate" 1               15.8 0
384 "Intermediate" 1              116.4 1 . 351 "Intermediate" 1  49.56666666666667 0
250 "Poor"         2              117.4 0 . 409 "Favorable"    0 59.333333333333336 1
505 "Poor"         2 40.266666666666666 0 . 230 "Intermediate" 1  40.43333333333333 0
227 "Favorable"    0  6.533333333333333 0 . 526 "Favorable"    0 45.733333333333334 1
158 "Poor"         2  7.533333333333333 0 . 382 "Poor"         2  61.43333333333333 1
 14 "Poor"         2               17.3 0 . 133 "Poor"         2 31.133333333333333 0
120 "Intermediate" 1  40.46666666666667 0 . 456 "Intermediate" 1  58.06666666666667 1
319 "Intermediate" 1  7.833333333333333 0 . 374 "Poor"         2               53.9 0
221 "Poor"         2 54.266666666666666 0 .  70 "Poor"         2               65.1 0
345 "Intermediate" 1               63.2 1 .  97 "Poor"         2 19.333333333333332 0
515 "Intermediate" 1               52.4 0 . 423 "Intermediate" 1 21.533333333333335 0
132 "Poor"         2 21.066666666666666 0 .  10 "Favorable"    0 60.266666666666666 1
299 "Intermediate" 1  52.56666666666667 0 . 304 "Intermediate" 1 29.233333333333334 1
 38 "Intermediate" 1               32.2 0 . 347 "Intermediate" 1 61.333333333333336 1
437 "Intermediate" 1 15.466666666666667 0 . 406 "Intermediate" 1               57.5 1
430 "Favorable"    0 31.666666666666668 0 . 344 "Poor"         2               60.4 1
228 "Intermediate" 1               21.6 0 . 236 "Poor"         2  37.06666666666667 0
498 "Poor"         2                 20 0 . 181 "Intermediate" 1  61.43333333333333 1
113 "Intermediate" 1  32.43333333333333 0 . 520 "Favorable"    0               25.3 0
 35 "Intermediate" 1  7.933333333333334 0 .  64 "Intermediate" 1 18.433333333333334 0
166 "Favorable"    0  45.03333333333333 1 . 206 "Favorable"    0                 43 1
216 "Poor"         2 120.96666666666667 1 . 507 "Poor"         2 54.733333333333334 1
338 "Intermediate" 1 24.833333333333332 0 . 302 "Intermediate" 1  33.56666666666667 1
185 "Favorable"    0               47.4 0 . 335 "Favorable"    0               24.2 0
208 "Intermediate" 1 49.766666666666666 0 .  11 "Intermediate" 1               45.2 0
480 "Intermediate" 1 24.566666666666666 0 . 308 "Intermediate" 1  57.06666666666667 1
359 "Favorable"    0              120.6 1 . 376 "Favorable"    0               52.3 1
182 "Favorable"    0               85.7 0 . 356 "Intermediate" 1               54.3 1
248 "Favorable"    0  81.93333333333334 1 . 329 "Favorable"    0 55.833333333333336 1
225 "Intermediate" 1  5.833333333333333 0 . 398 "Intermediate" 1 23.366666666666667 0
493 "Poor"         2  9.333333333333334 0 .  67 "Favorable"    0  58.36666666666667 1
531 "Poor"         2               43.6 0 . 534 "Favorable"    0               54.4 0
326 "Poor"         2  84.56666666666666 0 . 215 "Intermediate" 1 12.466666666666667 0
257 "Intermediate" 1               31.9 0 . 497 "Poor"         2 3.6333333333333333 0
315 "Poor"         2               28.1 0 . 167 "Favorable"    0 54.333333333333336 1
267 "Favorable"    0  57.43333333333333 0 . 458 "Intermediate" 1  50.53333333333333 1
222 "Favorable"    0              117.9 1 . 400 "Intermediate" 1 51.733333333333334 1
126 "Intermediate" 1  45.43333333333333 0 .  82 "Poor"         2  56.06666666666667 1
207 "Favorable"    0 22.233333333333334 1 . 447 "Intermediate" 1 47.266666666666666 1
516 "Poor"         2               15.7 0 . 524 "Favorable"    0               53.1 1
100 "Favorable"    0  66.53333333333333 0 . 131 "Favorable"    0  53.63333333333333 1
154 "Poor"         2               54.7 0 . 284 "Intermediate" 1               52.8 1
405 "Favorable"    0 110.36666666666666 1 . 349 "Intermediate" 1                2.3 1
114 "Poor"         2 25.966666666666665 0 . 110 "Poor"         2  35.46666666666667 0
193 "Intermediate" 1  64.76666666666667 1 . 475 "Intermediate" 1 33.733333333333334 0
176 "Favorable"    0               40.9 0 . 124 "Intermediate" 1 52.333333333333336 1
523 "Intermediate" 1 37.733333333333334 0 . 307 "Poor"         2 14.033333333333333 0
123 "Poor"         2              112.9 1 . 459 "Poor"         2               24.8 0
117 "Favorable"    0 112.23333333333333 1 .  16 "Intermediate" 1                 51 1
148 "Favorable"    0               14.7 0 . 420 "Poor"         2 17.433333333333334 1
195 "Intermediate" 1               45.5 1 . 144 "Intermediate" 1 17.633333333333333 0
436 "Intermediate" 1 111.26666666666667 1 . 119 "Favorable"    0  50.06666666666667 1
165 "Intermediate" 1  43.06666666666667 0 .  59 "Intermediate" 1 15.966666666666667 0
140 "Intermediate" 1 30.133333333333333 0 . 582 "Intermediate" 1               13.4 0
  3 "Favorable"    0  97.03333333333333 0 . 168 "Favorable"    0                 51 1
155 "Intermediate" 1 110.46666666666667 1 . 290 "Favorable"    0 21.633333333333333 0
444 "Favorable"    0 44.666666666666664 0 . 470 "Intermediate" 1  9.966666666666667 0
169 "Favorable"    0 31.533333333333335 0 . 354 "Intermediate" 1               48.4 1
231 "Intermediate" 1               77.6 0 . 305 "Intermediate" 1 1.4333333333333333 1
303 "Poor"         2  70.43333333333334 0 . 156 "Intermediate" 1               24.3 0
301 "Intermediate" 1               95.7 1 . 102 "Poor"         2               45.5 1
560 "Favorable"    0 109.33333333333333 1 . 494 "Intermediate" 1  41.46666666666667 0
361 "Intermediate" 1 17.566666666666666 0 . 241 "Intermediate" 1 49.733333333333334 1
316 "Intermediate" 1  89.76666666666667 0 .  81 "Intermediate" 1  50.43333333333333 1
452 "Favorable"    0 105.36666666666666 1 . 435 "Favorable"    0 14.333333333333334 0
432 "Intermediate" 1              109.8 1 . 310 "Intermediate" 1  47.96666666666667 1
 12 "Favorable"    0 109.36666666666666 1 . 163 "Favorable"    0               49.6 1
256 "Poor"         2  52.53333333333333 0 . 373 "Favorable"    0               43.9 1
end

stset survivalmonths, failure ( Death0==0)
stcox i.C
estat concordance
predict hr
generate invhr=1/hr
stset J, failure ( K==0)
stcox i.I
estat concordance
predict hr1
generate invhr1=1/hr1
. generate censind=1-_d if _st==1
somersd _t invhr invhr1, cenind(censind) tdist transf(c)
lincom invhr-invhr1

Thanks in advance.