[Joe Tuckles]

So to clarify... does this mean I cannot do any tests on my data? Cannot I do a test to see whether there is agreement between these measures or if they are significantly different? Can I not do any regression to see if variables predict these risk scores? Nothing??

[Nick Cox]

The only thing ruled out in this thread that I can recall -- sorry, but I won't re-read all posts -- is working descriptively on logit scale -- because you have some zeros and that's fatal.

What you (we) need to rule in is: What kind of "tests" do you think make sense? What is your model for the "data"? More crucially, what are your goals?

More simply, you can always measure agreement with measures like concordance correlation, as already pointed out.

As you're now revealing the risk measures come from somebody else's risk "algorithms". Precise advice is really hard without knowing what those "algorithms" are. Algorithms is your word, but in epidemiology or biostatistics (neither is my field) the recipes I've read about usually aren't very complicated.

You can consider trying to build regressions but I don't recall your mentioning possible predictors.

Frankly, you seem to be way out of your depth. If you're a student, then use whatever support is available to you at your institution. If you're a medic, then you need to consult a biostatistician or epidemiologist. There are other possibilities too.

[Joe Tuckles]

Ok. I think I just need to deal with one problem at a time. My first test that I plan to use is measure agreement between measures. I wish to use the concordance correlation for this. But now it seems I am unable to work on a logit scale due to the zeros. Is there any way of moving forward with this? Any other way I can use concordance correlation without working on a logit scale?

[Joe Tuckles]

I have spoke with a colleague and they said the two zeros generated in the first measure should actually be missing data. Not zero. Therefore, can I now use the logit scale?

[Joe Tuckles]

On #26: Stata is telling you, as I read it, that it doesn't know of a variable or scalar called prevl1. We can't comment beyond asking whether it's true. From #10 and #17 it would seem that your variables are called prev1 to prev4.

Yes exactly I don't have a variable called prev11. And as you can see from my code, I wrote

prev1`i' / 100

. I am unable to see the error in the code. Can you advise why stata thinks I have wrote prev11 instead of prev1`i'?

Thanks

[Nick Cox]

So, first time around the loop Stata substitutes 1 (one) for i (letter i) and the reference becomes

prev11 / 100

It seems that you want Stata to loop over prev1 to prev4. If so, prev is the prefix (stub) not prev1

If zero is really missing (that still seems suspect!) then you should recode to missing or drop the corresponding observations as useless. Note that logit() would also choke on 1: no instances in your example data but the principle is important.

[Joe Tuckles]

Hooray! Thank you :-) Here is what I have now ( I shall recode the missing data):

* Example generated by -dataex-. To install: ssc install dataex
clear
input float(risk_1 risk_logit_1 risk_2 risk_logit_2 risk_3 risk_logit_3 risk_4 risk_logit_4)
.02967858  -3.487202  .015984064   -4.12005  .008863377 -4.7169247  .014922163  -4.189873
.14070985 -1.8094066   .04402651  -3.077938  .011049236  -4.494283   .01942504  -3.921576
 .0914358 -2.2962284           .          .           .          .   .03949243  -3.191353
 .1360417  -1.848563   .04816825  -2.983688  .027212884  -3.576475  .064056836  -2.681784
        .          .  .008983527 -4.7033386  .001496917   -6.50285 .0018035504  -6.316193
.04482531 -3.0591214  .016181076 -4.1075993  .005019374  -5.289418  .016671507 -4.0772424
.00976722  -4.618908  .016757084 -4.0720353   .00754011   -4.87995  .012927463 -4.3353896
.14454609 -1.7780337   .07827424  -2.466029   .04156326  -3.138087   .04874776   -2.97112
 .3845862  -.4701269   .15632953  -1.685796   .06192381 -2.7179265   .06048964  -2.742887
.01049809 -4.5460086  .009204037 -4.6788664 .0038537504  -5.554847  .008022561  -4.817443
.02045249  -3.868986  .017633067  -4.020189  .007400888  -4.898727  .015751708  -4.134929
 .1603237 -1.6558214  .031970017 -3.4104645   .04738522    -3.0009   .04695547  -3.010462
.20577964  -1.350555   .11138762  -2.076645   .08531375 -2.3722456   .07435046  -2.521706
.01178254  -4.429284 .0020282855  -6.198534  .002420856   -6.02121  .006580936  -5.016975
.01436456  -4.228523  .016167369  -4.108461 .0022165442  -6.109587 .0031526214  -5.756363
.22599764 -1.2310504    .1242271 -1.9529954  .031392787  -3.429281   .04037205  -3.168408
.05128489 -2.9177125   .04406955  -3.076916  .011763887  -4.430887  .022554524  -3.769007
 .2008589 -1.3809347   .11428288  -2.047721   .04188088  -3.130143   .04721145 -3.0047565
 .0691531  -2.599772   .04880307  -2.969928   .01971524  -3.906451   .03730975  -3.250477
.06340435  -2.692719  .029332334  -3.499294   .02093821  -3.845019   .03327607  -3.369074
.07027753  -2.582434  .005582108  -5.182591   .01160607  -4.444553  .021429203 -3.8213384
        0          . .0011128157  -6.799748 .0008769826  -7.038146  .001733915  -6.355638
.00799016  -4.821522  .012400947  -4.377504 .0031622166  -5.753315  .006414833  -5.042707
.07922649 -2.4529035   .04228576  -3.120099   .02171835   -3.80764   .04859337 -2.9744544
.00356924  -5.631827  .002425109  -6.019451  .003661532  -5.606205   .00354345  -5.639105
 .1383318  -1.829215  .068722755 -2.6064765    .0247116 -3.6754606   .05645855 -2.8161335
.13870323 -1.8261025   .04157504  -3.137791   .03129016 -3.4326615   .03417343  -3.341536
 .1044165 -2.1490877   .04020252  -3.172793   .02829525  -3.536358   .03761077 -3.2421284
        .          .           .          .           .          .           .          .
        .          .           .          .           .          .           .          .
        .          .           .          .           .          .           .          .
        .          .           .          .           .          .           .          .
        .          .           .          .           .          .           .          .
        .          .           .          .           .          .           .          .
end

[Joe Tuckles]

I have generated the concord analysis using the logit variables and it has produced these results, which i was wondering if you could help me with interpreting:

concord risk_logit_1 risk_logit_2

Concordance correlation coefficient (Lin, 1989, 2000):

 rho_c   SE(rho_c)   Obs    [   95% CI   ]     P        CI type
---------------------------------------------------------------
 0.713     0.083      25     0.550  0.877    0.000   asymptotic
                             0.509  0.842    0.000  z-transform

Pearson's r =  0.854  Pr(r = 0) = 0.000  C_b = rho_c/r =  0.836
Reduced major axis:   Slope =     1.144   Intercept =     1.265

Difference = risk_logit_1 - risk_logit_2

        Difference                 95% Limits Of Agreement
   Average     Std Dev.             (Bland & Altman, 1986)
---------------------------------------------------------------
     0.756       0.703                 -0.621      2.133

Correlation between difference and mean = 0.251

Bradley-Blackwood F = 15.583 (P = 0.00005)

. concord risk_logit_1 risk_logit_3

Concordance correlation coefficient (Lin, 1989, 2000):

 rho_c   SE(rho_c)   Obs    [   95% CI   ]     P        CI type
---------------------------------------------------------------
 0.503     0.085      25     0.336  0.669    0.000   asymptotic
                             0.318  0.650    0.000  z-transform

Pearson's r =  0.901  Pr(r = 0) = 0.000  C_b = rho_c/r =  0.558
Reduced major axis:   Slope =     1.255   Intercept =     2.541

Difference = risk_logit_1 - risk_logit_3

        Difference                 95% Limits Of Agreement
   Average     Std Dev.             (Bland & Altman, 1986)
---------------------------------------------------------------
     1.460       0.601                  0.283      2.637

Correlation between difference and mean = 0.468

Bradley-Blackwood F = 93.869 (P = 0.00000)

. concord risk_logit_1 risk_logit_4

Concordance correlation coefficient (Lin, 1989, 2000):

 rho_c   SE(rho_c)   Obs    [   95% CI   ]     P        CI type
---------------------------------------------------------------
 0.618     0.082      25     0.457  0.779    0.000   asymptotic
                             0.431  0.754    0.000  z-transform

Pearson's r =  0.903  Pr(r = 0) = 0.000  C_b = rho_c/r =  0.684
Reduced major axis:   Slope =     1.447   Intercept =     2.646

Difference = risk_logit_1 - risk_logit_4

        Difference                 95% Limits Of Agreement
   Average     Std Dev.             (Bland & Altman, 1986)
---------------------------------------------------------------
     0.974       0.645                 -0.290      2.238

Correlation between difference and mean = 0.661

Bradley-Blackwood F = 57.423 (P = 0.00000)

. concord risk_logit_2 risk_logit_3

Concordance correlation coefficient (Lin, 1989, 2000):

 rho_c   SE(rho_c)   Obs    [   95% CI   ]     P        CI type
---------------------------------------------------------------
 0.690     0.089      25     0.516  0.864    0.000   asymptotic
                             0.474  0.828    0.000  z-transform

Pearson's r =  0.834  Pr(r = 0) = 0.000  C_b = rho_c/r =  0.828
Reduced major axis:   Slope =     1.097   Intercept =     1.115

Difference = risk_logit_2 - risk_logit_3

        Difference                 95% Limits Of Agreement
   Average     Std Dev.             (Bland & Altman, 1986)
---------------------------------------------------------------
     0.704       0.657                 -0.583      1.991

Correlation between difference and mean = 0.166

Bradley-Blackwood F = 14.486 (P = 0.00008)

. concord risk_logit_2 risk_logit_4

Concordance correlation coefficient (Lin, 1989, 2000):

 rho_c   SE(rho_c)   Obs    [   95% CI   ]     P        CI type
---------------------------------------------------------------
 0.778     0.075      25     0.631  0.925    0.000   asymptotic
                             0.583  0.888    0.000  z-transform

Pearson's r =  0.817  Pr(r = 0) = 0.000  C_b = rho_c/r =  0.952
Reduced major axis:   Slope =     1.264   Intercept =     1.207

Difference = risk_logit_2 - risk_logit_4

        Difference                 95% Limits Of Agreement
   Average     Std Dev.             (Bland & Altman, 1986)
---------------------------------------------------------------
     0.218       0.680                 -1.114      1.550

Correlation between difference and mean = 0.380

Bradley-Blackwood F = 3.377 (P = 0.05177)

. concord risk_logit_3 risk_logit_4

Concordance correlation coefficient (Lin, 1989, 2000):

 rho_c   SE(rho_c)   Obs    [   95% CI   ]     P        CI type
---------------------------------------------------------------
 0.833     0.051      25     0.733  0.933    0.000   asymptotic
                             0.702  0.910    0.000  z-transform

Pearson's r =  0.944  Pr(r = 0) = 0.000  C_b = rho_c/r =  0.883
Reduced major axis:   Slope =     1.152   Intercept =     0.084

Difference = risk_logit_3 - risk_logit_4

        Difference                 95% Limits Of Agreement
   Average     Std Dev.             (Bland & Altman, 1986)
---------------------------------------------------------------
    -0.486       0.364                 -1.199      0.227

Correlation between difference and mean = 0.396

Bradley-Blackwood F = 27.474 (P = 0.00000)

[Weiwen Ng]

Joe, I would prefer to respond in the forum and not by PM. In this case, if you access the help file for the command, you'll see some explanation there, as well as some references for more detailed reading. In fact, the second author for the command is the same Nick Cox on the forum, who is likely to have something smarter than this to say.

I am actually not very familiar with analyses of concordance, be they for binary, ordinal, or continuous measures. But here are my thoughts. The concordance correlation coefficient is interpreted a lot like a Pearson's or Spearman's coefficient (except that it has some different properties), in that 0 means no agreement at all, and 1 is perfect agreement. Lin, who developed this statistic, characterized anything less than 0.90 as poor agreement. So, by that standard, these 4 scores have poor agreement.

However, you should consider the context of his writing. Skimming his original paper, the examples he discusses are assays. Those usually measure some physiological parameter, e.g. hemoglobin A1C, viral load - basically, something that is tied to objective reality. The acceptable amount of agreement between two different devices to read the same physiological parameter should be very high.

But the expected or acceptable amount of agreement for something like a risk score for some disease might be quite different. It looks like some of the risk scores read systematically higher (in this sample) than some others, which is also worth thinking about. In context, the amount of agreement displayed could be acceptable. I don't know what is state of the art for concordance between continuous scores in health services research, though. If this subject is really important to what you're doing, this is where you start asking experts and doing background research on your own.

Here's a link to Lin's original paper. It's probably worth reading his intro.