Comparing sens, spec, ppv, npv of 2 tests on the same population

[Sarah Baer]

Hi,
I have used hours of googling including statalist to find out how I can compare sens, spec, ppv, npv of 2 test on the same population. I could not find a conclusive answer, and I apologize on beforehand for this basic question. But I am close to a deadline and I need to find the solution now.

Here is my dataset:

Binary disease variable (1/0): ischemic_cad
Binary test 1 (1/0): cle_tpc_isch_pp
Binary test 2 (1/0): cleerly_youden

I have calculated the test characteristics with diagt already and have also used roccomp for AUC. But I need to formally compare sens, spec, NPV, and PPV between the two tests.
Here ist the output from diagt:

. diagt ischemic_cad cle_tpc_isch_pp

ischemic_c | cle_tpc_isch_pp
ad | Pos. Neg. | Total
-----------+----------------------+----------
Abnormal | 269 62 | 331
Normal | 178 1,266 | 1,444
-----------+----------------------+----------
Total | 447 1,328 | 1,775

True abnormal diagnosis defined as ischemic_cad = 1


[95% Confidence Interval]
---------------------------------------------------------------------------
Prevalence Pr(A) 18.6% 16.9% 20.5%
---------------------------------------------------------------------------
Sensitivity Pr(+|A) 81.3% 76.6% 85.3%
Specificity Pr(-|N) 87.7% 85.9% 89.3%
ROC area (Sens. + Spec.)/2 0.84 0.82 0.87
---------------------------------------------------------------------------
Likelihood ratio (+) Pr(+|A)/Pr(+|N) 6.59 5.69 7.64
Likelihood ratio (-) Pr(-|A)/Pr(-|N) 0.21 0.17 0.27
Odds ratio LR(+)/LR(-) 30.86 22.48 42.36
Positive predictive value Pr(A|+) 60.2% 55.5% 64.7%
Negative predictive value Pr(N|-) 95.3% 94.1% 96.4%
---------------------------------------------------------------------------

. diagt ischemic_cad cleerly_youden

ischemic_c | cleerly_youden
ad | Pos. Neg. | Total
-----------+----------------------+----------
Abnormal | 285 46 | 331
Normal | 212 1,232 | 1,444
-----------+----------------------+----------
Total | 497 1,278 | 1,775

True abnormal diagnosis defined as ischemic_cad = 1


[95% Confidence Interval]
---------------------------------------------------------------------------
Prevalence Pr(A) 18.6% 16.9% 20.5%
---------------------------------------------------------------------------
Sensitivity Pr(+|A) 86.1% 81.9% 89.6%
Specificity Pr(-|N) 85.3% 83.4% 87.1%
ROC area (Sens. + Spec.)/2 0.86 0.84 0.88
---------------------------------------------------------------------------
Likelihood ratio (+) Pr(+|A)/Pr(+|N) 5.86 5.14 6.69
Likelihood ratio (-) Pr(-|A)/Pr(-|N) 0.16 0.12 0.21
Odds ratio LR(+)/LR(-) 36.00 25.56 50.72
Positive predictive value Pr(A|+) 57.3% 52.9% 61.7%
Negative predictive value Pr(N|-) 96.4% 95.2% 97.4%
---------------------------------------------------------------------------


How can I proceed with the comparisons? I highly appreciate your urgent help!

Sarah

[Carlo Lazzaro]

Sarah:
welcome to this forum.
Surely there are more efficient solutions, but the best one that springs to my mind is the following one (aim: comparing the proportion of correctly classified cars):

use "C:\Program Files\Stata18\ado\base\a\auto.dta"

. logit foreign mpg

Iteration 0:  Log likelihood =  -45.03321  
Iteration 1:  Log likelihood = -39.380959  
Iteration 2:  Log likelihood = -39.288802  
Iteration 3:  Log likelihood =  -39.28864  
Iteration 4:  Log likelihood =  -39.28864  

Logistic regression                                     Number of obs =     74
                                                        LR chi2(1)    =  11.49
                                                        Prob > chi2   = 0.0007
Log likelihood = -39.28864                              Pseudo R2     = 0.1276

------------------------------------------------------------------------------
     foreign | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
         mpg |   .1597621   .0525876     3.04   0.002     .0566922     .262832
       _cons |  -4.378866   1.211295    -3.62   0.000    -6.752961   -2.004771
------------------------------------------------------------------------------

. estat classification

Logistic model for foreign

              -------- True --------
Classified |         D            ~D  |      Total
-----------+--------------------------+-----------
     +     |         6             5  |         11
     -     |        16            47  |         63
-----------+--------------------------+-----------
   Total   |        22            52  |         74

Classified + if predicted Pr(D) >= .5
True D defined as foreign != 0
--------------------------------------------------
Sensitivity                     Pr( +| D)   27.27%
Specificity                     Pr( -|~D)   90.38%
Positive predictive value       Pr( D| +)   54.55%
Negative predictive value       Pr(~D| -)   74.60%
--------------------------------------------------
False + rate for true ~D        Pr( +|~D)    9.62%
False - rate for true D         Pr( -| D)   72.73%
False + rate for classified +   Pr(~D| +)   45.45%
False - rate for classified -   Pr( D| -)   25.40%
--------------------------------------------------
Correctly classified                        71.62%
--------------------------------------------------

. logit foreign trunk

Iteration 0:  Log likelihood =  -45.03321  
Iteration 1:  Log likelihood = -40.102483  
Iteration 2:  Log likelihood = -39.953293  
Iteration 3:  Log likelihood = -39.953144  
Iteration 4:  Log likelihood = -39.953144  

Logistic regression                                     Number of obs =     74
                                                        LR chi2(1)    =  10.16
                                                        Prob > chi2   = 0.0014
Log likelihood = -39.953144                             Pseudo R2     = 0.1128

------------------------------------------------------------------------------
     foreign | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
       trunk |  -.2092823   .0716298    -2.92   0.003    -.3496742   -.0688904
       _cons |   1.869341    .930566     2.01   0.045     .0454648    3.693217
------------------------------------------------------------------------------

. estat classification

Logistic model for foreign

              -------- True --------
Classified |         D            ~D  |      Total
-----------+--------------------------+-----------
     +     |         5             5  |         10
     -     |        17            47  |         64
-----------+--------------------------+-----------
   Total   |        22            52  |         74

Classified + if predicted Pr(D) >= .5
True D defined as foreign != 0
--------------------------------------------------
Sensitivity                     Pr( +| D)   22.73%
Specificity                     Pr( -|~D)   90.38%
Positive predictive value       Pr( D| +)   50.00%
Negative predictive value       Pr(~D| -)   73.44%
--------------------------------------------------
False + rate for true ~D        Pr( +|~D)    9.62%
False - rate for true D         Pr( -| D)   77.27%
False + rate for classified +   Pr(~D| +)   50.00%
False - rate for classified -   Pr( D| -)   26.56%
--------------------------------------------------
Correctly classified                        70.27%
--------------------------------------------------

. prtesti 74 .7162 74 0.7027

Two-sample test of proportions                     x: Number of obs =       74
                                                   y: Number of obs =       74
------------------------------------------------------------------------------
             |       Mean   Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
           x |      .7162   .0524092                      .6134799    .8189201
           y |      .7027   .0531333                      .5985607    .8068393
-------------+----------------------------------------------------------------
        diff |      .0135   .0746316                     -.1327752    .1597752
             |  under H0:   .0746398     0.18   0.856
------------------------------------------------------------------------------
        diff = prop(x) - prop(y)                                  z =   0.1809
    H0: diff = 0

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(Z < z) = 0.5718         Pr(|Z| > |z|) = 0.8565          Pr(Z > z) = 0.4282

.

[Andrea Discacciati]

Given the paired study design, you can use -mcc-/-mcci- to compare the sensitivity and specificity of the 2 diagnostic tests. See, for example,

Alonzo, T. A., Pepe, M. S., & Moskowitz, C. S. (2002). Sample size calculations for comparative studies of medical tests for detecting presence of disease. Statistics in medicine, 21(6), 835-852.

for the formulae to calculate relative sens and spec (cfr. "ratio" value in -mcc- output).

I am not aware of a Stata command to compare PPV and NPV of the 2 diagnostic tests in a paired study design. However, the formulae to calculate relative PPV and NPV can be found here:

 Moskowitz, C.S., and Pepe, M.S. (2006). Comparing the predictive values of diagnostic tests: sample size and analysis for paired study designs. Clin Trials, 3(3):272-9.

[Leonardo Guizzetti]

The overall strategy seems misguided to me. I would first compare correlated ROC areas between the two methods. Some methods have already been suggested and there is also the DeLong method.

My reasoning is that the AUC will show whether the overall performance of one test is different than the other. This essentially is the comparison of sensitivity and specificity, combined into a single overall metric. There’s no sense in then comparing sensitivity and specificity after testing AUC. Likewise, PPV and NPV are both affected by the overall performance and of prevalence (which is constant within your study). I don’t see what is to gain by further formal hypothesis testing.

E.R. DeLong, D.M. DeLong, and D.L. Clarke-Pearson (1988), "Comparing the Areas Under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach," Biometrics, 44, 837-845.

[Carlo Lazzaro]

I'm grateful to Leonardo for pointing this reference out to us.
Unfortunately, the article seems to be downloadable from https://www.jstor.org/stable/2531595 only.