Calculating a p-value for difference in diagnostic accuracy parameter by group

[Theresa Pfu]

Hi all,

I have calculated diagnostic accuracy for a molecular test using diagt

diagt ref_fq xdr_fq

I have then also calculated the accuracy of this test by subgroup (samples with high bacillary burden, samples with low bacillary burden):

diagt ref_fq xdr_fq if ultra_semiquant==0
diagt ref_fq xdr_fq if ultra_semiquant==1

I can see a difference in sensitivity of the test by bacillary burden albeit with overlapping confidence intervals. however I would like to calculate a p-value for this.

I have tried to do this with an immediate command prtesti. The # of samples with low bacillary burden is 56 and the # of samples with high bacillary burden is 69. Sensitivity in low burden samples was 100% and sensitivity in high burden samples was 77.3%, so I wanted to use the following command:

prtesti 56 1.0 69 0.773, level(95) count

but I get the error message that integers are expected where I include my proportions.

Would appreciate any help of how to use prtesti in this case- or which other command would be better.

Thanks,
Theresa

[Carlo Lazzaro]

Theresa:
you added the -count- option but 0.773 is actually a proportion. Therefore Stata complains on that.
You may want to take a look at the following toy-example:

. prtesti 30 0.4 45 0.6

Two-sample test of proportions                     x: Number of obs =       30
                                                   y: Number of obs =       45
------------------------------------------------------------------------------
             |       Mean   Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
           x |         .4   .0894427                      .2246955    .5753045
           y |         .6   .0730297                      .4568645    .7431355
-------------+----------------------------------------------------------------
        diff |        -.2   .1154701                     -.4263171    .0263171
             |  under H0:   .1177568    -1.70   0.089
------------------------------------------------------------------------------
        diff = prop(x) - prop(y)                                  z =  -1.6984
    H0: diff = 0

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(Z < z) = 0.0447         Pr(|Z| > |z|) = 0.0894          Pr(Z > z) = 0.9553

. di 30*0.4
12

. di 45*0.6
27

. prtesti 30 12 45 27, count

Two-sample test of proportions                     x: Number of obs =       30
                                                   y: Number of obs =       45
------------------------------------------------------------------------------
             |       Mean   Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
           x |         .4   .0894427                      .2246955    .5753045
           y |         .6   .0730297                      .4568645    .7431355
-------------+----------------------------------------------------------------
        diff |        -.2   .1154701                     -.4263171    .0263171
             |  under H0:   .1177568    -1.70   0.089
------------------------------------------------------------------------------
        diff = prop(x) - prop(y)                                  z =  -1.6984
    H0: diff = 0

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(Z < z) = 0.0447         Pr(|Z| > |z|) = 0.0894          Pr(Z > z) = 0.9553

.

[Leonardo Guizzetti]

The approach using a proportions test such as -prtest- would not be appropriate here since you have correlated data in the form of multiple diagnostic tests on the same dataset.

I would first really think about if you need to compare sensitivity? Is that really the best parameter to examine? Why not specificity or positive predictive value or overall diagnostic classification? If you believe sensitivity is most important, you might also consider whether simply showing the estimates and CIs is good enough? Having one method show 100% with that sample size is fairly good in my opinion.

if you decide you must have a test, then you’ll want to use McNemar’s test but you’ll need to do some data reshaping first. You need to restrict your sample to just the true positives, and the test will compare how well or not the diagnostic tests perform compared to each other. When both tests give the same result, this gives no information. However, it’s the disagreements that the test statistic is based on. There are several ways to perform this test but I think -symmetry- is most straightforward. You can have the asymptotic result at the symmetry test, or the exact test.

here is some example code

clear
input r1 r2 count
0 0 794
0 1 150
1 0 86
1 1 570
end  

symmetry r1 r2 [fw=count]