sensitivity estimates for median specificity from a bayesian Hierarchical summary ROC model

[Hari Balasubramanian]

Dear all

I am working on a diagnostic test accuracy review that includes 12 studies reporting the sensitivity and specificity of a test for detection of infection in children (tp,fp,tn and fn for each study enclosed in the excel sheet)
Since each study has reported the diagnostic test estimates at a different threshold value, we needed to obtain the pooled estimates of sensitivity using a HSROC model,

This model cannot be directly fitted in Stata. So, i employed a Bayesian algorithm to obtain a) the five parameters of the HSROC model and then b) summary estimates of sensitivity and specificity. ( model and Bayesian algorithm attached)

What i have got, is the jointly estimated sensitivity and specificity.
Considering that each study has used a different test threshold , i am required to report the pooled sensitivity for a fixed specificity ( median specificity of the 12 studies). and i am unable to get that? Guidance at this step would be greatly appreciated

Thanks

[Suhail Doi]

Hi Hari,

You have raised an interesting question whose answer is more or less philosophical but I will give it a go.

First let us keep in mind that the bivariate model

ssc install midas

or

ssc install metandi

is essentially equivalent to the HSROC model so this can easily be fitted in Stata. If you run your data in either of these Stata modules you will get more or less the same pooled Se and Sp as in your attached Word file.

The question is, can you really pool Se/Sp pairs when thresholds are different? The answer to this question is no because these measures quantify both test discrimination as well as threshold effects. What can be done is to pool a (more or less) threshold independent measure - the diagnostic odds ratio. However, this cannot be done reliably with midas or metandi because they start off with the Se/Sp pairs. There are two ways this can be done:
a) Use admetan and pool the DOR as follows:

admetan tp fp fn tn, ivhet or

b) Use a diagnostic meta-analysis module in Stata that starts with the DOR. Luckily there is one called diagma

ssc install diagma

which utilizes a split component synthesis method and if you run the following code:

diagma tp fp fn tn

it returns the same DOR as admetan.

Now, the DOR output is correct (diagma) but the pooling of Se and Sp is still conceptually wrong if thresholds are deemed to be different. But I noticed that the sROC plot in diagma did not have the typical appearance of varying thresholds. diagma has a cutplot option to test this as follows:

diagma tp fp fn tn, cutplot

The cutplots suggest that rather than the thresholds varying there is a lot of error. Since we already know that thresholds are different, the question then is why this is not seen? The answer is quite logical - there is much more systematic error than potential threshold effects completely overshadowing any potential threshold effects. In summary the pooled Se/Sp from diagma can be used as the error minimizing summaries and no further fancy analysis is required. The downside is that the paper on which diagma is based is still in review and so you only have basic clinical epidemiology principles to back you up.

One final point is that if you have the pooled DOR you can easily compute a Se for any Sp or vice versa but this does not make sense given that we have no idea which threshold this pair belongs to and more importantly there is not much impact of the thresholds given gross heterogeneity suggesting that the pooled values are the most reasonable estimates you can get but are associated with significant uncertainty.

Best
Suhail

[Hari Balasubramanian]

Hi Suhail,

Thank you the holistic approach to the problem and for explaining the heterogeneity aspect of the data

Regards
Hari