Dear Statalisters, this is my first post here and I tried to read the FAQ and the available help and pdf documentation. My apologies if I posted this wrong somehow, and many thanks in advance for any solutions or ideas or directions where to read or look for more info.
I am working on diagnostic test accuracy meta-analyses estimating sensitivity and specificity of a diagnostic test. I am using Stata 15.1.
I have previously been using the user-written midas (installed from SSC) and I am now also using metandi with the user-written command gllamm (installed from SSC).
The dataset:
The dataset includes several very small studies, some of which contain only true positive cases and other only true negative cases. That makes it difficult to define both sensitivity and specificity for some studies. In my understanding, this should present a problem for bivariate analysis.
When trying to use midas tp fp fn tn, res(all) an error message "error obtaining starting values; try fitting a marginal model in order to diagnose the problem r(459)" is returned.
When I perform the same meta-analysis using the metandi command metandi tp fp fn tn, and the same error message "error obtaining starting values; try fitting a marginal model in order to diagnose the problem r(459)" was returned.
I understand both midas and metandi uses meqrlogit (previously called xtmelogit) but in metandi it is possible to instead use gllamm to fit the bivariate model. When I perform the same meta-analysis using the command "metandi tp fp fn tn, gllamm" it starts to run but returns another error message
"Fitting univariate model for Sensitivity, Se = tp / (tp + fn)
Fitting univariate model for Specificity, Sp = tn / (tn + fp)
Fitting bivariate model:
Running adaptive quadrature
Iteration 0: log likelihood = -32.544583
Iteration 1: log likelihood = -32.267811
Iteration 2: log likelihood = -31.846541
Iteration 3: log likelihood = -31.639503
Iteration 4: log likelihood = -31.630623
Iteration 5: log likelihood = -31.630785
Iteration 6: log likelihood = -31.630775
Adaptive quadrature has converged, running Newton-Raphson
Iteration 0: log likelihood = -31.630775
Iteration 1: log likelihood = -31.630775
Maximum number of iterations exceeded.
r(498);"
I have not been able to find out how to increase maximum number of iterations, but what I am able to do is increasing the number of integration points from the default 5 to e.g. 9 in order to change fitting. To my surprise, then the meta-analysis runs and returns results.
The command "metandi tp fp fn tn, gllamm nip(9)" returns:
"Fitting univariate model for Sensitivity, Se = tp / (tp + fn)
Fitting univariate model for Specificity, Sp = tn / (tn + fp)
Fitting bivariate model:
Running adaptive quadrature
Iteration 0: log likelihood = -32.543795
Iteration 1: log likelihood = -32.297008
Iteration 2: log likelihood = -31.871231
Iteration 3: log likelihood = -31.645128
Iteration 4: log likelihood = -31.636484
Iteration 5: log likelihood = -31.636483
Adaptive quadrature has converged, running Newton-Raphson
Iteration 0: log likelihood = -31.636483
Iteration 1: log likelihood = -31.636474
Iteration 2: log likelihood = -31.636474
Meta-analysis of diagnostic accuracy
Log likelihood = -31.636474 Number of studies = 18
------------------------------------------------------------------------------
| Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
Bivariate |
E(logitSe) | 2.44292 .4347997 1.590728 3.295112
E(logitSp) | 3.024731 .6776895 1.696484 4.352978
Var(logitSe) | .3499651 .5664446 .0146653 8.35137
Var(logitSp) | 3.130233 2.748217 .5600884 17.49431
Corr(logits) | 1 . . .
-------------+----------------------------------------------------------------
HSROC |
Lambda | 5.973749 1.261162 3.501918 8.445581
Theta | 1.237839 1.179293 -1.073533 3.54921
beta | 1.095515 .8466506 1.29 0.196 -.5638899 2.754919
s2alpha | 4.186593 4.141301 .6023761 29.09737
s2theta | 1.46e-12 . . .
-------------+----------------------------------------------------------------
Summary pt. |
Se | .9200422 .0319859 .8307185 .9642607
Sp | .953679 .0299372 .8450749 .9872951
DOR | 236.903 212.8234 40.7275 1378.013
LR+ | 19.86229 13.03992 5.485314 71.92126
LR- | .0838415 .034342 .0375666 .1871184
1/LR- | 11.92727 4.885486 5.344211 26.61943
------------------------------------------------------------------------------
Covariance between estimates of E(logitSe) & E(logitSp) .0793657"
Does this make sense? In the metandi: Meta-analysis of diagnostic accuracy using hierarchical logistic regression The Stata Journal (2009) 9, Number 2, pp. 211–229 by Harbord and Whiting I read
"Because the bivariate model can sometimes prove difficult to fit, some care has been taken to provide good starting values. First, two separate univariate models are fit to sensitivity and specificity. These provide excellent starting values for the two mean and two variance parameters of the bivariate model. A reasonable starting value for the correlation parameter is obtained from the correlation between the posterior means (empirical Bayes predictions) of the two univariate models."
I do not understand this on such a detailed level, but I guess it could make sense if it is possible to fit a separate univariate model for sensitivity and another for specificity even when some studies include only true positives (or only true negatives). Then there would be starting points. I do not know how starting values are choosen in midas. But this only seems to work using gllamm instead of meqrlogit, and that I do not quite understand.
It is of course quite possible to exclude the very small studies from the meta-analysis, but if it is possible to include them I would prefer to use all data available on the subject. As expected, the summary estimates do not differ significantly when the meta-analysis is run with the small only tp and only tn studies excluded. And I would prefer to try to understand what I am actually doing and why some analyses works and some don't. If you have advice on what you would do in this situation, I am most thankful for advice.
Is it logical that metandi with gllamm and increased nip does run and return results despite the small studies with only tp:s or only tn:s wheres the other attempted commands did not?
Is it possible to change maximum of iterations in metandi?
Is there an option that allows for modification/change of starting points when using metandi or midas?
Do you have any advice on whether to 1) use the full dataset and the results from metandi tp fp fn tn, gllamm nip(9) or 2) exclude the small studies with only tp:s and the study with only tn:s?
I am working on diagnostic test accuracy meta-analyses estimating sensitivity and specificity of a diagnostic test. I am using Stata 15.1.
I have previously been using the user-written midas (installed from SSC) and I am now also using metandi with the user-written command gllamm (installed from SSC).
The dataset:
| study | tp | fp | fn | tn |
| A | 1 | 1 | 0 | 0 |
| B | 2 | 0 | 0 | 5 |
| C | 1 | 1 | 0 | 7 |
| D | 7 | 0 | 0 | 6 |
| E | 1 | 0 | 0 | 0 |
| F | 9 | 0 | 0 | 0 |
| G | 12 | 1 | 1 | 41 |
| H | 2 | 1 | 0 | 8 |
| I | 2 | 0 | 0 | 8 |
| J | 1 | 0 | 1 | 1 |
| K | 9 | 3 | 2 | 15 |
| L | 2 | 0 | 0 | 8 |
| M | 3 | 2 | 0 | 1 |
| N | 5 | 0 | 0 | 22 |
| O | 1 | 1 | 0 | 2 |
| P | 0 | 0 | 0 | 3 |
| Q | 20 | 0 | 1 | 67 |
| R | 11 | 1 | 3 | 3 |
The dataset includes several very small studies, some of which contain only true positive cases and other only true negative cases. That makes it difficult to define both sensitivity and specificity for some studies. In my understanding, this should present a problem for bivariate analysis.
When trying to use midas tp fp fn tn, res(all) an error message "error obtaining starting values; try fitting a marginal model in order to diagnose the problem r(459)" is returned.
When I perform the same meta-analysis using the metandi command metandi tp fp fn tn, and the same error message "error obtaining starting values; try fitting a marginal model in order to diagnose the problem r(459)" was returned.
I understand both midas and metandi uses meqrlogit (previously called xtmelogit) but in metandi it is possible to instead use gllamm to fit the bivariate model. When I perform the same meta-analysis using the command "metandi tp fp fn tn, gllamm" it starts to run but returns another error message
"Fitting univariate model for Sensitivity, Se = tp / (tp + fn)
Fitting univariate model for Specificity, Sp = tn / (tn + fp)
Fitting bivariate model:
Running adaptive quadrature
Iteration 0: log likelihood = -32.544583
Iteration 1: log likelihood = -32.267811
Iteration 2: log likelihood = -31.846541
Iteration 3: log likelihood = -31.639503
Iteration 4: log likelihood = -31.630623
Iteration 5: log likelihood = -31.630785
Iteration 6: log likelihood = -31.630775
Adaptive quadrature has converged, running Newton-Raphson
Iteration 0: log likelihood = -31.630775
Iteration 1: log likelihood = -31.630775
Maximum number of iterations exceeded.
r(498);"
I have not been able to find out how to increase maximum number of iterations, but what I am able to do is increasing the number of integration points from the default 5 to e.g. 9 in order to change fitting. To my surprise, then the meta-analysis runs and returns results.
The command "metandi tp fp fn tn, gllamm nip(9)" returns:
"Fitting univariate model for Sensitivity, Se = tp / (tp + fn)
Fitting univariate model for Specificity, Sp = tn / (tn + fp)
Fitting bivariate model:
Running adaptive quadrature
Iteration 0: log likelihood = -32.543795
Iteration 1: log likelihood = -32.297008
Iteration 2: log likelihood = -31.871231
Iteration 3: log likelihood = -31.645128
Iteration 4: log likelihood = -31.636484
Iteration 5: log likelihood = -31.636483
Adaptive quadrature has converged, running Newton-Raphson
Iteration 0: log likelihood = -31.636483
Iteration 1: log likelihood = -31.636474
Iteration 2: log likelihood = -31.636474
Meta-analysis of diagnostic accuracy
Log likelihood = -31.636474 Number of studies = 18
------------------------------------------------------------------------------
| Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
Bivariate |
E(logitSe) | 2.44292 .4347997 1.590728 3.295112
E(logitSp) | 3.024731 .6776895 1.696484 4.352978
Var(logitSe) | .3499651 .5664446 .0146653 8.35137
Var(logitSp) | 3.130233 2.748217 .5600884 17.49431
Corr(logits) | 1 . . .
-------------+----------------------------------------------------------------
HSROC |
Lambda | 5.973749 1.261162 3.501918 8.445581
Theta | 1.237839 1.179293 -1.073533 3.54921
beta | 1.095515 .8466506 1.29 0.196 -.5638899 2.754919
s2alpha | 4.186593 4.141301 .6023761 29.09737
s2theta | 1.46e-12 . . .
-------------+----------------------------------------------------------------
Summary pt. |
Se | .9200422 .0319859 .8307185 .9642607
Sp | .953679 .0299372 .8450749 .9872951
DOR | 236.903 212.8234 40.7275 1378.013
LR+ | 19.86229 13.03992 5.485314 71.92126
LR- | .0838415 .034342 .0375666 .1871184
1/LR- | 11.92727 4.885486 5.344211 26.61943
------------------------------------------------------------------------------
Covariance between estimates of E(logitSe) & E(logitSp) .0793657"
Does this make sense? In the metandi: Meta-analysis of diagnostic accuracy using hierarchical logistic regression The Stata Journal (2009) 9, Number 2, pp. 211–229 by Harbord and Whiting I read
"Because the bivariate model can sometimes prove difficult to fit, some care has been taken to provide good starting values. First, two separate univariate models are fit to sensitivity and specificity. These provide excellent starting values for the two mean and two variance parameters of the bivariate model. A reasonable starting value for the correlation parameter is obtained from the correlation between the posterior means (empirical Bayes predictions) of the two univariate models."
I do not understand this on such a detailed level, but I guess it could make sense if it is possible to fit a separate univariate model for sensitivity and another for specificity even when some studies include only true positives (or only true negatives). Then there would be starting points. I do not know how starting values are choosen in midas. But this only seems to work using gllamm instead of meqrlogit, and that I do not quite understand.
It is of course quite possible to exclude the very small studies from the meta-analysis, but if it is possible to include them I would prefer to use all data available on the subject. As expected, the summary estimates do not differ significantly when the meta-analysis is run with the small only tp and only tn studies excluded. And I would prefer to try to understand what I am actually doing and why some analyses works and some don't. If you have advice on what you would do in this situation, I am most thankful for advice.
Is it logical that metandi with gllamm and increased nip does run and return results despite the small studies with only tp:s or only tn:s wheres the other attempted commands did not?
Is it possible to change maximum of iterations in metandi?
Is there an option that allows for modification/change of starting points when using metandi or midas?
Do you have any advice on whether to 1) use the full dataset and the results from metandi tp fp fn tn, gllamm nip(9) or 2) exclude the small studies with only tp:s and the study with only tn:s?
Comment