Reproducing Cochrane meta-analysis in -meta-

[Robert Grant]

I'm preparing some teaching material on meta-analysis. I have some neat examples from Cochrane reviews. I run the data in Stata, with DerSimonian-Laird random effects, which is supposedly the same method (says Cochrane handbook) but get slightly different weights, effect sizes, and CIs. I remember this from a few years back too. The differences are just on the cusp of being too big to be rounding error, but then the data I am using were rounded to 1 dp in the Cochrane publication and I know from personal experience that under the hood, their RevMan software works with whatever precision you give it and always displays a disappointing 1 or 2dp.

I guess I'm just looking for reassurance that others who have encountered this believe the formulas in Stata and RevMan are the same, and the difference we see can just be brushed aside as accumulated rounding error.

=========================
here's some output and a Cochrane screen grab is attached:
. meta summarize, subgroup(japan)

Effect-size label: Weight loss (kg)
Effect size: _meta_es
Std. Err.: _meta_se
Study label: study

Subgroup meta-analysis summary Number of studies = 12
Random-effects model
Method: DerSimonian-Laird
Group: japan

Effect Size: Weight loss (kg)
--------------------------------------------------------------------
Study | Effect Size [95% Conf. Interval] % Weight
------------------+-------------------------------------------------
Group: 0 |
Auvichayapat 2008 | -1.100 -1.986 -0.214 5.22
Hill 2007 | -0.200 -0.425 0.025 9.22
Hsu 2008 | -0.100 -1.035 0.835 4.96
Diepvens2005 | 0.000 -0.377 0.377 8.42
|
theta | -0.217 -0.524 0.091
------------------+-------------------------------------------------
Group: 1 |
Kozuma 2005 | -1.300 -1.432 -1.168 9.55
Takase 2008 | -1.200 -1.371 -1.029 9.43
Nagao 2007 | -0.600 -0.752 -0.448 9.50
Takeshita 2008 | -0.400 -0.597 -0.203 9.34
Kajimoto 2005 | -0.400 -0.583 -0.217 9.39
Suzuki 2009 | -0.200 -0.553 0.153 8.56
Kataoka 2004 | -0.100 -0.363 0.163 9.05
Takashima 2004 | 0.000 -0.540 0.540 7.37
|
theta | -0.549 -0.892 -0.205
------------------+-------------------------------------------------
Overall |
theta | -0.476 -0.774 -0.179
--------------------------------------------------------------------

Heterogeneity summary
-----------------------------------------------------------------------------
Group | df Q P > Q tau2 % I2 H2
---------------+-------------------------------------------------------------
0 | 3 5.07 0.167 0.038 40.85 1.69
1 | 7 163.14 0.000 0.228 95.71 23.31
---------------+-------------------------------------------------------------
Overall | 11 202.10 0.000 0.236 94.56 18.37
-----------------------------------------------------------------------------
Test of group differences: Q_b = chi2(1) = 1.99 Prob > Q_b = 0.159

.
end of do-file

=========================

If you want to try it:

// get the data
import delimited "http://www.robertgrantstats.co.uk/data/cochrane_green_tea_weight_loss.csv", varnames(1) delimiter(",") clear

// calculate the effect size (mean difference, in this case)
meta esize n_tea mean_tea sd_tea n_control mean_control sd_control, ///
esize(mdiff) random(hschmidt) ///
studylabel(study) eslabel("Weight loss (kg)")

// get the MA results
meta summarize, subgroup(japan)

// compare with Analysis 1,4, page 61 in https://www.cochranelibrary.com/cdsr...pub2/epdf/full


sending good wishes to y'all from Hampshire UK
Robert

[Robert Grant]

Particularly, I notice the relatively large shift in the weights of studies by Hsu and Kajimoto. Tweaking the means does not make much difference to the weights.

[Robert Grant]

I put the rounded-off numbers into RevMan and get the Cochrane results, modulo rounding error. So there is still some discrepancy between them and Stata, both purporting to run DerSimonian-Laird, and both manuals referring to the source paper from 1986 rather than giving formulas. Something must be awry somewhere. On the face of it, both should be calculating weights as 1/(sigma^2 + tau^2). Mean differences and their CIs match, so sigma is not to blame. It must be the formula for tau. Or am I missing something else?

If I get time next week, I'll calculate the two tau estimates and see if that sheds any light.

[Leonardo Guizzetti]

Hi Robert, I don't know if you solved your issue yet. I have endeavoured to look at this using a differerent, known data set, but the difference appears to come down to how study-level variance is computed between the two programs. The default approach of -meta esize- is to use the a pooled SD, whereas RevMan and -metan- use an unpooled SD. RevMan documents this in their help file with a file called Statistical Algorithms.

webuse metaescnt, clear
meta esize n1 m1 sd1 n2 m2 sd2, es(mdiff) random(dlaird)
meta summ

metan n1 m1 sd1 n2 m2 sd2, nostandard randomi nograph

// effect size as absolute mean difference
gen double md = m1 - m2
// unpooled SE -- matches metan and RevMan
gen double se_md = sqrt( (sd1^2)/n1 + (sd2^2)/n2 )
// -- pooled SE
gen double se_md2 = sqrt( (1/n1 + 1/n2) * ((n1-1)*(sd1^2) + (n2-1)*(sd2^2)) / (n1 + n2 - 2) )

meta set md se_md, random(dlaird)
meta summ
meta set md se_md2, random(dlaird)
meta summ

list se_md _seES se_md2 _meta_se, sep(0)

and selected output

. webuse metaescnt, clear
(Fictional summary data for continuous outcomes)

. meta esize n1 m1 sd1 n2 m2 sd2, es(mdiff) random(dlaird)
. meta summ

  Effect-size label:  Mean Diff.
        Effect size:  _meta_es
          Std. Err.:  _meta_se

Meta-analysis summary                     Number of studies =     10
Random-effects model                      Heterogeneity:
Method: DerSimonian-Laird                             tau2 =  0.0211
                                                    I2 (%) =   98.67
                                                        H2 =   75.40

--------------------------------------------------------------------
            Study |     Mean Diff.    [95% Conf. Interval]  % Weight
------------------+-------------------------------------------------
         Study  1 |         -0.824      -0.851      -0.796     10.19
         Study  2 |         -1.110      -1.168      -1.052      9.87
         Study  3 |         -0.902      -0.959      -0.844      9.88
         Study  4 |         -0.899      -0.949      -0.848      9.96
         Study  5 |         -1.066      -1.080      -1.052     10.25
         Study  6 |         -0.999      -1.035      -0.964     10.12
         Study  7 |         -0.767      -0.839      -0.695      9.66
         Study  8 |         -1.125      -1.167      -1.082     10.05
         Study  9 |         -1.022      -1.086      -0.958      9.79
         Study 10 |         -0.786      -0.808      -0.764     10.22
------------------+-------------------------------------------------
            theta |         -0.950      -1.041      -0.859
--------------------------------------------------------------------
Test of theta = 0: z = -20.41                    Prob > |z| = 0.0000
Test of homogeneity: Q = chi2(9) = 678.61          Prob > Q = 0.0000

.
. metan n1 m1 sd1 n2 m2 sd2, nostandard randomi nograph

           Study     |     WMD   [95% Conf. Interval]     % Weight
---------------------+---------------------------------------------------
1                    | -0.824      -0.850    -0.797         10.20
2                    | -1.110      -1.173    -1.046          9.78
3                    | -0.902      -0.961    -0.842          9.84
4                    | -0.899      -0.943    -0.854         10.04
5                    | -1.066      -1.080    -1.053         10.27
6                    | -0.999      -1.036    -0.963         10.11
7                    | -0.767      -0.839    -0.695          9.64
8                    | -1.125      -1.163    -1.086         10.10
9                    | -1.022      -1.083    -0.960          9.81
10                   | -0.786      -0.811    -0.762         10.21
---------------------+---------------------------------------------------
D+L pooled WMD       | -0.950      -1.039    -0.861        100.00
---------------------+---------------------------------------------------

  Heterogeneity chi-squared = 656.00 (d.f. = 9) p = 0.000
  I-squared (variation in WMD attributable to heterogeneity) =  98.6%
  Estimate of between-study variance Tau-squared =  0.0202

  Test of WMD=0 : z=  20.84 p = 0.000

. meta set md se_md, random(dlaird)
. meta summ

  Effect-size label:  Effect Size
        Effect size:  md
          Std. Err.:  se_md

Meta-analysis summary                     Number of studies =     10
Random-effects model                      Heterogeneity:
Method: DerSimonian-Laird                             tau2 =  0.0202
                                                    I2 (%) =   98.63
                                                        H2 =   72.89

--------------------------------------------------------------------
            Study |    Effect Size    [95% Conf. Interval]  % Weight
------------------+-------------------------------------------------
         Study  1 |         -0.824      -0.850      -0.797     10.20
         Study  2 |         -1.110      -1.173      -1.046      9.78
         Study  3 |         -0.902      -0.961      -0.842      9.84
         Study  4 |         -0.899      -0.943      -0.854     10.04
         Study  5 |         -1.066      -1.080      -1.053     10.27
         Study  6 |         -0.999      -1.036      -0.963     10.11
         Study  7 |         -0.767      -0.839      -0.695      9.64
         Study  8 |         -1.125      -1.163      -1.086     10.10
         Study  9 |         -1.022      -1.083      -0.960      9.81
         Study 10 |         -0.786      -0.811      -0.762     10.21
------------------+-------------------------------------------------
            theta |         -0.950      -1.039      -0.861
--------------------------------------------------------------------
Test of theta = 0: z = -20.84                    Prob > |z| = 0.0000
Test of homogeneity: Q = chi2(9) = 656.00          Prob > Q = 0.0000

. meta set md se_md2, random(dlaird)
. meta summ

  Effect-size label:  Effect Size
        Effect size:  md
          Std. Err.:  se_md2

Meta-analysis summary                     Number of studies =     10
Random-effects model                      Heterogeneity:
Method: DerSimonian-Laird                             tau2 =  0.0211
                                                    I2 (%) =   98.67
                                                        H2 =   75.40

--------------------------------------------------------------------
            Study |    Effect Size    [95% Conf. Interval]  % Weight
------------------+-------------------------------------------------
         Study  1 |         -0.824      -0.851      -0.796     10.19
         Study  2 |         -1.110      -1.168      -1.052      9.87
         Study  3 |         -0.902      -0.959      -0.844      9.88
         Study  4 |         -0.899      -0.949      -0.848      9.96
         Study  5 |         -1.066      -1.080      -1.052     10.25
         Study  6 |         -0.999      -1.035      -0.964     10.12
         Study  7 |         -0.767      -0.839      -0.695      9.66
         Study  8 |         -1.125      -1.167      -1.082     10.05
         Study  9 |         -1.022      -1.086      -0.958      9.79
         Study 10 |         -0.786      -0.808      -0.764     10.22
------------------+-------------------------------------------------
            theta |         -0.950      -1.041      -0.859
--------------------------------------------------------------------
Test of theta = 0: z = -20.41                    Prob > |z| = 0.0000
Test of homogeneity: Q = chi2(9) = 678.61          Prob > Q = 0.0000

.
. list se_md _seES se_md2 _meta_se, sep(0)

     +-----------------------------------------------+
     |     se_md       _seES      se_md2    _meta_se |
     |-----------------------------------------------|
  1. | .01360201   .01360201   .01397216   .01397216 |
  2. | .03230698   .03230698   .02969402   .02969402 |
  3. | .03027551   .03027551   .02929978   .02929978 |
  4. | .02257029   .02257029   .02586993   .02586993 |
  5. | .00681115   .00681115   .00730872   .00730872 |
  6. | .01866232   .01866232   .01806767   .01806767 |
  7. | .03676651   .03676651   .03669382   .03669382 |
  8. | .01952884   .01952884   .02174497   .02174497 |
  9. | .03137941   .03137941   .03253224   .03253224 |
 10. | .01259328   .01259328   .01108668   .01108668 |
     +-----------------------------------------------+

And the output from RevMan (I couldn't figure out how to show more decimal points).

[Robert Grant]

Thank you Leonardo, it looks like you tracked down the difference.