Meta Analysis - Data input problem

[Janet Hill]

I am trying to learn meta analysis and I am using an example from Bland, An Introduction to Medical Statistics:

* Example generated by -dataex-. To install: ssc install dataex
clear
input float(or ll ul)
 .29 .03  3.12
5.09  .5 52.29
 .35 .16   .74
 .18 .08   .55
1.24 .52  2.96
 .17 .07   .43
 .71 .14  3.66
 .58 .25  1.36
  .4 .24   .66
 .01   0   .21
 .82 .27  2.54
1.44 .37  5.67
 .61 .29  1.28
 .59 .34  1.04
 .46  .1  2.05
  .5 .17  1.54
 .98 .56  1.72
end

When I use admetan, from SSC, I have no problem:

. admetan or ll ul

Studies included: 17
Participants included: Unknown

Meta-analysis pooling of aggregate data
using the fixed-effect inverse-variance model

--------------------------------------------------------------------
Study                |   Effect    [95% Conf. Interval]   % Weight
---------------------+----------------------------------------------
1                    |     0.290      0.030     3.120       0.21
2                    |     5.090      0.500    52.290       0.00
3                    |     0.350      0.160     0.740       6.00
4                    |     0.180      0.080     0.550       9.14
5                    |     1.240      0.520     2.960       0.34
6                    |     0.170      0.070     0.430      15.57
7                    |     0.710      0.140     3.660       0.16
8                    |     0.580      0.250     1.360       1.64
9                    |     0.400      0.240     0.660      11.44
10                   |     0.010      0.000     0.210      45.76
11                   |     0.820      0.270     2.540       0.39
12                   |     1.440      0.370     5.670       0.07
13                   |     0.610      0.290     1.280       2.06
14                   |     0.590      0.340     1.040       4.12
15                   |     0.460      0.100     2.050       0.53
16                   |     0.500      0.170     1.540       1.08
17                   |     0.980      0.560     1.720       1.50
---------------------+----------------------------------------------
Overall effect       |     0.193      0.122     0.264     100.00
--------------------------------------------------------------------

Test of overall effect = 0:  z =   5.336  p = 0.000


Heterogeneity Measures
---------------------------------------------------------
                     |     Value      df     p-value
---------------------+-----------------------------------
Cochran's Q          |     39.61     16      0.001
I² (%)               |     59.6%
Modified H²          |     1.475
tau²                 |    0.0420
---------------------------------------------------------

I² = between-study variance (tau²) as a percentage of total variance
Modified H² = ratio of tau² to typical within-study variance

However when I try to use meta set from Stata 16 I get the following error message:

. meta set or ll ul
confidence intervals not symmetric
    CIs defined by variables ll and ul must be symmetric and based on a normal distribution. If you are
    working with effect sizes such as odds ratios, risk ratios, or hazard ratios, the CIs should be
    specified on the log scale.
r(459);

I appreciate from the Manual that the CI should be symmetric but I do not understand how I can resolve this error.

Thank you,
Janet

[Red Owl]

Janet Hill

The help file for -admetan- states:

Confidence intervals are assumed to be symmetric, and the standard error is derived as (CI width)/2z. Hence, supplied confidence limits must be based on a Normal distribution, or the pooled result will not be accurate.

Confidence limits for odds ratios need not be symmetric, and the CIs for the ORs in your example data are not, as Stata warned.

So, although it appears from the study-level plots of the input data in the forest plot that -admetan- (from SSC) worked with your example data, the calculation of the weights and overall effect size made by -admetan- are incorrect because your example data do not satisfy the distributional assumptions for the specified CIs.

I believe the solution to this problem is the same whether you are using -admetan- or the new meta set command in Stata 16, and that is to transform the data to be the logs of the OR and the confidence limits. As the Stata 16 manual (meta set, p. 93) states:

Meta-analysis uses effect sizes in a metric that makes them approximately normally distributed
such as log odds-ratios instead of odds ratios and log hazard-ratios instead of hazard ratios. The
corresponding standard errors or CIs should be provided in the same metric as effect sizes. For example,
if you are working with log hazard-ratios, the provided CIs should be for the log hazard-ratios and
not the hazard ratios.

I hope this helps.

Red Owl
Stata/IC 16.0 (Windows 10, 64-bit)

[Janet Hill]

Thank you for the advice.

I think what I am missing is how this transformation should be carried out - it is probably obvious but I cannot see it. Taking logs of the variables results in the same error message.

Janet

[David Fisher]

Hi Janet,

My first thought was that the problem lay with the fact that one of your lower confidence limits is zero. Hence, when you take logs (as Red Owl correctly advises), you create a missing value. Then, when meta set checks to see if the limits are symmetrical, it finds that one of the limits is of infinite/missing size, and hence not symmetric.

However, a closer look reveals that the problem is more subtle than that. meta set uses the "relative difference" function reldif() to check for symmetry, and its default tolerance is 1e-6. But when you have very small ratio values -- such as some of your lower limits here -- known only to two decimal places, they become very imprecise, and may fail the "relative difference" check despite in truth being symmetric. Possibly StataCorp would call this a feature not a bug, and would discourage users from analysing on the basis of such imprecise values!


Assuming that you cannot obtain a dataset with any greater precision, I can suggest three solutions:

1) Use admetan in preference to meta (but take heed of the warning message in the admetan help file! -- thanks Red Owl for quoting this)

2) Force meta set to accept your inputted values by specifying a larger tolerance value for the "relative difference" check; this appears to be the undocumented option civartol() ...try civartol(0.1) or civartol(0.5)

3) Assuming that your imprecise values do in truth represent symmetric intervals, you could replace the problematic lower limits (e.g. those known only to one or zero significant figures) with the odds ratio divided by the ratio of the upper limit to the odds ratio (that is, force the interval to be exactly symmetric on the log scale).


It's worth stating again that, whichever solution you choose, you must take logs before analysing!

I hope this helps.

David.

[Janet Hill]

Hello David,

Thank you for your advice. Setting civartol(0.2) and modifying the 0 lower limit resolved the issue. I must get into the habit of checking undocumented options.

The problem with taking data from the literature is that one has no control, or even idea, of it's precision.

As both Red Owl and you state, I will always take logs before the analysis.

Thank you,
Janet