Meta-analysis of RCTs

[Lars Pete]

Dear all,

I had a basic question pertaining to meta-analysis of RCTs.
If we have ATEs and CIs of the same intervention from 10 different RCT studies (all standardized differences), how can we calculate the Expected ATE (standardized difference) for the 11th study which we are conducting? Is it merely a simple average of the 10 ATEs? or a weighted average? If it is weighted average, what will be the weights?

Thanks.

[Weiwen Ng]

In terms of how, you could have Stata's meta analysis command do all the calculations for you, unless you want to do them by hand (or more likely in Excel) to teach yourself the math. However, in all meta-analysis estimates, each study is weighted by the inverse of its variance (in fixed effects meta-analysis), or a more complex formulation that's still fundamentally pretty similar to the inverse of the study variance (random effects meta-analysis). Imagine that you had one very large study and 9 very small ones. Chances are that the very large study is much closer to the truth than the tiny ones because it will produce a more precise estimate of the treatment effect (assuming all the studies are of adequate quality). If you just averaged the ATEs from the 10 studies, you would be under-representing the big study.

In Stata's intro to meta analysis, see the first forest plot on page 8. I count 19 studies in the forest plot. If someone had done a simple average, the weight for each study should be 5.26% (NB for emphasis: nobody does a simple average.) Instead, the smallest two studies with the widest confidence intervals have weights of 1.69-1.72%. There's one study with a weight of 9.64%, and two more with weights over 9%. That plot is a random effects meta analysis.

[Lars Pete]

Thanks for your answer Weiwen.
So I will find the std errors from the CI's and use them to give weights to estimates to take a weighted average.
Just to be doubly sure, you're recommending inverse of variance and not SD?

[Lars Pete]

I think it should be inverse of std error and not variance. What do you think?

[Weiwen Ng]

Thanks for your answer Weiwen.
So I will find the std errors from the CI's and use them to give weights to estimates to take a weighted average.
Just to be doubly sure, you're recommending inverse of variance and not SD?

You mean that you will use Stata's meta analysis commands to do this, I hope? Again, there's no reason to do this by hand.

If you read the documentation, the methods and formulas for the common effect and fixed effect analyses (pgs 4 and 5) say that the weight for the j-th study is 1 over the σ-hat_j squared. On pg 3, σ-hat_j is defined as the study's standard error. (Remember, the SE is the standard deviation of the sample mean, and variance is always SD^2.) Hence, the weight is the inverse of the variance, as opposed to the SE. Sometimes you have studies that report CIs but not standard errors, but Stata can derive the SE from the CIs (or you can, but again, there's a reason why we made computer programs).

[Chang Xu]

Hi Lars,
I think you mean you want to predict the potential range of a new further study by the pooled effects of the 10 RCTs. If you mean this, the "admetan" provide the prediction intervals. Install admetan and see the help file to take the prediction interval.

Best regards
Chang

[David Fisher]

Hi Lars,

Just to follow up on Chang Xu's comment:

I agree that what you describe sounds more like a Prediction Interval than the standard pooled result. Firstly, I would recommend doing some reading to check exactly what you want to achieve and what the recommended methodology would be. If indeed it is a Prediction Interval [1], then either the official Stata 16 meta suite or the user-written metan command (ssc install metan) will do this for you.

(Note that admetan is now superseded by an updated version of metan; see https://www.statalist.org/forums/for...alysis-command)

Thanks,

David.


[1] Higgins JPT, Thompson SG, Spiegelhalter DJ. 2009. A re-evaluation of random-effects meta-analysis. JRSS Series A 172: 137-159