metan: A comprehensive update to the popular meta-analysis command

[David Fisher]

With thanks as ever to Kit Baum, I am excited to announce a major update to the user-written command "metan", version 4.0, now available via SSC.

Firstly, a bit of history: as described in this thread I previously released v3.x of the admetan / ipdmetan meta-analysis command suite, and presented it at the 2018 London UK Stata Conference. One of the discussions that emerged was the possibility of merging admetan with the ever-popular but venerable metan, which had not been updated for a decade and currently has no dedicated support. Over the past year I have worked with Professor Jonathan Sterne, a co-author of metan, and Professor Julian Higgins -- both eminent figures in the meta-analysis field -- to make this happen. An important aspect of this work has been to make the update as seamless as possible -- that is, to enable existing metan users to continue to use the command in the same way as before, if they so wish. We have also ensured that all default behaviour is in keeping with current best practice in the field.

Much of the "new" functionality is simply carried over from admetan, and as such is described in this thread. However, all new behaviour, and differences from the previous version and from Stata 16's meta suite, are fully documented in the *.sthlp files. In particular:

• Standard heterogeneity output, including I-squared, is always based on a single Q statistic calculated directly from the data (either Cochran's Q, or a common-effect alternative e.g. using the Mantel-Haenszel log-effect size and weights) rather than on tau-squared. Tau-squared is reported in a separate table. If the user also wishes to see I-squared as derived from tau-squared and sigma-squared (see Higgins & Thompson 2002), they may use the option isqparam (see documentation).
• Returned and saved effect sizes, standard errors and confidence intervals are *always* on the interval/log scale, even with Mantel-Haenszel methods. This avoids confusion when comparing between models (see below) and enables standardised interface with other commands. (However, note that previously-returned statistics such as r(RR) are still, additionally, available.)
• A major new feature: the results of multiple models -- inverse-variance, Mantel-Haenszel, DerSimonian-Laird, REML, and so on -- may be presented simultaneously beneath the list of individual studies on-screen and in a forest plot. Presented weights are taken from the first model in the list. All pooled results, heterogeneity statistics etc. are returned in matrices. This can also be combined with a (single) subgroup variable with by().
• metan now incorporates the functionality of the user-written programs metacum, metaninf and metaprop, respectively for cumulative and influence meta-analysis and meta-pooling of proportions. The older programs call metan from within them, and *should* still work OK (due to our desire for the update to be seamless), but it is recommended that metan now be used instead, with the options cumulative, influence or proportion as required (see documentation).

As a result of this update, the admetan package will no longer be maintained. However, ipdmetan remains as a separate package -- albeit with a dependency on metan -- and will continue to be maintained (a small update to ipdmetan is also now available via SSC).

Since this is such a major update, there are bound to be teething troubles; so please don't hesitate to contact me ([email protected]) with any bugs or other issues. If all else fails, the previous version of metan is still available within the package under the name metan9.ado.

I hope you find the updated package useful.

Season's greetings to all!

David.


David Fisher
Statistician
MRC Clinical Trials Unit at UCL, London, UK
e-mail: [email protected]


P.S. Apologies to anyone who has started threads here, or replied to existing threads, with issues related to admetan, ipdmetan or forestplot (or, indeed, metan) over the last couple of months; I've been putting all my energy into getting this update out. I will now try to catch up!

[Suhail Doi]

Hi David

I figured out where the bug is described in post #57:
The original command used was:

metan lnes lneslci lneshci if smoking !=0 & ma==1,  model(ivhet)  study( studyid )  forestplot(astext(85) textsize(100) boxscale(50) spacing(1) leftjustify range(0.3 2.1) dp(2))  extraline(yes) hetinfo(isq) by(smokcat) eform(HR) noover

When I drop extraline(yes) in the following code then all is well:

metan lnes lneslci lneshci if smoking !=0 & ma==1,  model(ivhet)  study( studyid )  forestplot(astext(85) textsize(100) boxscale(50) spacing(1) leftjustify range(0.3 2.1) dp(2)) hetinfo(isq) by(smokcat) eform(HR) noover

Regards
Suhail

[Suhail Doi]

Hi David,

There is a bug in the subgroup forest plots in the latest version - the heterogeneity estimate in the last subgroup is repeated across previous subgroups but the main output page has the right values (only affects the forest plot). An example is given below where the I2 in the first group is 0%

Regards
Suhail

[David Fisher]

Dear all,

With thanks as ever to Kit Baum, an updated version of metan (v4.07 15sep2023) is now available via SSC.

The following bugs have (hopefully) been fixed (among other more minor ones!):

• Proportion data: fixed bug which caused predictive intervals to sometimes be displayed incorrectly (e.g. with subgroups)
• The issue with Quality Effects weights described in post #51 above

Other improvements and changes:

• Implemented "pooled" heterogeneity variance across subgroups (see e.g. Borenstein et al 2009)
• Weights from multiple models can now be displayed in the forest plot and/or saved to the "results set", with new option allweights
• Heterogeneity information display on forest plots is now more clearly arranged, particularly in complex plots e.g. with multiple models. I have also implemented the long-requested change so that the heterogeneity p-value under homogeneity is shown as "p < 0.001" rather than "p = 0.000".
• Alignment of columns of data in forest plots is now handled via mlabpos(), which should hopefully reduce the risk of columns looking "wiggly" (see e.g. https://www.statalist.org/forums/for...text-alignment)
• Due to a subtle but annoying error involving prediction intervals, I have introduced a new value for _USE (_USE==7) specifically for storing prediction interval data. It shouldn't make any obvious difference to how output is displayed.

As ever, thanks to all users who have reported issues -- please continue to do so!


I also presented a poster on metan at this year's Cochrane Colloquium in London earlier this month, and had a few interesting chats as a result. In order to "show off" the capabilities, I put together a series of forest plot examples, some of which were inspired by previous posts here on Statalist, or from email communications -- so thankyou for those. (All data for the examples were taken from published articles!) See our GitHub page here (scroll down to the "Examples" section). If I can find the time, I will try to put up more examples e.g. of particular analyses approaches or use-cases.


Best wishes,

David.

[Helen Jiahuan He]

Thank you very much David, somehow there is no such error anymore so the issue has been resolved!

[Suhail Doi]

Thanks David for the update in #53. Sounds good and I am glad it was nothing too serious. Look forward to your thoughts on the other issue as I have been working on the 'conclusiveness of meta-analyses' and all methods so far use the CMA results and from that perspective CMA in the main results and saved results seem logical though admetan can be used as well.

Regards
Suhail

[David Fisher]

Suhail Doi : Thanks for this example. The error arises as part of a check that I included in the code, to check that the total weight remains the same before & after the correction detailed in your 2015 CCT paper (which of course should always be the case, mathematically speaking). I have tracked the issue down to a line of code where a variable is not generated in double-precision, so that in this specific example, we see sufficient rounding-error for the error message to be triggered. This will be fixed in the next version. In the meantime, one way to avoid the issue is to run separate models in each subgroup; only "subgroup" calculations are affected by this issue, not "overall" calculations. Apologies!
(P.S. I will contact you separately about the issue you raised back in April in post #50. I have been too busy to respond until recently, but I do have some thoughts on that.)

Helen Jiahuan He . Unfortunately I am unable to replicate this error. Furthermore, if you type ssc describe metan, you can see that the file labbe.ado is listed as part of the package. Could it be a network or firewall issue? Ultimately, I guess the thing to do would be to contact SSC/RePEc via the email address they provide.

Thanks,
David.

[Helen Jiahuan He]

Hello,
I ran into an error while I was installing the metan package (please see the screenshot below and the attachment). Basically, I got an error from scc install saying "apparent error in package file for metan" after running the command "ssc install metan, all replace". I am using Stata version 17.0BE, and I also tried Stata version 18.0BE, but I got the same error. I wonder whether someone could help me to address this situation. Thank you very much in advance!


Best,
Helen

[Suhail Doi]

Hi David

With this dataset:

* Example generated by -dataex-. For more info, type help dataex
clear
input str21 study int year str11 location_firstdose int(txanodeath1m txadeath1m planodeath1m pladeath1m) byte qi
"CRASH-2 2010"        2010 "Hospital"    8597 1463 8454 1613 34
"Chakroun-Walha 2019" 2019 "Hospital"      69   27   65   19 28
"CRASH-3 2019"        2019 "Hospital"    3758  855 3622  892 33
"Rowell 2020"         2020 "Prehospital"  563   94  256   53 34
"Guyette 2020"        2020 "Prehospital"  406   41  407   49 35
"PATCH 2023"          2023 "Prehospital"  540  113  498  139 34
end

There is an error when running this code:

metan txadeath1m txanodeath1m pladeath1m planodeath1m  , or qe(qi) study( study )  forestplot(astext(75) textsize(100) boxscale(55) spacing(1.5) leftjustify dp(2) range(0.4 1.3))  extraline(no) hetinfo(isq) by( location_firstdose )

It says:
Error encountered whilst calculating quality weights

Any thoughts?

Regards
Suhail

[Suhail Doi]

Hi David

I just realized that metan does not report cumulative results in _ES _seES _LCI _UCI when the cumulative option is selected and this is different from what admetan does (which respects whichever option is selected). This makes things very difficult when metan is used within simulations or other ado files so would it be possible to revert back to the former behavior or is there an option to activate this? It also seems logical (at least to me) that these new variables report what the output reports?

Many thanks
Regards
Suhail

[David Fisher]

Dear all,

With thanks as ever to Kit Baum, an updated version of metan (v4.06 12oct2022) is now available via SSC.

The following bugs have (hopefully) been fixed:

• Proportion data: fixed bug which caused an exit error when proportion was used with cumulative | influence and with saving() | clear
• Proportion data: proportion with counts now gives the correct denominators for subgroups in the forest plot (they previously appeared doubled)
• When hetinfo(tausq) was used, the value of tau-squared from the first subgroup was repeated across all other subgroups in the forest plot; this has been fixed
• With Quality Effects model, the code has been improved so as to avoid negative weights via rounding error when one or more weights is exactly zero.

Other improvements:

• Added a new option labtitle() to over-ride the default "Subgroup and Study" title with by()
• Minor improvements to the handling and display of zero cells in special cases (e.g. by(), cumulative, influence)
• Major re-ordering of code and subroutines to hopefully improve readability and maintenance going forward. The codebase has effectively been split into three parts: metan.ado, metan_analysis.ado and metan_output.ado; whose respective content should be self-explanatory.

As ever, thanks to all users who have reported issues. In particular, I must apologize if sometimes I do not respond due to workload. When preparing an update, I do try to look back through my correspondence and check that important bug reports or requests have been incorporated. So, please continue to stay in touch!

Best wishes,

David.

[dr Mheissen]

Hi David, thanks for this update and your huge efforts.
how can I get the between subgroups p value/Z to check the statistically differences between them?

[Suhail Doi]

Hi David,

Thanks - I thought that was the QE update but good to know that it is not and look forward to that.

Yes, the FTT issue as raised may be true but is not really that significant because our simulation runs varying study numbers (per MA) and randomly selecting study size between 1 to 100 events (size=event/p) only brings this up in <1% of pooled meta-analysis results in each run and only at extremes of p so it should be easy to fix.

I don't believe the research community has the option to move away from FTT In meta-analysis as the authors have suggested as the logit transform is really bad at those same extremes of proportions, so even without addressing this, the FTT works better. Once this is fixed then perhaps there will only be the possibility to do a meta-analysis of proportions with metan as I cannot fix this in MetaXL but the caveat is that it does not matter that much.

Regards
Suhail

[David Fisher]

Hi Suhail,
Just to say that, for various reasons, the most recent version of metan does not yet have all the changes related to the QE model that we previously discussed (although the code is basically ready). I saw your recent email regarding the paper on double-arcsine, and am happy as ever to work with you to have metan return the most appropriate results.
Sorry for any confusion! I will let you know when the next release is imminent (as soon as possible!! but I am constantly being diverted by more urgent work...) and you can check that everything has been resolved as far as possible.
Regards,
David.

[Suhail Doi]

Hi David,

I notice with the new update to metan, zero quality entries for the QE model now always results in the error message "negative weights encountered". This is a Stata issue as discussed above but perhaps its good in a way given that we do not really want people to enter zero and lose the study information.

However, this will confuse users so I suggest this error message be changed to "zero entries for quality information not allowed" as that is the only way Stata can generate a negative weight. We actually do not recommend zero entries in this paper [1] and suggest adding 1 to all counts of safeguards if zero is encountered. Ultimately, whoever uses the QE model needs to have read up so i am not too concerned about giving more information other than this new message.

Also, in the help file under model specs the information for qwt is "variable containing Quality Effect weights" and I recommend this be updated to "variable containing study quality information" as the weights are generated by the model.

Regards
Suhail

Reference
1. Stone JC, Gurunathan U, Aromataris E, Glass K, Tugwell P, Munn Z, Doi SAR. Bias Assessment in Outcomes Research: The Role of Relative Versus Absolute Approaches. Value Health. 2021 Aug;24(8):1145-1149