[Helen Jiahuan He]

Hello,
I ran into an error while I was installing the metan package (please see the screenshot below and the attachment). Basically, I got an error from scc install saying "apparent error in package file for metan" after running the command "ssc install metan, all replace". I am using Stata version 17.0BE, and I also tried Stata version 18.0BE, but I got the same error. I wonder whether someone could help me to address this situation. Thank you very much in advance!


Best,
Helen

[Suhail Doi]

Hi David

With this dataset:

* Example generated by -dataex-. For more info, type help dataex
clear
input str21 study int year str11 location_firstdose int(txanodeath1m txadeath1m planodeath1m pladeath1m) byte qi
"CRASH-2 2010"        2010 "Hospital"    8597 1463 8454 1613 34
"Chakroun-Walha 2019" 2019 "Hospital"      69   27   65   19 28
"CRASH-3 2019"        2019 "Hospital"    3758  855 3622  892 33
"Rowell 2020"         2020 "Prehospital"  563   94  256   53 34
"Guyette 2020"        2020 "Prehospital"  406   41  407   49 35
"PATCH 2023"          2023 "Prehospital"  540  113  498  139 34
end

There is an error when running this code:

metan txadeath1m txanodeath1m pladeath1m planodeath1m  , or qe(qi) study( study )  forestplot(astext(75) textsize(100) boxscale(55) spacing(1.5) leftjustify dp(2) range(0.4 1.3))  extraline(no) hetinfo(isq) by( location_firstdose )

It says:
Error encountered whilst calculating quality weights

Any thoughts?

Regards
Suhail

[Suhail Doi]

Hi David

I just realized that metan does not report cumulative results in _ES _seES _LCI _UCI when the cumulative option is selected and this is different from what admetan does (which respects whichever option is selected). This makes things very difficult when metan is used within simulations or other ado files so would it be possible to revert back to the former behavior or is there an option to activate this? It also seems logical (at least to me) that these new variables report what the output reports?

Many thanks
Regards
Suhail

[David Fisher]

Dear all,

With thanks as ever to Kit Baum, an updated version of metan (v4.06 12oct2022) is now available via SSC.

The following bugs have (hopefully) been fixed:

• Proportion data: fixed bug which caused an exit error when proportion was used with cumulative | influence and with saving() | clear
• Proportion data: proportion with counts now gives the correct denominators for subgroups in the forest plot (they previously appeared doubled)
• When hetinfo(tausq) was used, the value of tau-squared from the first subgroup was repeated across all other subgroups in the forest plot; this has been fixed
• With Quality Effects model, the code has been improved so as to avoid negative weights via rounding error when one or more weights is exactly zero.

Other improvements:

• Added a new option labtitle() to over-ride the default "Subgroup and Study" title with by()
• Minor improvements to the handling and display of zero cells in special cases (e.g. by(), cumulative, influence)
• Major re-ordering of code and subroutines to hopefully improve readability and maintenance going forward. The codebase has effectively been split into three parts: metan.ado, metan_analysis.ado and metan_output.ado; whose respective content should be self-explanatory.

As ever, thanks to all users who have reported issues. In particular, I must apologize if sometimes I do not respond due to workload. When preparing an update, I do try to look back through my correspondence and check that important bug reports or requests have been incorporated. So, please continue to stay in touch!

Best wishes,

David.

[dr Mheissen]

Hi David, thanks for this update and your huge efforts.
how can I get the between subgroups p value/Z to check the statistically differences between them?

[Suhail Doi]

Hi David,

Thanks - I thought that was the QE update but good to know that it is not and look forward to that.

Yes, the FTT issue as raised may be true but is not really that significant because our simulation runs varying study numbers (per MA) and randomly selecting study size between 1 to 100 events (size=event/p) only brings this up in <1% of pooled meta-analysis results in each run and only at extremes of p so it should be easy to fix.

I don't believe the research community has the option to move away from FTT In meta-analysis as the authors have suggested as the logit transform is really bad at those same extremes of proportions, so even without addressing this, the FTT works better. Once this is fixed then perhaps there will only be the possibility to do a meta-analysis of proportions with metan as I cannot fix this in MetaXL but the caveat is that it does not matter that much.

Regards
Suhail

[David Fisher]

Hi Suhail,
Just to say that, for various reasons, the most recent version of metan does not yet have all the changes related to the QE model that we previously discussed (although the code is basically ready). I saw your recent email regarding the paper on double-arcsine, and am happy as ever to work with you to have metan return the most appropriate results.
Sorry for any confusion! I will let you know when the next release is imminent (as soon as possible!! but I am constantly being diverted by more urgent work...) and you can check that everything has been resolved as far as possible.
Regards,
David.

[Suhail Doi]

Hi David,

I notice with the new update to metan, zero quality entries for the QE model now always results in the error message "negative weights encountered". This is a Stata issue as discussed above but perhaps its good in a way given that we do not really want people to enter zero and lose the study information.

However, this will confuse users so I suggest this error message be changed to "zero entries for quality information not allowed" as that is the only way Stata can generate a negative weight. We actually do not recommend zero entries in this paper [1] and suggest adding 1 to all counts of safeguards if zero is encountered. Ultimately, whoever uses the QE model needs to have read up so i am not too concerned about giving more information other than this new message.

Also, in the help file under model specs the information for qwt is "variable containing Quality Effect weights" and I recommend this be updated to "variable containing study quality information" as the weights are generated by the model.

Regards
Suhail

Reference
1. Stone JC, Gurunathan U, Aromataris E, Glass K, Tugwell P, Munn Z, Doi SAR. Bias Assessment in Outcomes Research: The Role of Relative Versus Absolute Approaches. Value Health. 2021 Aug;24(8):1145-1149

[Suhail Doi]

Hi David,

By adjusting qi downwards I can confirm that the negative value encountered is between -0.0000000000000001 (fifteen zeroes) and -0.00000000000000001 (sixteen zeroes)

Therefore it confirms your diagnosis and all that needs to be done is what I suggested in the previous post

Regards
Suhail

[Suhail Doi]

Hi David,

Thanks. Looks like all the computations are working as intended so must be a rounding problem. Best then to add a line that replaces w_hat_doubledash_j to 0 if it is <0 at the point where it is computed

This should solve this and given that it will be a very very small negative number should create no issues

Regards
Suhail

[David Fisher]

Hi Suhail,

Agree with your correction to my expression for w_hat_doubledash.

We then see that: sum(w_hat_doubledash_j) = sum[ Q_j/v_j ] + sum[ tau_hat_j ] = sum[ Q_j/v_j ] + sum[ (1-Q_j)/v_j ] = sum[1/v_j] as you say ("sum of weights remains constant").

However, I still see no reason to doubt my original diagnosis. My code as currently written does not formally check that the sum of weights has remained constant. We both know that it is algebraically true; but due to computer precision/rounding (see e.g. Bill Gould's presentation, in particular Slide 37: https://www.stata.com/meeting/boston...on14_gould.pdf) it may not be true according to Stata's computations (at least, not in its currently-coded form, which involves subtraction of two near-identical quantities).

The current code can be seen by opening "metan.ado" in a text editor such as Notepad++ and jumping to line 9265 ... but I admit that the more I look at it, the clunkier it appears, and I am a little embarrassed by it!!

Best wishes,

David.


P.S. Sorry, I should say: of course if you still have concerns please formulate them as a worked example and I will look into it!

[Suhail Doi]

Hi David,

What I meant by sum of weights remains constant is that:

sum[1/v_j] = sum[w_hat_doubledash_j]
(Note that the hatted weights are not normalized weights)

Regarding the second point, thanks for the simplification. The first one is correct:

tau_hat_j = Q_j * { sum[ (1-Q_j)/v_j ] / sum[Q_j] }

The second one that follows should be updated to:

w_hat_doubledash_j = (Q_j/v_j) + tau_hat_j = Q_j { (1/v_j) + sum[ (1-Q_j)/v_j ] / sum[Q_j] }

(In our paper w_j actually referred to the normalized RE weight and not 1/v_j)


Regards
Suhail

[David Fisher]

Hi Suhail,

Could you clarify what you mean by "sum of weights remains constant before and after application of qwt" ? Do you mean, in the metan output? If so, could you give a working example? Or do you mean theoretically -- in which case could you point to the relevant section of your article?


On a related note, I did a bit of algebra with the aim of correcting the rounding-error issue, and it works out (I think) that:

tau_hat_j = Q_j * { sum[ (1-Q_j)/v_j ] / sum[Q_j] }

so that:

w_doubledash_j = (Q_j/v_j) + tau_hat_j = Q_j { w_j + sum[ (1-Q_j)/v_j ] / sum[Q_j] }

If correct, this will greatly simplify calculations and hopefully solve the rounding-error issue.


Thanks,

David.

[Suhail Doi]

Hi David,

After giving this some more thought, I am not sure if this can be due to rounding because the sum of weights remains constant before and after application of qwt so perhaps the next thing to check is for a bug at the step where conversion of the user input to a rank takes place?

Regards
Suhail

[Suhail Doi]

Hi David,

Yes, I just checked all the calculations and negative weight is not possible if the Q_dash_j is applied so must be a rounding issue. And also what you have implemented is correct, Q_dash_j should always be implemented after the re-scaling (dividing by the maximum) of the qwt entered by the user regardless of if all the qwt are 1 or not. If they are 1, then Q_dash_j also defaults to 1 so there will be no issue at all.

PS: Interesting observation and I know why it failed on your computer - try creating the rank yourself by running:

gen ranknew = qi/3

Now run:

metan c1 nc1 c2 nc2 , or model(qe,qwt(ranknew))  study( study )  forestplot(astext(85) textsize(100) boxscale(35) spacing(1) leftjustify range(0.3 2) dp(2))  extraline(yes) hetinfo(q pvalue tausq)

and voilà .... no error message
This confirms the rounding issue