STATA made new variables with underscore in front of them

[John William]

Dear helpers,

I am very new to STATA and currently making a meta-analysis of cohen's kappa value. i included 3 variables initially which were the cohen's kappa value (ES), lower CI (lci), and upper CI (uci).
Yesterday when i tried with 'meta' command, it replied with:
effect size variable ES must be > CI lower limit, lci

Then i used metan and/or admetan and it worked.

i tried to add more record (shown in screenshot below) and to recreate the forest plot with metan and admetan today, but it replied with:
admetan ES lci uci, studylabel(Author)
Effect size and/or confidence interval limits invalid;
order should be effect_size lower_limit upper_limit
Error in admetan.ProcessInputVarlist

but i still can recreate the forest plot if i don't add the new record.

my questions are:
1. why and how are there new variables with underscore?
2. why the admetan and metan command don't work anymore?

any help is greatly appreciated, and please forgive me if i violate any rules. Screenshot of a piece of my record is shown below.

[John William]

i just found out row 73 was incorrect using this command
browse if !(lci <= ES & ES <= uci)

so question number 2 is answered

[Nick Cox]

1. why and how are there new variables with underscore?

See the help:

By default, metan adds the following new variables to the data set:

      _ES        Effect size (ES)
      _seES      Standard error of ES 
                 or, when OR or RR are specfied: 
      _selogES   the standard error of its logarithm 
      _LCI       Lower confidence limit for ES
      _UCI       Upper confidence limit for ES
      _WT        Study percentage weight
      _SS        Study sample size

[John William]

ah i see. Thank you Nick Cox for your help.

[David Fisher]

Hi John William ,

Although Nick has (of course) been very helpful, your OP demonstrates a few different issues which might be helpful to others, so I'm going to do a quick but slightly deeper dig.

Firstly: excluding the Evangelista study observations, you say that meta didn't work. I suspect that the actual error message you saw was as follows:

confidence intervals not symmetric
    CIs defined by variables lci and uci must be symmetric and based on a normal distribution. If you are working with effect sizes in the
    original metric, such as odds ratios or hazard ratios, with meta set, you should specify the effect sizes and CIs in a normalizing
    metric, such as the log metric.

    The default tolerance to determine the CI asymmetry is 1e-6. Effect sizes and their CIs are often reported with limited precision that,
    after the normalizing transformation, may lead to asymmetric CIs. In that case, the default of 1e-6 may be too stringent. You may
    loosen the tolerance by specifying option civartolerance().
r(459);

This is telling you that meta detected a potential issue with your input data, such that one or more confidence limits is further (or closer) from the effect size estimate (i.e. the central point of a symmetric interval) than the other limit. Although meta does not tell you which observations this applies to, it is easy to find out ourselves (note that I've labelled the points in the scatter plot by observation number, because the authors are non-unique):

* Example generated by -dataex-. For more info, type help dataex
clear
input str17 author float(es lci uci)
"Shah, 2016"         .81      .75      .87
"Shahbazipar, 2018"   .9      .81      .99
"Shahbazipar, 2018"  .92      .83      .99
"Shahbazipar, 2018"  .94      .85        1
"Shahbazipar, 2018"  .24      .15      .62
"Shekarchi, 2018"   .884 .8778726 .8901274
"Shekarchi, 2018"   .745 .7419567 .7480432
"Shekarchi, 2018"    .66 .6574658 .6625342
"Shekarchi, 2018"   .343  .340923  .345077
"Shekarchi, 2018"   .826  .691296 .9607039
"Tourghabe, 2018"   .272     .115     .429
"Tourghabe, 2018"   .499     .271     .725
"Tourghabe, 2018"   .691     .503     .877
"Tourghabe, 2018"   .469     .045     .892
"Unlüer, 2014"     .861 .8522202 .8697798
"Yesilaras, 2014"   .388     .022     .664
"Evangelista, 2016"   .7       .6      .78
"Evangelista, 2016"  .51      .37      .62
"Evangelista, 2016"  .53      .39      .63
"Evangelista, 2016"  .61       .5      .74
"Evangelista, 2016"  .54      .43      .71
"Evangelista, 2016"  .65      .56      .74
"Evangelista, 2016"  .29       .9      .47
"Evangelista, 2016"  .53      .23      .83
end

gen es_to_lci = es - lci
gen uci_to_es = uci - es
gen obs = _n
scatter es_to_lci uci_to_es if author!="Evangelista, 2016", mlabel(obs)

The scatter plot makes it clear that the problematic observation is number 5, which we therefore list:

. list in 5, ab(20)

     +-------------------------------------------------------------------+
     |            author    es   lci   uci   es_to_lci   uci_to_es   obs |
     |-------------------------------------------------------------------|
  5. | Shahbazipar, 2018   .24   .15   .62         .09         .38     5 |
     +-------------------------------------------------------------------+

The upper limit is much further from the central point (0.62) than is the lower limit (0.09). Either this data has been inputted incorrectly, or it comes from a model in which the data were transformed prior to (or as part of) analysis. If the latter, it may be best to use the estimated standard error in preference to the confidence limits, if available. I don't know enough about Kappa statistics to advise about modelling and/or data transforms, I'm afraid.


You also say "I then tried metan and/or admetan and it worked". Two things to note here:
a) admetan is no longer supported, and is deprecated in favour of metan. For the benefit of other readers: please use metan; there is no good reason to favour admetan.
b) The reason it worked, is simply that metan does not (currently) implement this check for asymmetric confidence limits. It assumes that the data is correct, and proceeds to back-calculate the standard error on the basis of that assumption. Whether or not this is good enough in practice, depends on the reason for the original asymmetry: was it data input error (not OK!) or as a result of some data transformation (possibly but not necessarily OK).


Next: with the Evangelista observations included, you saw a different error:

. metan es lci uci
Effect size and/or confidence interval limits invalid;
order should be effect_size lower_limit upper_limit
Error in metan.ProcessData
r(9);

As you correctly observed in post #2, this is due to a true data input error -- there is no other explanation.

Finally, to answer the question you posed in #1 (acknowledging that Nick has already done so!): meta and metan (and indeed admetan) all leave behind a set of additional "internal" variables containing the information the program itself has calculated, based on your input data and the assumptions used within the calculation code.

I hope this is helpful.
BW,
David.


P.S. Nick Cox : it's a very minor point, but it looks like your version of metan is out-of-date, as the wording of that section of documentation is now quite different.

[Nick Cox]

David Fisher Thanks for chiming in so authoritatively.

And thanks for the correction. It doesn't matter one hoot, but I have never run a meta-analysis, ever. I must have downloaded metan previously to see if there was a quick answer I could give to someone's question.