Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Admetan (meta-analysis) and error with robust models

    Hi,
    I note that admetan has included the IVhet and quality effects models of meta-analysis which is great for the community of researchers. However a concern I have is that the use of the DL tau squared within these methods does not correspond to any actual probability distribution and therefore is not replaceable. However admetan does allow this to happen. For example take the dataset below:
    Code:
    * Example generated by -dataex-. To install: ssc install dataex
    clear
    input str11 studyname float(es lo95ci hi95ci qi rescaled lnes lnlo lnhi)
    "Leenders"    .99 .98    1   8     1 -.010050327 -.020202687           0
    "Nagura"      .91 .86  .95   8     1  -.09431065   -.1508229  -.05129331
    "Sahyoun"     .95 .83 1.08   7  .875  -.05129331   -.1863296   .07696108
    "Strandhagen"  .9 .81  .99   5  .625  -.10536055  -.21072103 -.010050327
    "Tucker"      .93 .84 1.03   6   .75  -.07257068   -.1743534  .029558774
    "Whiteman"    .84 .72  .99   6   .75   -.1743534    -.328504 -.010050327
    "Zhang"       .94 .91  .97 7.5 .9375  -.06187541  -.09431065  -.03045918
    end
    The IVhet model can be run correctly as:
    Code:
     admetan lnes lnlo lnhi, re(ivhet) rr forestplot(astext(70) boxscale(50) spacing(1) leftjustify lcols(studyname) dp(2))
    or wrongly as
    Code:
     admetan lnes lnlo lnhi, re(reml,ivhet) rr forestplot(astext(70) boxscale(50) spacing(1) leftjustify lcols(studyname) dp(2))
    I would consider that the second code above is no longer the IVhet model

    Similarly, the quality effects model can be run correctly as:
    Code:
     admetan lnes lnlo lnhi, qe(rescaled) rr forestplot(astext(70) boxscale(50) spacing(1) leftjustify lcols(studyname) dp(2))
    or wrongly as:
    Code:
     admetan lnes lnlo lnhi, re(reml,qe(rescaled)) rr forestplot(astext(70) boxscale(50) spacing(1) leftjustify lcols(studyname) dp(2))
    Again the second is no longer the quality effects model

    Any thoughts on this would be welcome

    Last edited by Suhail Doi; 15 Jun 2018, 01:16.
    Regards
    Suhail Doi

  • #2
    Dear Suhail,

    I am the author of admetan. Thankyou for getting in touch: it's great to hear from the author of one the new methods I have added!

    Could you expand a little on your comments?

    From reading your articles, it seemed to me that you were suggesting alternative weighting schemes. You make use of an additive heterogeneity estimator tausq, but I could see no specific reason why the DerSimonian-Laird estimator should be the only choice. Does the problem lie with the scale parameter/ICC? If so, could you explain where the derivation of your methods fails if (e.g.) the REML estimator of tausq is used in place of DerSimonian-Laird?

    Many thanks,

    David.

    Comment


    • #3
      Hi David,

      Great, thanks for creating admetan, and good to hear from its author. As you are aware, the most common approach to overdispersion in the fixed effect model is to generalize it to allow for extraneous variation – this of course is the classic random effects model. The later is a parametric approach and thus parameter estimates may be obtained by maximum likelihood. Thus the reml estimate of tau squared applies to this conventional random effects model only. However if the parametric assumption is deemed grossly inaccurate (which it is) then we have to deal with overdispersion using a quasi-likelihood approach. This is why we have to resort to the method of moments estimate, which requires assumptions only on the form of the mean and variance. Indeed the quasi-likelihood method of McCullagh or Wedderburn can be considered a similar moment approach. Thus only the DL tau squared can be used for creation of the overdispersion correction in both these models (IVhet and QE) if we are to adequately deal with the overdispersion expected in the fixed effect model under heterogeneity of treatment effects. The latter is what both these models do quite well and simulation studies (code on my researchgate page) demonstrate that they both have coverage close to the nominal level. If you substitute the reml tau squared this is unlikely to be the case. Thus if you could disable reml tau squared for both these models within admetan that would be good as under reml they no longer are IVhet nor QE models.
      Last edited by Suhail Doi; 15 Jun 2018, 05:53.
      Regards
      Suhail Doi

      Comment


      • #4
        Dear David,

        Just to more specifically answer your question regarding weighting schemes, yes the quality effects model uses a different weighting scheme while the IVhet model uses the exact same weights as the classic fixed effect model. However, you will note that the DL tau squared is not involved in the weights at all in either of these models even though both the models are suitable for heterogeneous data and aim to be alternatives to the classic random effects model. The DL tau squared only features in the overdispersion correction.
        Regards
        Suhail Doi

        Comment


        • #5
          Dear Suhail,

          I've been doing some reading around this, and it's a fascinating example of something that I thought was "basic" but actually is much more complicated. In particular, I've not come across the concept of quasi-likelihoods before.

          I can understand the basic idea of IVHet/QE: that the classic random-effects model assumes a normal distribution for the study effects, but that this is unjustified. Hence, as an alternative, you turn to quasi-likelihood, which I presume has some performance disadvantages when normality actually applies; but has the advantage of being "assumption-free". I have read Kulinskaya & Olkin's paper (Statistical Modelling 2014) but will need to study it some more to completely understand how your own work is based on it. Also, Kulinskaya & Olkin refer to normally-distributed effects throughout, as far as I can see?

          Thankyou for your second message -- you are of course quite right that tau-squared only occurs in the multiplicative factor rather than in the weights themselves. In order to have tau-squared derive from a method-of-moments framework, could you in theory use any of the tau-squared estimators discussed by DerSimonian and Kacker (Contemporary Clinical Trials 2007)?

          Although, as I say, I don't yet understand all the theory behind the IVHet/QE models, I am happy to restrict admetan to method-of-moments tausquared estimators with the IVHet/QE models based on your advice, when I next update (hopefully this summer).

          Regards,

          David.
          Last edited by David Fisher; 21 Jun 2018, 09:52.

          Comment


          • #6
            Dear David,

            Many thanks, that update would be great and look forward to it.

            Regarding Kulinskaya and Olkin's paper, the main idea we borrowed was the modeling of extraneous variation as a covariance from which an ICC is computed and then used as a scale parameter for overdispersion. You are right that they subsequently use these ideas in conventional frameworks and that is where we depart from the paper.

            I have not looked in much detail at DerSimonian and Kackers paper so cannot comment on that but given that all large scale simulations we carried out used the conventional DL measure in the conceptualization of the ICC (and demonstrates a very good result) it may be best to just restrict to this for the purposes of IVhet and QE. Further advances in these models would preferably look at what to do with few studies (less than five) rather than look for alternative scale parameters as this is not an exact science and tweaking a scale parameter that already solves the overdispersion problem may not add much more value and may sometimes create a problem (e.g. as we saw with the reml tau squared).

            NB I do intend to look at DerSimonian and Kackers paper and will email you if any new ideas emerge :-}
            Last edited by Suhail Doi; 22 Jun 2018, 06:11.
            Regards
            Suhail Doi

            Comment


            • #7
              Dear David,

              One more request (if possible) when you update in summer. I note that QE accepts rescaled quality (between zero and 1) and thus when subgroups are created these need to be rescaled again which is quite difficult for users. Could admetan allow raw scores (whatever the scale used) and then to rescale its simply study score divided by maximum score assigned within meta-analyzed studies. This maximum score will of course change as subgroups of different studies are created and thus it is better to allow rescaling within admetan from the raw scores. You will note that the maximum possible raw score is not relevant to this form of rescaling which is why any scale can be used without knowledge of its maximum and it works well. Also, with this form of rescaling there is always a study with a rescaled score of 1 (i.e. all rescaled scores start from 1 for the "best" study regardless of its status (good or poor) as these are relative scores [relative to the best study])
              Regards
              Suhail Doi

              Comment


              • #8
                Thanks Suhail. I'll look into doing this.
                David.

                Comment


                • #9
                  Dear David

                  I came across a blog post by Wolfgang Viechtbauer where he states that "What Doi et al. describe are RE models with different weights than the default ones". This made me realize that you have probably assumed the same when you code IVhet as re(ivhet) in admetan.
                  The IVhet is a fixed effect model as is the quality effects model. In fact the IVhet model IS the quality effects model except that by default quality is set to 1 for all studies (which is the default when quality is equal). Perhaps to make this clear it could be coded outside the re framework in your next update otherwise the assumption that it is a random instead of fixed effect model could be perpetuated?

                  Many thanks


                  "
                  Regards
                  Suhail Doi

                  Comment


                  • #10
                    Dear Suhail,

                    Thanks for your message. Could you link me to that Viechtbauer article?

                    I'm afraid I've not thought so deeply about this as you suppose. The admetan coding of re(ivhet) is simply an extension of the coding of other alternative models such as re(dl) or re(reml). This is partly for user simplicity: following previous Stata meta-analysis commands, the default option is either "traditional fixed-effects" or Mantel-Haenszel; and the next alternative is DerSimonian-Laird (specified using re alone with no suboption). I recognise that models are now entering the mainstream which deviate from the simple "tau-squared estimator" type: yours is one; others include Henmi & Copas (Stat Med 2010; 10.1002/sim.4029) and Biggerstaff & Tweedie (Stat Med 1997). The difficulty is how to code them other than using re(...); the risk is that the syntax becomes overly idiosyncratic. (I made an exception for Quality Effects because it requires specification of an additional variable.) I suppose I could introduce an option named, for instance, model(...), as a synonym for re(...)?

                    To help my understanding, could I ask what, to you, is the essence of a "fixed effect(s)" or "random effects" model? For instance, how would you classify the Henmi-Copas and Biggerstaff-Tweedie models I mentioned, where the confidence interval is derived separately from the point estimate? Could it not be argued that the IVHet model is also of this type?

                    Regards,

                    David.

                    Comment


                    • #11
                      Dear David,

                      Apologies for not responding sooner but was changing computers and that usually gets me tied up in knots for many days :-)

                      The statement by Viechtbauer was not an article - it was an opinion on his R blog

                      The essence of a random effects model is that it conceptualizes heterogeneity in terms of normally distributed random effects with a common variance. Thus both Henmi and Copas as well as Biggerstaff and Tweedie methods can be classified as random effects approaches and the authors actually state this in their papers. Thus use of re(xyz) is therefore fine for these methods.

                      My suggestion is to extend iv. Thus iv would be the classic fixed effect model and iv(het) would be the IVhet model. If any other fixed effect(s) model came along then it would be iv(xyz). The qe is fine as it is since it is the only model that uses extraneous data (other than ES plus SE) and it is unlikely that other such models would emerge in the near future.

                      Regards
                      Suhail

                      Edit: Actually model(xyz) as you have suggested instead of re would be fine too if it means that less work is needed on the codes
                      Last edited by Suhail Doi; 14 Oct 2018, 15:11.
                      Regards
                      Suhail Doi

                      Comment


                      • #12
                        Hi David,

                        Noted that you have updated admetan with model(ivhet) and the QE now uses the raw score and re-scales it. This is really great so now all these analyses can be run on Stata. To illustrate how these models perform I posted a simulation code here:
                        https://www.statalist.org/forums/for...91#post1472891

                        Might have to update the code to use the new syntax in admetan. Thanks again and great to see this running on Stata

                        Regards
                        Suhail
                        Regards
                        Suhail Doi

                        Comment


                        • #13
                          Hi Suhail,

                          Yes, that's correct. You can also run IVHet by typing admetan, ivhet. Also note that the previous syntax, re(ivhet), still works, so there is no need (beyond aesthetics!) to alter your simulation code. Sorry if that wasn't made clear in the documentation.

                          My "day job" as a trial statistician has rather taken over recently, but my colleagues and I did have an interesting discussion about fixed- vs random-effects models and the controversy between yourself and Wolfgang Viechtbauer regarding the IVHet model. We concluded that it rather depends on how you frame the question, and which modelling assumptions you start from ... but certainly the IVHet model and the recently-discussed multiplicative heterogeneity model (e.g. Mawdsley et al RSM 2017) both offer interesting new avenues for meta-analysis methodology :-)

                          Best wishes,

                          David.

                          Comment


                          • #14
                            Hi David,

                            Okay, I did not realize that so I am really grateful that you allowed it to be backward compatible while using the new codes and its really good to know that [admetan, ivhet] works as that makes it much easier for translation to end users as well.

                            I agree with your comments and that is exactly what one eminent journal editor once wrote to me (in his rejection letter of an interesting paper) saying: "Different individuals engaged in a meta analysis may have different goals and perspectives on what the parameter of interest is and the appropriate model to consider. You have a particular view and dismiss the view one may take when considering a random effects model. I'm sure not everyone would agree with you on the primary question that a meta analysis is trying to answer."

                            I think this sums up where the debate is at the moment. The most pressing need in clinical and public health research is to answer this question but it will not happen until the dissenting views are published so that a robust discussion can take place. Unfortunately that won't happen anytime soon because we are all very uncomfortable with where questioning random effects (old and new as Jim Hodges calls it in his book) takes us.

                            Once again thanks for creating admetan - its a great program and a great addition to Stata

                            Regards
                            Suhail

                            NB you may consider adding the Doi plot in the future :-))
                            Regards
                            Suhail Doi

                            Comment

                            Working...
                            X