Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Type M and Type S errors

    Gelman and Carlin introduced a command in R (retrodesign()) to calculate Type M and Type S errors in their 2014 paper, entitled "Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors" (link here: http://www.stat.columbia.edu/~gelman...ower_final.pdf ). The command itself only takes a few inputs:

    Code:
    retrodesign <- function(A, s, alpha=.05, df=Inf, n.sims=10000){
      z <- qt(1-alpha/2, df)
      p.hi <- 1 - pt(z-A/s, df)
      p.lo <- pt(-z-A/s, df)
      power <- p.hi + p.lo
      typeS <- p.lo/power
      estimate <- A + s*rt(n.sims,df)
      significant <- abs(estimate) > s*z
      exaggeration <- mean(abs(estimate)[significant])/A
      return(list(power=power, typeS=typeS, exaggeration=exaggeration))
    }
    A is the hypothesized true effect size, s is the standard error of the estimate, alpha is the significance threshold, and df is the degrees of freedom. qt is the quantile function in R, and pt is the distribution function. Unfortunately, I lack the programming chops to translate their code to Stata. If anyone has suggestions (aside from simply using R), I'd certainly be interested.

  • #2
    Can you give us some examples and what the results should be? Somebody who wants to adapt the code would have a better chance if they had ways to assess whether the results their code is giving are correct.

    Incidentally, R can be called from Stata. Check out the user-written commands rsource and rcall.
    -------------------------------------------
    Richard Williams, Notre Dame Dept of Sociology
    Stata Version: 17.0 MP (2 processor)

    EMAIL: [email protected]
    WWW: https://www3.nd.edu/~rwilliam

    Comment


    • #3
      Sure.

      Here's what the results look like in a situation where the true effect size, A, equals 0.1, the standard error, s, equals 3.28, and alpha equals 0.05:

      Power: 0.05010648
      Type-S: 0.4644228
      Type-M: 76.82889

      Or to use another example from their paper: If A = 2 and s = 8.1 and alpha = 0.05, then power = .05701294, Type-S (the probability of getting the sign wrong) = .2396177, and Type-M (the exaggeration factor) = 9.571811.

      Comment


      • #4
        Code:
        *! version 1.0.0 25oct2017 daniel klein
        program retrodesigni // , rclass
            version 14.2
            
            syntax anything                     ///
            [ ,                                 ///
                Level(cilevel)                     ///
                DF(numlist integer max = 1 > 0) ///
                NSIMs(integer 10000)             ///
            ]
            
            if (`nsims' < 1) {
                display as err "nsims() invalid -- " _continue
                error 125
            }
            
            gettoken b     anything : anything
            gettoken se anything : anything
            if (`"`anything'"' != "") {
                error 198
            }
            capture noisily confirm number `b'
            if (_rc) {
                exit 198
            }
            capture noisily assert (`se' > 0)
            if (_rc) {
                exit 198
            }
            
            tempname power typeS ex
            mata : retrodesigni_ado()
            
            display
            display as txt %12s "b"         " = " as res `b'
            display as txt %12s "se"         " = " as res `se'
            display as txt %12s "level"     " = " as res `level'
            display
            display as txt %12s "power"     " = " as res r(power)
            display as txt %12s "typeS"     " = " as res r(typeS)
            display as txt %12s "Exagg."     " = " as res r(exaggeration)
        end
        
        version 14.2
        
        mata :
        
        void retrodesigni_ado()
        {
            real scalar b, se, alpha, df, nsims
            real scalar z, lo, hi, power, typeS, exagg
            real colvector coef
            
            b         = strtoreal(st_local("b"))
            se         = strtoreal(st_local("se"))
            alpha     = 1-strtoreal(st_local("level"))/100
            df         = strtoreal(st_local("df"))
            nsims     = strtoreal(st_local("nsims"))
            
            if (missing(df)) {
                z         = invnormal(1-alpha/2)
                lo         = normal(-z-b/se)
                hi         = 1-normal(z-b/se)
                coef     = b:+se:*rnormal(nsims, 1, 0, 1)
            }
            else {
                z         = (-1)*invttail(df, 1-alpha/2)
                lo         = 1-ttail(df, -z-b/se)
                hi         = ttail(df, z-b/se)
                coef     = b:+se:*rt(nsims, 1, df)
            }
            
            power = hi+lo
            typeS = lo/power
            exagg = mean(abs(select(coef, abs(coef):>(se*z))))/b
            
            st_rclear()
            st_numscalar("r(power)", power)
            st_numscalar("r(typeS)", typeS)
            st_numscalar("r(exaggeration)", exagg)
        }
        
        end
        Do not expect the results to match exactly. I have used the normal distribution the case of infinite degrees of freedom. Also the exaggeration is found by random draws from the normal (or t-) distribution, so it will differ from run to run.

        Ah, I forgot. The syntax is

        Code:
        retrodesigni A s [ , level(#) df(#) nsims(#) ]
        Here is the demonstration with the values supplied

        Code:
        set seed 42
        retrodesigni  0.1 3.28
        retrodesigni 2 8.2
        which gives

        Code:
        . set seed 42
        
        . retrodesigni  0.1 3.28
        
                   b = .1
                  se = 3.28
               level = 95
        
               power = .05010648
               typeS = .46442284
              Exagg. = 76.311976
        
        . retrodesigni 2 8.2
        
                   b = 2
                  se = 8.2
               level = 95
        
               power = .05684226
               typeS = .24219428
              Exagg. = 9.6565924
        
        . 
        end of do-file
        Best
        Daniel
        Last edited by daniel klein; 25 Oct 2017, 10:30.

        Comment


        • #5
          Amazing! Thanks, Daniel.

          Comment


          • #6
            daniel klein , that could be a handy program. You might consider wring a short help file and sending it off to SSC. (Although writing help files is what I like least about programming!) Give it a more descriptive name. Maybe typemserror.
            -------------------------------------------
            Richard Williams, Notre Dame Dept of Sociology
            Stata Version: 17.0 MP (2 processor)

            EMAIL: [email protected]
            WWW: https://www3.nd.edu/~rwilliam

            Comment


            • #7
              Thanks as usual to Kit Baum, a revised version of the program in #4, rdesigni, is now available from the SSC.

              In Stat, type

              Code:
              ssc install rdesigni
              Most options have been renamed to better match Stata terminology, the output has been changed and more results are returned in r(). The example above can be replicated with one call now

              Code:
              . set seed 42
              
              . rdesigni (0.1 2) (3.28 8.2) , parallel
              
              Design analysis
              --------------------------------------------------------------------
                                                              type S     type M
                   alpha         se          D      power      error      error
              --------------------------------------------------------------------
                     .05       3.28         .1   .0501065   .4644228   76.31198
                     .05        8.2          2   .0568423   .2421943   9.656592
              --------------------------------------------------------------------
              Best
              Daniel


              Gelman, Andrew, Carlin, John (2014). Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science, 9, 641-651.

              Comment

              Working...
              X