Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Anyone knows what is an "adjusted within-R2"?

    Dear all,
    I'm trying to understand what is the adjusted R2 in the context of a fixed effects regression (xtreg, xtivreg2) and would like some feedback/comments. My problem is that the formula I thought makes more sense is not used anywhere in Stata, so I'm not sure of what's going on...

    Background: explanation of the adjusted R2 from Quora

    Common Formulas:
    rss: Sum-of-squares of the residuals
    tss: Sum-of-squares of the dependent variable (after demeaning)
    wss: Within-sum-of-squares; similar to TSS but instead of demeaning wrt to the entire sample, demean the dependent variable within each group.

    N: Number of obs.
    L: Number of fixed effects (including the constant)
    K: Number of other regressors besides the fixed effects (and the constant)

    R2 = 1 - rss / tss
    R2 Adjusted = 1 - [ rss / (N-L-K) ] / [ tss / (N-1) ]
    R2 Within = 1 - rss / wss

    Controversial Formula:
    R2 Within Adjusted = 1 - [ rss / (N-L-K) ] / [ wss / (N-L) ]

    The above is how I would write it based on the Quora link. Within each term in brackets, the numerator is is divided by the degrees of freedom available after computing that number.

    This formula has two nice properties:
    1. With no regressors besides the fixed effects (K=0), it collapses to the within-R2 (which is also zero).
    2. If there is only one fixed effect (L=1), it collapses to the adjusted-R2.

    Problem: It's not what other packages do, not sure if I'm missing something...

    1) xtreg computes it as

    1 - [ rss / (N-L-K) ] / [ wss / (N-1) ]
    (The only difference is -1 instead of -L)

    2) (areg doesn't compute within estimates)

    3) xtivreg2 (ssc) computes it as
    1 - [ rss / (N-L-K) ] / [ wss / (N-0) ]

    Note how the only difference is whether to substract the number of fixed effects L, substract 1, or substract 0.



    Any opinions? Is that a bug in the other programs? Should I follow what they do?

    Best,
    Sergio





    Previous discussions
    [1] http://www.stata.com/statalist/archi.../msg00615.html
    [2] http://www.stata.com/statalist/archi.../msg00201.html

    Alternative Formulas (less relevant)
    [1] http://stats.stackexchange.com/quest...-r/63766#63766
    [2] http://stats.stackexchange.com/quest...re/55932#55932
    [3] http://mlrv.ua.edu/2007/Vol33_1/Leach-Henson-p1-11.pdf

  • #2
    Sergio: Stata computes

    R2 = SSreg/ SStotal

    where

    SStotal = total sum of squares
    SSreg = regression sum of squares (or explained sum of squares)
    ____________________________________


    Adjusted R2 = R2 - [(1- R2)*(p/n-p-1)]


    where p is the number of regressors (excluding constant), and n is the number of observations. After running the regressions, you can always compute the R2 and adjusted R2 manually using the following commands

    Code:
    scalar R2 = e(mss)/ (e(mss)+ e(rss))
    di R2
    scalar aR2 = R2 - ((1- R2)* (e(df_m) / (e(N)-e(df_m)-1)))
    di aR2
    Stata will simply implement the above formula using the output from a fixed effects regression (as it would with OLS). The following example illustrates


    Code:
    clear 
    input y x1 x2 x3 x4 id year
    78 19  45 44  15   1 1
    23 19  17 47  72   1 2
    10 19  32 62  65   1 3
    34 19  11 21  20   1 4
    77 19  42 23  100  1 5
    91 55  12 13  14   2 1
    62 55  27 37  47   2 2
    33 55  13 14  15   2 3
    16 55  58 68  78   2 4
    99 55  80 90  70   2 5
    20 51  18 62  82   3 1
    38 39  39 11  63   3 2
    40 87  46 93  90   3 3
    56 03  64 80  28   3 4
    73 200 88 103  36  3 5
    115 70  85 18  85  4 1
    49 51  67 22 76    4 2
    57 28  49 26  96   4 3
    74 32  31 41  77   4 4
    110 16  12 60  80  4 5
    24 112  26 20  26  5 1
    111 123 81 82  37  5 2
    64 45  59 39  49   5 3
    39 72  31 29  92   5 4
    79 80  16 77  107  5 5
    37 47  19 89  12   6 1
    23 61  38 45  22   6 2
    32 83  82 83  66   6 3
    120 115 91 116 108 6 4
    7 150  54 93  72   6 5
    92 28  30 41  90   7 1
    100 28 40 96 102   7 2
    108 28  50 29  59  7 3
    116 28  60 42  76  7 4
    128 28  70 80  94  7 5
    39 7  55 103  106  8 1
    51 50  27 98  62   8 2
    73 61  19 81  74   8 3
    94 86  112 99  53  8 4
    103 99  67 102  10 8 5
    89 80  105 54  69  9 1
    62 90  97 108  62  9 2
    13 100  102 92  39 9 3
    100 110 81 66  85  9 4
    115 120 92 50  67  9 5
    40 37  75 19  14  10 1
    92 65  87 5  34   10 2
    56 72  92 15  40  10 3
    119 128  23 21 63 10 4
    84 80  82 67  29  10 5
    end


    Running the fixed effects model

    Code:
    
    . xtset id year
           panel variable:  id (strongly balanced)
            time variable:  year, 1 to 5
                    delta:  1 unit
    
    . xtreg y x*, fe
    
    Fixed-effects (within) regression               Number of obs      =        50
    Group variable: id                              Number of groups   =        10
    
    R-sq:  within  = 0.1448                         Obs per group: min =         5
           between = 0.0000                                        avg =       5.0
           overall = 0.0538                                        max =         5
    
                                                    F(4,36)            =      1.52
    corr(u_i, Xb)  = -0.3074                        Prob > F           =    0.2161
    
    ------------------------------------------------------------------------------
               y |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
    -------------+----------------------------------------------------------------
              x1 |   .2398038   .1525894     1.57   0.125    -.0696618    .5492694
              x2 |   .2376799   .2046493     1.16   0.253    -.1773682     .652728
              x3 |   .0511757   .2110712     0.24   0.810    -.3768965     .479248
              x4 |   .0676377   .1813813     0.37   0.711    -.3002206     .435496
           _cons |   32.27467   17.37543     1.86   0.071    -2.964345    67.51368
    -------------+----------------------------------------------------------------
         sigma_u |  22.933562
         sigma_e |   31.13536
             rho |  .35172042   (fraction of variance due to u_i)
    ------------------------------------------------------------------------------
    F test that all u_i=0:     F(9, 36) =     1.94               Prob > F = 0.0774
    
    . scalar R2 = e(mss)/ (e(mss)+ e(rss))
     
    . scalar aR2 = R2 - ((1- R2)* (e(df_m) / (e(N)-e(df_m)-1)))
    
    . di R2
    .14475503
    
    . di aR2
    -.16408343

    The adjusted R2 can be seen on the ereturn list below. Note that, in this case p= 13 (df_m) because you have 4 regressors + (10-1) individual dummies.

    Code:
    . ereturn list
    
    scalars:
                   e(rank) =  5
                    e(rss) =  34898.78424281887
                   e(df_m) =  13
                   e(df_r) =  36
                     e(r2) =  .1447550276722099
                   e(rmse) =  31.13536049914366
                    e(mss) =  5906.815757181132
                   e(r2_a) =  -.1640834345572697
                     e(ll) =  -234.6515333278909
                   e(ll_0) =  -238.5607166714206
                    e(tss) =  59530.49999999999
                      e(N) =  50
                   e(df_b) =  4
                   e(r2_w) =  .1447550276722099
                   e(df_a) =  9
                      e(F) =  1.523300681328726
                    e(F_f) =  1.938003049221467
                   e(Tbar) =  5
                   e(Tcon) =  1
                  e(g_min) =  5
                    e(rho) =  .3517204164928344
                  e(sigma) =  38.66987126953578
                e(sigma_e) =  31.13536049914366
                   e(r2_b) =  .0000103737332557
                   e(r2_o) =  .0537633657871388
                   e(corr) =  -.3074031244891103
                e(sigma_u) =  22.93356209991883
                     e(ui) =  22.93356209991883
                    e(N_g) =  10
                  e(g_max) =  5
                  e(g_avg) =  5
    
    macros:
                e(cmdline) : "xtreg y x*, fe"
                    e(cmd) : "xtreg"
           e(marginsnotok) : "E U UE SCore STDP XBU"
                e(predict) : "xtrefe_p"
                  e(model) : "fe"
                 e(depvar) : "y"
                   e(ivar) : "id"
                    e(vce) : "conventional"
             e(properties) : "b V"
    
    matrices:
                      e(b) :  1 x 5
                      e(V) :  5 x 5
    
    functions:
                 e(sample)   
    I am not familiar with the algorithm used by other software, but Stata's algorithm is consistent (FE and OLS).

    Comment


    • #3
      Hi Andrew,

      Thanks for the detailed reply!
      I agree that in general both the R2 and adjusted R2 are consistent, but why xtreg uses that formula makes no sense for me:
      1. It is negative if you run it without any regressors.
      2. The entire point of the adjusted R2 is to fix the bias of the standard R2, which xtreg's formula doesn't do at all. You cannot use the same formula if to obtain the within total-sum-of-squares you used e(df_a) instead of just 1 degrees of freedom. Note that fixed effects e(df_a) can be HUGE so the effect is not trivial at all.

      Comment


      • #4
        I have taken a look at EViews. It reports the R2 from OLS with N-1 individual dummies after running a FE regression. Experience indicates that researchers use this R2 estimate (after all, it makes more sense).

        Code:
        . reg y x* i.id
        
              Source |       SS       df       MS              Number of obs =      50
        -------------+------------------------------           F( 13,    36) =    1.95
               Model |  24631.7158    13  1894.74737           Prob > F      =  0.0562
            Residual |  34898.7842    36  969.410673           R-squared     =  0.4138
        -------------+------------------------------           Adj R-squared =  0.2021
               Total |     59530.5    49  1214.90816           Root MSE      =  31.135
        
        ------------------------------------------------------------------------------
                   y |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
        -------------+----------------------------------------------------------------
                  x1 |   .2398038   .1525894     1.57   0.125    -.0696618    .5492694
                  x2 |   .2376799   .2046493     1.16   0.253    -.1773682     .652728
                  x3 |   .0511757   .2110712     0.24   0.810    -.3768965     .479248
                  x4 |   .0676377   .1813813     0.37   0.711    -.3002206     .435496
                     |
                  id |
                  2  |   5.516458     20.443     0.27   0.789    -35.94386    46.97678
                  3  |  -19.72369   21.75645    -0.91   0.371    -63.84782    24.40044
                  4  |   25.48315   21.16516     1.20   0.236    -17.44178    68.40809
                  5  |  -1.339484   22.04283    -0.06   0.952    -46.04442    43.36545
                  6  |  -26.87833   23.29873    -1.15   0.256    -74.13035    20.37368
                  7  |   54.39856   20.76321     2.62   0.013     12.28882     96.5083
                  8  |   7.928215   22.87142     0.35   0.731    -38.45717    54.31359
                  9  |  -6.158041    24.9925    -0.25   0.807    -56.84518     44.5291
                 10  |   11.91863   23.21828     0.51   0.611    -35.17024    59.00749
                     |
               _cons |   27.16012   18.15717     1.50   0.143    -9.664322    63.98457
        ------------------------------------------------------------------------------
        Last edited by Andrew Musau; 10 Apr 2015, 13:04.

        Comment


        • #5
          That would be the same R2 reported by areg, and makes sense if you don't have many fixed effects. But in a context of many fixed effects, most of the R2 will be explained by the FEs, and that's where the within R2 makes sense.

          I guess the choice between standard-R2 and within-R2 depends on your objectives and how you think of your fixed effects. But in any case, the adjusted R2 reported by xtreg,fe is simply misleading in my opinion, as it has a *huge* downwards bias.

          Comment


          • #6
            As an update, I did a few Montecarlos that showed that the xtreg_fe formula is quite unbiased, in particular under two conditions:
            1) if T is small (e.g. each individual has on avg. only 3-5 observations)
            2) if N is not so large (in which case it doesn't matter whether you compute R2 or adjusted R2)

            I know changing the reported estimates would break backwards compatibility, but if the adjusted R2 is clearly wrong, why not fix it?

            Sergio

            Comment


            • #7
              I agree with your take on the adjusted within R2. Usually, you do not see people reporting it (in most cases, it has no meaning). To see this, you can look at the formula

              Adjusted R2 = R2 - [(1- R2)*(p/n-p-1)]

              The problem arises because n (the number of observations) is a function of p (the model degrees of freedom). In simple terms, if you have more individuals in the panel, you have more data. The usual case is that you have a few regressors (say 7), and a large cross-sectional dimension (100+). Since fixed effects regression regards the individual effects as parameters to be estimated, you will have a problem with the within adjusted R2.
              (p/n-p-1) in the formula above is usually greater than 1 and [(1- R2)*(p/n-p-1)] exceeds the within R2 so that you get a negative adjusted R2.

              Stata usually does not display the value of the within adjusted R2 in the main result window - it is just a redundant statistic in the ereturn list. There are other ways of assessing what value a single regressor adds to the model prior to running it.


              Comment

              Working...
              X