How to extract multiple values from generated results

[Jakub Cirkowski]

Hello,

I am wondering if somebody can help me with my loop. I am trying to generate a list of incremental R2 for each variable in every year. I need it to examine the time trend in them. I built the following code but it doesn't work. First of all,I don't know why each variable has one incremental R2 for all years - which doesnt make sense. Secondly, I am not sure how I can generate the table including the values from generated 896 incremental R2 equations (56 years times 16 incremental R2 in every year) .

I use the following code and obtain following results. My desired outcome is a list incremental R2 for each variable for every year . From this form I could run the regression on time.

I would be very grateful if somebody could point out errors in my code and help me with generating the table.

    gen incatpr =.
    gen incapxpr =.
    gen incceqpr =.
    gen inchcpr =.
    gen inccogspr =.
    gen incdvcpr =.
    gen incintanpr =.
    gen incibpr =.
    gen incoancfpr =.
    gen inrevtprpr =.
    gen incspipr =.
    gen incxadpr =.
    gen incxsgapr =.
    gen increvgrowpr =.
    gen inccocipr =.
    gen time = pyear-1961
    
    
local varlist16 "atpr capxpr ceqpr chpr cogspr dvcpr intanpr ibpr oancfpr revtpr spipr xadpr xrdpr xsgapr revgrowpr ocipr"
reg prc atpr capxpr ceqpr chpr cogspr dvcpr intanpr ibpr oancfpr revtpr spipr xadpr xrdpr xsgapr revgrowpr ocipr if size == 0 & pyear >=1962 & pyear <=1971
local r2_16 = `e(r2)'
foreach var of local varlist16 {
    local varlist15: list varlist16 - var
    forvalues i = 1962(1)2017 {
    display `i'
    qui reg prc `varlist15'  if size == 0 & pyear >=1962 & pyear <=1971
    local r2_`var' = `e(r2)'
    noi di "regression without `var' :  r-squared is `r2_`var''"
    noi di "difference between full regression & regression without `var' is " `r2_16'-`r2_`var''
    }
}

regression without revgrowpr :  r-squared is .4880738219715385
difference between full regression & regression without revgrowpr is .02703107
2015
regression without revgrowpr :  r-squared is .4880738219715385
difference between full regression & regression without revgrowpr is .02703107
2016
regression without revgrowpr :  r-squared is .4880738219715385
difference between full regression & regression without revgrowpr is .02703107
2017
regression without revgrowpr :  r-squared is .4880738219715385
difference between full regression & regression without revgrowpr is .02703107
1962
regression without ocipr :  r-squared is .5150345109748508
difference between full regression & regression without ocipr is .00007038
1963
regression without ocipr :  r-squared is .5150345109748508
difference between full regression & regression without ocipr is .00007038
1964
regression without ocipr :  r-squared is .5150345109748508
difference between full regression & regression without ocipr is .00007038
1965
regression without ocipr :  r-squared is .5150345109748508
difference between full regression & regression without ocipr is .00007038

Thanks in advance for your help.

Jakub

[Jorrit Gosens]

You dont reference your years i in your loop, except for the display `i'

Perhaps what you wanted was more:

foreach var of local varlist16 {
    local varlist15: list varlist16 - var
    forvalues i = 1962(1)2017 {
    display `i'
    qui reg prc `varlist15'  if size == 0 & pyear >=1962 & pyear <=1971 & year==`i'
    local r2_`var' = `e(r2)'
    noi di "regression without `var' :  r-squared is `r2_`var''"
    noi di "difference between full regression & regression without `var' is " `r2_16'-`r2_`var''
    }
}

This is a bit of a guess, but at least it shouldn't be expected to be a functional loop if the values looped over are not referenced.
Edit: that is, youre just running the exact same list of operations in your loop if you do not reference the changing value of i each time around the loop.

I'm confused about the variable pyear and difference with year. If the range of pyear is 1962-1971, would that conflict with values of year for 1962-2017?

[Carole J. Wilson]

Your regression in the loop is not indexed by i, so it runs the regression for the full sample:

qui reg prc `varlist15'  if size == 0 & pyear >=1962 & pyear <=1971

Instead, what you want is:

local varlist16 "atpr capxpr ceqpr chpr cogspr dvcpr intanpr ibpr oancfpr revtpr spipr xadpr xrdpr xsgapr revgrowpr ocipr"
reg prc atpr capxpr ceqpr chpr cogspr dvcpr intanpr ibpr oancfpr revtpr spipr xadpr xrdpr xsgapr revgrowpr ocipr if size == 0 & pyear >=1962 & pyear <=1971
local r2_16 = `e(r2)'
foreach var of local varlist16 {
    local varlist15: list varlist16 - var
    forvalues i = 1962(1)2017 {
        display `i'
        qui reg prc `varlist15'  if size == 0 & pyear==`i'
        local r2_`var'_`yr' = `e(r2)'
        noi di "regression without `var' (year=`i'):  r-squared is `r2_`var''"
        noi di "difference between full regression (all years) & regression without `var' (year=`i') is " `r2_16'-`r2_`var'_`yr''
        }
    }

[Jakub Cirkowski]

Thank both of you very much !!!

The code works perfectly.

However, the problem of extraction of generated results remains.

Does anybody know how to extract the list of variables with corresponding difference between regressions for all years ?

regression without xsgapr :  r-squared is .6192836985435957
difference between full regression & regression without xsgapr is -.1041788
2014
regression without xsgapr :  r-squared is .7045800495473287
difference between full regression & regression without xsgapr is -.18947515
2015
regression without xsgapr :  r-squared is .7454627143212222
difference between full regression & regression without xsgapr is -.23035782
2016
regression without xsgapr :  r-squared is .6367964694218158
difference between full regression & regression without xsgapr is -.12169157
2017
regression without xsgapr :  r-squared is .7986824504784598
difference between full regression & regression without xsgapr is -.28357755
1964
regression without revgrowpr :  r-squared is .6545219589361939
difference between full regression & regression without revgrowpr is -.13941706
1965
regression without revgrowpr :  r-squared is .6274487815853103
difference between full regression & regression without revgrowpr is -.11234389
1966
regression without revgrowpr :  r-squared is .5809616546441536
difference between full regression & regression without revgrowpr is -.06585676
1967
regression without revgrowpr :  r-squared is .5022760750460422
difference between full regression & regression without revgrowpr is .01282882
1968
regression without revgrowpr :  r-squared is .441660068176022
difference between full regression & regression without revgrowpr is .07344483 

Thanks in advance for your help.

Jakub

[Jorrit Gosens]

I don't have a very clear idea of what you would like to end up with.
You are saying that you are getting the values you were after, but you'd want some other format? If so, please be more specific about that format.
My best guess is that you would want to create something using putexcel: https://www.stata.com/manuals13/pputexcel.pdf
That is, I dont see a neat way of putting these r2 values into the dataset you are using to run your regressions on.

[Carole J. Wilson]

Like Jorrit, I'm not sure if you want the info stored in your current dataset, a new dataset, an excel sheet or what. If you want to store your incremental r^2 differences in your current dataset, the following will create a new variable for each var in varlist16 with the value of the r^2 difference in each year. The new var is wo_`var'_dif (for regression without varname).

local varlist16 "atpr capxpr ceqpr chpr cogspr dvcpr intanpr ibpr oancfpr revtpr spipr xadpr xrdpr xsgapr revgrowpr ocipr"
reg prc atpr capxpr ceqpr chpr cogspr dvcpr intanpr ibpr oancfpr revtpr spipr xadpr xrdpr xsgapr revgrowpr ocipr if size == 0 & pyear >=1962 & pyear <=1971
local r2_16 = `e(r2)'
foreach var of local varlist16 {
    local varlist15: list varlist16 - var
    gen double wo_`var'_dif=.
    forvalues i = 1962(1)2017 {
        display `i'
        qui reg prc `varlist15'  if size == 0 & pyear==`i'
        local r2_`var'_`yr' = `e(r2)'
        noi di "regression without `var' (year=`i'):  r-squared is `r2_`var''"
        noi di "difference between full regression (all years) & regression without `var' (year=`i') is " `r2_16'-`r2_`var'_`yr''
        replace wo_`var'_dif= `r2_16'-`r2_`var'_`yr'' if pyear==`i'
        }
    }

[Jakub Cirkowski]

HI

Thank you very much for your help, but the obtained results slightly wrong

According to the code the difference in R2 for ocipr is -.1360871. However, I checked it and two regressions with and without ocipr gave the results presented below

      Source |       SS           df       MS      Number of obs   =     1,639
-------------+----------------------------------   F(15, 1623)     =    202.00
       Model |  457515.304        15  30501.0203   Prob > F        =    0.0000
    Residual |  245065.975     1,623  150.995672   R-squared       =    0.6512
-------------+----------------------------------   Adj R-squared   =    0.6480
       Total |  702581.279     1,638    428.9263   Root MSE        =    12.288

------------------------------------------------------------------------------
         prc |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        atpr |  -.0473956   .0126528    -3.75   0.000    -.0722132   -.0225781
      capxpr |  -.4281035   .1942955    -2.20   0.028    -.8091999   -.0470072
       ceqpr |   .8004429   .0429124    18.65   0.000     .7162733    .8846125
        chpr |  -.0912967   .1338569    -0.68   0.495    -.3538471    .1712537
      cogspr |   -1.01856   .2360408    -4.32   0.000    -1.481537   -.5555835
       dvcpr |   8.202519   .7348511    11.16   0.000     6.761162    9.643875
     intanpr |  -.0384099   .0521661    -0.74   0.462    -.1407298    .0639101
        ibpr |   1.872293   .1718382    10.90   0.000     1.535245    2.209341
     oancfpr |   .0849994   .2145587     0.40   0.692    -.3358418    .5058406
      revtpr |   1.002838   .2338169     4.29   0.000     .5442235    1.461453
       spipr |  -.9435355    .395705    -2.38   0.017    -1.719682   -.1673892
       xadpr |    .424245   .7946852     0.53   0.594    -1.134472    1.982962
       xrdpr |   6.651558   .5499055    12.10   0.000     5.572959    7.730158
      xsgapr |  -.8430097   .2487702    -3.39   0.001    -1.330954   -.3550651
   revgrowpr |    .086105   .0478222     1.80   0.072    -.0076947    .1799047
       _cons |    2.83014    .438045     6.46   0.000     1.970947    3.689334
------------------------------------------------------------------------------
regression without ocipr (year=2017):  r-squared is 
difference between full regression (all years) & regression without ocipr (year=2017) is -.1360871
(3,547 real changes made) 

The following regressions calculated manually show that the difference is 0.0045

reg prc atpr capxpr ceqpr chpr cogspr dvcpr intanpr ibpr oancfpr revtpr spipr xadpr xrdpr xsgapr revgrowpr ocipr if size == 0 & pyear ==2017

      Source |       SS           df       MS      Number of obs   =     1,639
-------------+----------------------------------   F(16, 1622)     =    193.20
       Model |  460798.482        16  28799.9051   Prob > F        =    0.0000
    Residual |  241782.797     1,622   149.06461   R-squared       =    0.6559
-------------+----------------------------------   Adj R-squared   =    0.6525
       Total |  702581.279     1,638    428.9263   Root MSE        =    12.209

------------------------------------------------------------------------------
         prc |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        atpr |  -.0477911   .0125719    -3.80   0.000      -.07245   -.0231322
      capxpr |   -.447374   .1930927    -2.32   0.021    -.8261114   -.0686366
       ceqpr |   .7852392   .0427601    18.36   0.000     .7013684    .8691099
        chpr |  -.0646671   .1331192    -0.49   0.627    -.3257707    .1964364
      cogspr |  -1.079269   .2348831    -4.59   0.000    -1.539975   -.6185624
       dvcpr |   8.151144   .7302191    11.16   0.000     6.718872    9.583416
     intanpr |  -.0375107   .0518318    -0.72   0.469     -.139175    .0641536
        ibpr |   1.919798   .1710356    11.22   0.000     1.584324    2.255272
     oancfpr |   .0282047   .2135256     0.13   0.895    -.3906102    .4470196
      revtpr |   1.062164   .2326606     4.57   0.000     .6058168    1.518511
       spipr |   -.906242   .3932468    -2.30   0.021    -1.677567   -.1349168
       xadpr |   .3882193   .7896246     0.49   0.623    -1.160572    1.937011
       xrdpr |   6.526313   .5470292    11.93   0.000     5.453355    7.599271
      xsgapr |  -.8811586    .247308    -3.56   0.000    -1.366235   -.3960818
   revgrowpr |   .0847681   .0475163     1.78   0.075    -.0084316    .1779678
       ocipr |   .6368771    .135705     4.69   0.000     .3707017    .9030525
       _cons |   2.911577   .4355807     6.68   0.000     2.057217    3.765937 

reg prc atpr capxpr ceqpr chpr cogspr dvcpr intanpr ibpr oancfpr revtpr spipr xadpr xrdpr xsgapr revgrowpr if size == 0 & pyear ==2017

      Source |       SS           df       MS      Number of obs   =     1,639
-------------+----------------------------------   F(15, 1623)     =    202.00
       Model |  457515.304        15  30501.0203   Prob > F        =    0.0000
    Residual |  245065.975     1,623  150.995672   R-squared       =    0.6512
-------------+----------------------------------   Adj R-squared   =    0.6480
       Total |  702581.279     1,638    428.9263   Root MSE        =    12.288

------------------------------------------------------------------------------
         prc |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        atpr |  -.0473956   .0126528    -3.75   0.000    -.0722132   -.0225781
      capxpr |  -.4281035   .1942955    -2.20   0.028    -.8091999   -.0470072
       ceqpr |   .8004429   .0429124    18.65   0.000     .7162733    .8846125
        chpr |  -.0912967   .1338569    -0.68   0.495    -.3538471    .1712537
      cogspr |   -1.01856   .2360408    -4.32   0.000    -1.481537   -.5555835
       dvcpr |   8.202519   .7348511    11.16   0.000     6.761162    9.643875
     intanpr |  -.0384099   .0521661    -0.74   0.462    -.1407298    .0639101
        ibpr |   1.872293   .1718382    10.90   0.000     1.535245    2.209341
     oancfpr |   .0849994   .2145587     0.40   0.692    -.3358418    .5058406
      revtpr |   1.002838   .2338169     4.29   0.000     .5442235    1.461453
       spipr |  -.9435355    .395705    -2.38   0.017    -1.719682   -.1673892
       xadpr |    .424245   .7946852     0.53   0.594    -1.134472    1.982962
       xrdpr |   6.651558   .5499055    12.10   0.000     5.572959    7.730158
      xsgapr |  -.8430097   .2487702    -3.39   0.001    -1.330954   -.3550651
   revgrowpr |    .086105   .0478222     1.80   0.072    -.0076947    .1799047
       _cons |    2.83014    .438045     6.46   0.000     1.970947    3.689334
------------------------------------------------------------------------------ 

After looking at the code I realized that the difference is calculated between the whole regression for years 1962-2017 and regression without the variable in any given year.
The desired effect is the difference between the whole regression for year X and regression without the variable in year X

I hope it will help to adjust the code.

Thanks for your time

Jakub

[Carole J. Wilson]

I don’t have access to my computer for a while; perhaps someone else can help. You basically want to put the “full” regression (with -if year==`i’) and its r^2 macro inside the year loop that defines the macro i.

[Jorrit Gosens]

Like so:

local varlist16 "atpr capxpr ceqpr chpr cogspr dvcpr intanpr ibpr oancfpr revtpr spipr xadpr xrdpr xsgapr revgrowpr ocipr"
foreach var of local varlist16 {
    local varlist15: list varlist16 - var
    gen double wo_`var'_dif=.
    forvalues i = 1962(1)2017 {
        display `i'
        qui reg prc `varlist16'  if size == 0 & pyear==`i'
        local r2_16 = `e(r2)'
        qui reg prc `varlist15'  if size == 0 & pyear==`i'
        local r2_`var'_`yr' = `e(r2)'
        noi di "regression without `var' (year=`i'):  r-squared is `r2_`var''"
        noi di "difference between full regression (all years) & regression without `var' (year=`i') is " `r2_16'-`r2_`var'_`yr''
        replace wo_`var'_dif= `r2_16'-`r2_`var'_`yr'' if pyear==`i'
        }
    }

Although Im not 100% sure if you want the if size == 0 in that line, but Im guessing you do.