Predict residuals by year

[Shuai Yue]

Hi,

I'm new to Stata and have a regression of firm weekly return on market weekly returns: reg frim_r M_lag2 M_lag1 M_r M_lead1 M_lead2
I would like to estimate the regression by year and keep residuals for each year, so I tried the following code but ended with no observations r(2000).
Could someone please show me the right way to keep residuals by year?

Code:
gen residuals=.
levelsof year, local(level)
foreach i of local level {
reg frim_r M_lag2 M_lag1 M_r M_lead1 M_lead2 if year ==`i'
predict _res, res
replace residuals=_res if year ==`i'
drop _res
}

Kind Regards

[Clyde Schechter]

Although you don't say so explicitly, I'm going to assume that your variables M_lag2 and M_lag1 are first and second lagged values of variable M, and the M_lead1 and M_lead2 are the first and second forward values of M. It follows that in the first and second years, as well as the last and penultimate years, at least one of these variables will have a missing value. So for those two first and two last years, there will be no observations in the estimation sample because any observation with any of the regression variables missing is necessarily excluded from the regression analysis. When Stata discovers that the estimation sample is empty, it throws an error message and halts. Your residuals variable contains only missing values because the very first regression failed and Stata stopped there.

There are a few ways to work around this. One would be to remove the first two and last two years from local macro level before you run the loop. Another way, that I think is better overall, though longer to code, is to -capture- the regression and skip it when there are no observations for that year. Here's an example of how this would work. You didn't provide any example data, so I'm using one of the Stata example data sets and doing a simplified version of your regression:

clear*
webuse grunfeld
gen kstock_lag1 = L1.kstock
gen kstock_lead1 = F1.kstock

gen residuals = .
levelsof year, local(level)
foreach i of local level {
    capture reg mvalue kstock kstock_lag1 kstock_lead1 if year == `i'
    if inlist(c(rc), 2000, 2001) {
        display as error "Year `i' skipped due to no/insufficient observations"
    }
    else if c(rc) != 0  {
        display as error "Unexpected error year `i'
        error c(rc)
    }
    else {
        predict _res, res
        replace residuals = _res if year == `i'
        drop _res
    }
}

If it turns out that there are other years, besides the first two and last two, where missing values in the regression variables causes you to have no observations (or too few observations to do the regression), they will, similarly be skipped over, and you will get a warning message pointing out that it has happened. Then you can go back when you're done and check whether this means there is something wrong with your data, or if that's just the way things are.

In the event that some other problem arises with doing the regression in one of the years, you will get a message that something unexpected has gone wrong, and Stata will stop at that point. You will then have to look into what went wrong and fix the problem, and then try again.

[Shuai Yue]

Thank you so much for the reply, Clyde. I found there is no observation in 1990 so I deleted that.

I'm using the weekly data from 1990 to 2018 and would like to run regressions by year. If Mkt_r is the value in week t, lagM1 will be that in t-1 and LeadM1 is that in t+1.

Below is an example of my data

* Example generated by -dataex-. To install: ssc install dataex
clear
input long stkcd int year byte week double(Stk_r Mkt_r tradingweek lagM1 lagM2 leadM1 leadM2)
1 1991 14        .   .01657 199114        .        . -.004338 -.001181
1 1991 15 -.029826 -.004338 199115   .01657        . -.001181  .004164
1 1991 16 -.029462 -.001181 199116 -.004338   .01657  .004164  .000254
1 1991 17 -.024637  .004164 199117 -.001181 -.004338  .000254  .001524
1 1991 18  -.02977  .000254 199118  .004164 -.001181  .001524  .013455
1 1991 19 -.029521  .001524 199119  .000254  .004164  .013455  .040301
1 1991 20 -.029701  .013455 199120  .001524  .000254  .040301  .047594
1 1991 21 -.024685  .040301 199121  .013455  .001524  .047594  .044847
1 1991 22 -.024804  .047594 199122  .040301  .013455  .044847  .041766
1 1991 23  -.03945  .044847 199123  .047594  .040301  .041766  .050837
1 1991 24 -.029451  .041766 199124  .044847  .047594  .050837  .051166
1 1991 25 -.030067  .050837 199125  .041766  .044847  .051166   .01772
1 1991 26 -.024397  .051166 199126  .050837  .041766   .01772   .01266
1 1991 27 -.034716   .01772 199127  .051166  .050837   .01266  .040175
1 1991 28 -.019506   .01266 199128   .01772  .051166  .040175  .040406
1 1991 29 -.039789  .040175 199129   .01266   .01772  .040406  .046973
1 1991 30  -.02428  .040406 199130  .040175   .01266  .046973    .0445
1 1991 31  .124178  .046973 199131  .040406  .040175    .0445  .050892
1 1991 33 -.459644    .0445 199133  .046973  .040406  .050892  .049483
1 1991 34 -.012658  .050892 199134    .0445  .046973  .049483  .050748
1 1991 35 -.038462  .049483 199135  .050892    .0445  .050748  .027405
1 1991 36 -.076667  .050748 199136  .049483  .050892  .027405  .022339
1 1991 37  .043321  .027405 199137  .050748  .049483  .022339  .017499
1 1991 38 -.024221  .022339 199138  .027405  .050748  .017499   .02275
1 1991 39  .028369  .017499 199139  .022339  .027405   .02275  .049355
1 1991 40  .096552   .02275 199140  .017499  .022339  .049355  .050603
1 1991 41  .528302  .049355 199141   .02275  .017499  .050603  .042979
1 1991 42 -.012346  .050603 199142  .049355   .02275  .042979  .041211
1 1991 43 -.002083  .042979 199143  .050603  .049355  .041211  .044025
1 1991 44  .206681  .041211 199144  .042979  .050603  .044025  .041878
1 1991 45  .117647  .044025 199145  .041211  .042979  .041878  .035895
1 1991 46  .068111  .041878 199146  .044025  .041211  .035895    .0367
1 1991 47 -.075362  .035895 199147  .041878  .044025    .0367  .045077
1 1991 48 -.137931    .0367 199148  .035895  .041878  .045077  .040979
1 1991 49  .014545  .045077 199149    .0367  .035895  .040979  .037872
1 1991 50 -.014337  .040979 199150  .045077    .0367  .037872  .041577
1 1991 51  .003636  .037872 199151  .040979  .045077  .041577  .014433
1 1991 52  .059783  .041577 199152  .037872  .040979  .014433  .017048
1 1991 53  .003419  .014433 199153  .041577  .037872  .017048  .037466
1 1992  1 -.008518  .017048 199201  .014433  .041577  .037466  .031793
1 1992  2  .024055  .037466 199202  .017048  .014433  .031793  .007856
1 1992  3 -.033557  .031793 199203  .037466  .017048  .007856   .01764
1 1992  4  .027778  .007856 199204  .031793  .037466   .01764  .027027
1 1992  5  .081081   .01764 199205  .007856  .031793  .027027  .058605
1 1992  7  -.00625  .027027 199207   .01764  .007856  .058605  .022732
1 1992  8 -.004717  .058605 199208  .027027   .01764  .022732  .034958
1 1992  9  -.00316  .022732 199209  .058605  .027027  .034958  .023726
1 1992 10  .017433  .034958 199210  .022732  .058605  .023726  .027579
1 1992 11  .006231  .023726 199211  .034958  .022732  .027579  .005619
1 1992 12  .004644  .027579 199212  .023726  .034958  .005619  .033805
1 1992 13  .111502  .005619 199213  .027579  .023726  .033805  .005259
1 1992 14  .085774  .033805 199214  .005619  .027579  .005259  .007991
1 1992 15  .140655  .005259 199215  .033805  .005619  .007991 -.074257
1 1992 16  .081081  .007991 199216  .005259  .033805 -.074257  .082643
1 1992 17    .0625 -.074257 199217  .007991  .005259  .082643   .05833
1 1992 18  .132353  .082643 199218 -.074257  .007991   .05833  .158328
1 1992 19  .106494   .05833 199219  .082643 -.074257  .158328 1.639432
1 1992 20  .064554  .158328 199220   .05833  .082643 1.639432 -.064188
1 1992 21 -.020948 1.639432 199221  .158328   .05833 -.064188 -.123802
1 1992 22  .018018 -.064188 199222 1.639432  .158328 -.123802 -.118466
1 1992 23 -.073009 -.123802 199223 -.064188 1.639432 -.118466  .006806
1 1992 24 -.081146 -.118466 199224 -.123802 -.064188  .006806 -.006143
1 1992 25  -.05974  .006806 199225 -.118466 -.123802 -.006143  .034364
1 1992 26  .024862 -.006143 199226  .006806 -.118466  .034364  .006581
1 1992 27  .103774  .034364 199227 -.006143  .006806  .006581 -.052621
1 1992 28 -.032967  .006581 199228  .034364 -.006143 -.052621 -.074419
1 1992 29  .068182 -.052621 199229  .006581  .034364 -.074419 -.187662
1 1992 30  .083924 -.074419 199230 -.052621  .006581 -.187662 -.034127
1 1992 31 -.005453 -.187662 199231 -.074419 -.052621 -.034127 -.130368
1 1992 32 -.013158 -.034127 199232 -.187662 -.074419 -.130368  .050422
1 1992 33 -.028889 -.130368 199233 -.034127 -.187662  .050422 -.052017
1 1992 34 -.014874  .050422 199234 -.130368 -.034127 -.052017 -.069821
1 1992 35 -.029036 -.052017 199235  .050422 -.130368 -.069821 -.007742
1 1992 36  -.07177 -.069821 199236 -.052017  .050422 -.007742 -.040611
1 1992 37  .115979 -.007742 199237 -.069821 -.052017 -.040611 -.026901
1 1992 38 -.016166 -.040611 199238 -.007742 -.069821 -.026901 -.037478
1 1992 39   .06338 -.026901 199239 -.040611 -.007742 -.037478 -.047126
1 1992 40 -.015453 -.037478 199240 -.026901 -.040611 -.047126  .036567
1 1992 41 -.001121 -.047126 199241 -.037478 -.026901  .036567 -.149131
1 1992 42  -.01459  .036567 199242 -.047126 -.037478 -.149131 -.100718
1 1992 43 -.070615 -.149131 199243  .036567 -.047126 -.100718 -.141422
1 1992 44 -.020833 -.100718 199244 -.149131  .036567 -.141422 -.138108
1 1992 45 -.048811 -.141422 199245 -.100718 -.149131 -.138108  -.03874
1 1992 46 -.102632 -.138108 199246 -.141422 -.100718  -.03874  .692627
1 1992 47 -.085044  -.03874 199247 -.138108 -.141422  .692627  .290519
1 1992 48  .113782  .692627 199248  -.03874 -.138108  .290519  .075364
1 1992 49  .122302  .290519 199249  .692627  -.03874  .075364 -.028867
1 1992 50  .033333  .075364 199250  .290519  .692627 -.028867  .089143
1 1992 51  .027295 -.028867 199251  .075364  .290519  .089143 -.000453
1 1992 52   .01087  .089143 199252 -.028867  .075364 -.000453  .181008
1 1992 53  .063321 -.000453 199253  .089143 -.028867  .181008  .190109
1 1993  2  .039326  .181008 199302 -.000453  .089143  .190109  .036919
1 1993  3  .020541  .190109 199303  .181008 -.000453  .036919  .119556
1 1993  4  .036017  .036919 199304  .190109  .181008  .119556  .081236
1 1993  6  .079755  .119556 199306  .036919  .190109  .081236  .040278
1 1993  7  .005682  .081236 199307  .119556  .036919  .040278  -.11507
1 1993  8  .107345  .040278 199308  .081236  .119556  -.11507 -.157154
1 1993  9        0  -.11507 199309  .040278  .081236 -.157154  .086001
1 1993 10 -.047619 -.157154 199310  -.11507  .040278  .086001 -.044936
1 1993 11  .053571  .086001 199311 -.157154  -.11507 -.044936 -.134937
end

Sorry I barely know the capture commend, could you please explain a bit more on

if inlist(c(rc), 2000, 2001) { ... }

in your example?

I also tried my previous code after deleting the observations in 1990, this time I could get 28 seperate regressions for each year. However, I still couldnt be able to save the residuals, may I have some suggestions on this if possible? The first regression results in 1991 is :

      Source |       SS           df       MS      Number of obs   =       586
-------------+----------------------------------   F(5, 580)       =      7.30
       Model |  .308156658         5  .061631332   Prob > F        =    0.0000
    Residual |  4.89622583       580  .008441769   R-squared       =    0.0592
-------------+----------------------------------   Adj R-squared   =    0.0511
       Total |  5.20438249       585   .00889638   Root MSE        =    .09188

----------------------------------------------------------------------------------
cashdividendre~d |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-----------------+----------------------------------------------------------------
           lagM2 |   .0096599   .2104759     0.05   0.963    -.4037281    .4230478
           lagM1 |  -.7980629    .325675    -2.45   0.015    -1.437709   -.1584168
vwcashreinvested |   1.481904   .3773196     3.93   0.000     .7408249    2.222983
          leadM1 |   .0151335   .3798454     0.04   0.968    -.7309066    .7611735
          leadM2 |   .1134313    .289891     0.39   0.696    -.4559326    .6827953
           _cons |  -.0037562   .0068822    -0.55   0.585    -.0172732    .0097608
----------------------------------------------------------------------------------
(15,105 missing values generated)
(0 real changes made)

Thanks again

[Nick Cox]

Clyde's suggestions look fine.

if inlist(c(rc), 2000, 2001)

is a way of saying: is the return code 2000 or 2001?

Those return codes have these explanations

. error 2000
no observations
r(2000);

. error 2001
insufficient observations
r(2001);

Your report that you still weren't able to save the residuals is hard to answer, as you don't even show the code you used or precisely what means.

[Shuai Yue]

I figured it out, the last step of my code went wrong.

Thanks a lot for the explanation, Nick. And thanks again for the help, Clyde.

[Monicucha Paganini]

how can I estimate a production function using acfest with translog form and save the residuals of the first stage estimation? i need to calculate the corrected markup following DeLoecker
the command prodest doesnt work in my computer!
thanks