Lasso Panel Regression

[Steven Sanders]

Hello everyone,

I do not know how to calculate a variable, which is based on a lasso panel regression.
The Variable is based on the following paper:
Avramov, Kaplanski, and Subrahmanyam (2020): "Anchoring on Past Fundamentals"

The authors write: "The FDI index is calculated every month from all available data up to that month
using the standard least absolute shrinkage and selection (LASSO) procedure of Tibshirani
(1996). In particular, consider month J. We run a LASSO panel regression of monthly stock
returns realized up to month J on previous-months’ deviations. Slope coefficients from the
panel regression reflect sources of both time-series and cross-sectional return predictability
from deviation variables. FDI is computed as the fitted value of the panel regression using
time J realizations of deviation variables."

Furthermore, the authors write: "LASSO is implemented via Python module LassoLarsIC, with a lambda penalty parameter to minimize
the BIC information criterion."

Because I never worked with Python, I try to replicate the FDI index in Stata16.

The following is a data example of my unbalanced panel data, where

permno = firm identifier
ret = return in month t of firm k
ym = year-month identifier
var1 - var118 = exogenous variables (in the data example I only show var1 - var11)

The authors call the exogenous variables "deviations" because they are calulated as deviations from prior accounting data. The deviations are calculated as cross-sectional percentiles in a given month.

* Example generated by -dataex-. To install: ssc install dataex
clear
input double permno float ym double ret float(var1 var2 var3 var4 var5 var6 var7 var8 var9 var10 var11)
10001 331    8.33333358168602 . .08778143 . .25148198  .02637777 . .13633068 .50901943 . . .27258027
10001 332  -2.230769209563732 . .09192546 . .25493273 .029166667 . .13260457 .50795454 . .  .2702089
10001 333  1.9999999552965164 .  .0924282 . .25499412 .030060606 . .12838341  .5129731 . . .26795053
10001 334 -2.9411764815449715 . .04977489 .  .3070353   .0360944 .  .4903027 .14438382 . .  .4795017
10001 335  -3.353535383939743 . .04934292 .  .3090301  .03162961 .  .4919071 .14459734 . .  .4878597
10001 336   6.382978707551956 . .05675923 .  .3243317  .03350084 .  .5117344  .1591724 . . .52232635
10001 337   7.999999821186066 .         . .         .          . .         .         . . .         .
10001 338  -7.629629969596863 .  .8717818 .  .4917102    .939869 .  .4916268  .7178662 . . .52687186
10001 339  3.0612245202064514 .  .8788185 .  .4997748   .9458584 .   .490991  .7567137 . .  .5404236
10001 340  1.9801979884505272 .  .9152083 .  .6039018   .8686937 . .53977406  .7808341 . .  .5277962
10001 341  -1.203883532434702 .  .9120529 .  .6102285   .8653846 .  .5450378  .7800528 . .  .5274048
10001 342  2.9999999329447746 .  .9156533 .  .6166743   .8687868 .  .5581564  .7901952 . . .53111935
10001 343   2.912621386349201 .  .2231749 . .03380589 .029767234 . .52227837  .9076075 . .  .4080688
10001 344 -2.1132076159119606 . .22533172 . .03647286  .02990983 . .52528214  .9076122 . .  .4088542
10001 345  3.9215687662363052 . .22916143 . .03790938  .03117836 .  .5351021  .9041902 . .  .3916591
10001 346                   0 . .12903225 . .04856705  .05915304 .  .7506862  .8308304 . .  .7017002
10001 347 -2.1132076159119606 . .12931667 . .04966238  .05572893 .   .744149  .8251594 . .  .6958718
10001 348  1.9607843831181526 . .13986014 . .05959752  .05421053 .   .731956  .8194842 . .  .6840873
10001 349  3.8461539894342422 .         . .         .          . .         .         . . .         .
10001 350  1.7777778208255768 .  .9203105 . .27206385   .9485208 .  .6455606 .52442396 . .  .6149083
10001 351   7.407407462596893 .  .9241877 . .26810935   .9496738 .  .6425178 .56118923 . .  .6144165
10001 352  -3.448275849223137 .  .9014828 .  .6707317   .9097301 . .59421295  .6401974 . .  .5600534
10001 353  1.7142856493592262 .  .8977046 .  .6707774   .9054267 .   .595397  .6408827 . . .56233424
10001 354    3.57142873108387 .   .895193 .  .6732283    .910596 . .59563637    .63517 . .   .550384
10001 355  27.586206793785095 .  .2030888 .  .6678043   .0584809 .  .6485844  .6143384 . . .58198404
10001 356 -2.7027027681469917 . .20237795 .   .669571   .0605161 .  .6509125  .6136621 . .  .5823068
10001 357   7.042253762483597 .  .2021699 .  .6703169  .06020942 .  .6500608  .6083471 . .  .5776586
10001 358  3.9473682641983032 .  .1091703 .  .3634511  .05717593 .  .8114097  .2470211 . .  .7247395
10001 359  3.7974681705236435 . .10144927 .  .3673103  .05024088 .   .808043 .24443434 . .  .7243935
10001 360 -1.8518518656492233 . .10795455 .  .3648827  .05533172 .  .7946747  .2458011 . .  .7136831
10001 361  -.6289307959377766 .  .7725258 .  .4955958   .8823374 .  .7749187 .29238844 . .  .6972287
10001 362  1.2658228166401386 .  .7831681 .  .4950448   .8898264 .  .7760532 .29151732 . .  .6995039
10001 363                   0 .  .8059105 . .49600375   .9043729 .  .7775055  .3145853 . .  .7024482
10001 364 -1.2658228166401386 .  .7033281 .  .7646924    .765256 .  .7769286  .3990262 . .  .6999771
10001 365  1.4102564193308353 .  .6984615 .  .7619155   .7635104 .  .7714286  .3954181 . .  .7004076
10001 366  2.5641025975346565 .  .6961236 .  .7622795   .7612558 .  .7688279  .3866602 . .  .7023211
10001 367  -5.000000074505806 .  .2318458 . .39251685  .08222433 .  .7414087  .8596326 . .  .6658122
10001 368   4.078947380185127 .  .2385536 .  .3920715  .08692596 .  .7428023  .8579057 . .    .66391
10001 369 -1.2820512987673283 . .23649557 .   .390492   .0857488 .  .7513621  .8571429 . .   .670778
10001 370                   0 . .12779221 .  .3455462  .04157907 .   .832258  .6638889 . .  .7518332
10001 371    .129870162345469 . .12590486 .   .349829   .0397952 .  .8330953  .6677427 . .  .7550834
10001 372   1.315789483487606 . .14263804 .  .3635623  .04036494 .  .8292269  .6525796 . .  .7550244
10001 373  1.2987012974917889 .  .9039834 .  .3021132   .9499068 .  .8056769  .2327791 . .  .7333681
10001 374 -1.1538460850715637 .  .9099488 .  .3033254   .9544132 .  .8097839 .23317307 . .  .7359791
10001 375  3.9473682641983032 .  .9225875 .  .3062748   .9594756 .  .8068209  .2415014 . .  .7372723
10001 376                   0 .  .8351704 .  .6531073   .7243368 .   .755377  .7237594 . .  .6949842
10001 377   7.848101109266281 .  .8336315 .  .6587731   .7272099 .  .7548326  .7170888 . .  .6927412
10001 378   -3.57142873108387 .  .8339974 .  .6647128   .7366658 .  .7461954  .7047664 . .  .6763534
10001 379  13.580246269702911 .  .2402278 .  .3927181  .09605048 .   .776123  .8441047 . .   .720531
10001 380  1.3043479062616825 .       .24 .  .3913832   .0985199 .  .7690844  .8447248 . .  .7129902
10001 381   13.04347813129425 . .23054293 .   .388535   .0911271 .  .7637621  .8398474 . .  .7053698
10001 382  13.461539149284363 . .09192708 . .55938435  .04649533 .  .7214337  .1715743 . .  .6755393
10001 383   -.677966047078371 . .08654583 . .56074977  .04202067 .  .7165484  .1704649 . .  .6731333
10001 384   -5.17241396009922 . .09059776 .  .5601648  .04347826 .  .7064062  .1869107 . .  .6673985
10001 385 -20.000000298023224 .  .8529324 .  .3766945   .8810281 .   .620699 .54652995 . .  .6122821
10001 386   8.181818574666977 .  .8568006 .  .3814988   .8912164 .  .6182308  .5452398 . .  .6093567
10001 387  1.0638297535479069 .   .862766 .   .372044   .8972896 .  .5998043  .5750708 . .  .5851661
10001 388  1.0526316240429878 .   .816023 .  .5458515    .788785 . .42517415  .6882966 . .  .3956372
10001 389  -.7708333432674408 .  .8090062 . .54062074   .7845048 .  .4231504  .6857603 . .  .3937044
10001 390   6.382978707551956 .  .8033723 . .53592885   .7888427 .  .4277078   .700921 . .   .396299
10001 391   3.999999910593033 . .21289474 .  .3406185   .7768576 .  .3158151  .6295428 . .  .1831706
10001 392  16.596153378486633 . .21553297 .  .3418567   .7766581 .  .3184775  .6283391 . . .18489523
10001 393  -2.500000037252903 .  .2166982 .  .3336522    .776482 .  .3182045   .625693 . . .19039656
10001 394 -1.7094017937779427 .  .8828875 .  .5137678  .08055101 .  .4696318  .9530527 . .  .4146621
10001 395  -1.513043511658907 .  .8780611 . .51944566  .07709802 .  .4687062  .9535931 . .  .4133155
10001 396                   0 .  .8721781 .  .5356444  .08510638 .  .4682841  .9535996 . .  .4205085
10001 397   1.785714365541935 .  .8107558 .  .6103299   .8791037 .  .6084273 .19989683 . .  .5045942
10001 398    1.10526317730546 .  .8125802 .  .6115777   .8859589 .  .6043695  .2043946 . .  .5029465
10001 399   7.017543911933899 .   .732983 .  .4930202   .1869201 .  .6403428 .25659364 . .  .5394679
10001 400  -.8196720853447914 .  .7252451 .  .4834854  .17571175 .  .6406463 .25498796 . . .53512573
10001 401  10.247933864593506 .  .7169949 .  .4789735  .17096846 .  .6388262 .25261095 . . .53098106
10001 402  -.7575757801532745 .         . .         .          . .         .         . . .         .
10001 403 -3.0534351244568825 .         . .         .          . .         .         . . .         .
10001 404   6.614173203706741 .  .1915459 .   .791863   .6659355 .  .6567768  .2089004 . . .53916174
10001 405   2.985074557363987 .  .1863417 .  .8022522   .6606033 .  .6647031 .20572206 . . .54448396
10001 406 -1.4492753893136978 .  .3585495 .  .8904518  .07094521 . .59419644  .2781955 . .  .4932079
10001 407   9.117647260427475 .  .3512875 .  .8911036  .06638024 . .59339935   .284869 . .  .4887837
10001 408   -4.76190485060215 .  .3698037 .  .8827002  .07482993 .  .5881406 .28527132 . .  .4938735
10001 409                   0 .  .8539604 .   .803243    .862274 . .47598475  .2022938 . .  .4102063
10001 410  -.4285714589059353 .  .8538813 .   .800298   .8724934 .  .4791434 .20017163 . .  .4093635
10001 411 -14.492753148078918 .   .852019 .  .7893194   .8764744 .   .482562  .2033791 . .  .4075349
10001 412   6.779661029577255 .   .566582 .   .677537   .1640706 .  .3557346  .7231303 . .  .3080208
10001 413  10.730158537626266 . .56079686 .  .6779831  .16110425 . .35766885  .7205674 . .  .3081599
10001 414   7.246376574039459 . .55985576 .  .6852086  .15416127 .  .3639576  .7317177 . .  .3155028
10001 415 -2.7027027681469917 .         . .         .          . .         .         . . .         .
10001 416  3.8333334028720856 . .06681716 .  .4686209   .1655242 . .05528053  .4019044 . .  .9279869
10001 417 -5.4054055362939835 . .06616962 . .47560975   .1606895 .  .0514066 .40011695 . .  .9298511
10001 418  -4.285714402794838 . .18424566 .  .6567457  .09182183 .   .671564  .3591031 . .  .9077471
10001 419 -3.3432837575674057 . .18503937 .  .6563158  .09128482 .  .6691688  .3599788 . .  .9098562
10001 420              -3.125 .   .188203 .  .6738095  .10289247 .   .658371  .3442873 . .   .898957
10001 421 -2.6209676638245583 .  .6641963 .  .6045765   .5909932 .  .7194169 .33770955 . .  .8801778
10001 422   .6376811303198338 .  .6738598 .  .6017009    .603777 .  .7130095  .3441454 . .  .8735735
10001 423                   0 .  .6660746 .  .5920513   .6085845 .  .7068611  .3627398 . .  .8677686
10001 424   5.000000074505806 .  .3189178 .  .3729131   .6863327 .  .7466905 .28598356 . .  .6089491
10001 425   6.031746044754982 . .31415835 .  .3720496   .6857392 .  .7479486 .28095645 . .  .6082051
10001 426                   0 .  .3069998 .  .3770256   .6806196 .  .7396483  .2711803 . .  .6052966
10001 427  -3.030303120613098 .         . .         .          . .         .         . . .         .
10001 428   4.374999925494194 .  .2111349 .    .11854   .7231737 .  .7101398  .6795135 . .  .6740028
10001 429  -3.030303120613098 . .20350404 .  .1131872   .7407407 .  .7124835  .6691309 . .  .6635936
10001 430               9.375 .  .5987985 .  .4607222  .10689293 .  .7370875  .4209894 . .  .6853063
end
format %tm ym

I hope someone can help me with the lasso panel regression stated above.

I think I need some kind of rolling lasso panel regression because the authors only use exogenous variables from the past to not have a look-ahead bias.

There are also many missing variables. The number of observations of the exogenous variables reach from 10,000 to 2,000,000.

Best regards,
Steven

[Steven Sanders]

I think I found a solution but now I have new problems.

I´m using the lasso2 command from SSC.

I´m using the following code to loop lasso regressions (here, only 4 exogenous variables as an example):

forvalues i = 331(1)600 {
lasso2 ret var1 var2 var3 var4 if ym <= `i', lic(bic) postres
}

But I don´t know how to save the beta coefficients in each loop.

I was thinking about a code like this but it does not work:

gen beta_var1 =.
gen beta_var2 =.
gen beta_var3 =.
gen beta_var4 =.

forvalues i = 331(1)600 {
lasso2 ret var1 var2 var3 var4 if ym <= `i', lic(bic) postres
replace beta_var1 = _b[var1] if ym == `i'
replace beta_var2 = _b[var2] if ym == `i'
replace beta_var3 = _b[var3] if ym == `i'
replace beta_var4 = _b[var4] if ym == `i'
}

The reason why it does not work is the following: When the lasso regression drops one of the exogenous variables, the loop breaks. For example when var4 is dropped as an beta coefficient, the error message is the following: [var4] not found r(111);


I hope someone can help me. To summarize my problem, I try to loop lasso regressions but I don´t know how to save the beta coefficients.

Best regards,
Steven

[Xiaomeng Wang]

Hello Steven, I am also dealing with the lasso implementation in stata. I need to filter the dominant variables in the unbalanced panel data, but I don't find the appropriate way to implement it. If possible, could u share your solution? I'd appreciate it very much!