loop for residuals; taking a while

[Wessel de Kroo]

Hi legends,

Applying the following code takes over 2 hours now, no outcome just yet. am I making an error , is there a faster/different command.

. dataex permno date returns mrp smb hml

----------------------- copy starting from the next line -----------------------

* Example generated by -dataex-. To install: ssc install dataex
clear
input double permno long date double(returns mrp smb hml)
35503 12449 -.014925372786819935 -.0601 .0268 .032799999999999996
47861 12449             -.078125 -.0601 .0268 .032799999999999996
12257 12449  .032258063554763794 -.0601 .0268 .032799999999999996
46893 12449   .03921568766236305 -.0601 .0268 .032799999999999996
77170 12449 -.015384615398943424 -.0601 .0268 .032799999999999996
41100 12449   .21134831011295319 -.0601 .0268 .032799999999999996
68161 12449  .014184396713972092 -.0601 .0268 .032799999999999996
60038 12449 -.029411764815449715 -.0601 .0268 .032799999999999996
86423 12449  .032258063554763794 -.0601 .0268 .032799999999999996
79624 12449 -.056603774428367615 -.0601 .0268 .032799999999999996
11378 12449 -.028846153989434242 -.0601 .0268 .032799999999999996
10294 12449   .05797101557254791 -.0601 .0268 .032799999999999996
91970 12449   .03703703731298447 -.0601 .0268 .032799999999999996
45313 12449    .0181818176060915 -.0601 .0268 .032799999999999996
59222 12449  .010989011265337467 -.0601 .0268 .032799999999999996
12086 12449   -.0833333358168602 -.0601 .0268 .032799999999999996
77360 12449  .009009009227156639 -.0601 .0268 .032799999999999996
38527 12449   .12953367829322815 -.0601 .0268 .032799999999999996
22753 12449 -.031578946858644485 -.0601 .0268 .032799999999999996
79405 12449   -.1538461595773697 -.0601 .0268 .032799999999999996
77218 12449  -.05263157933950424 -.0601 .0268 .032799999999999996
52231 12449    .1818181872367859 -.0601 .0268 .032799999999999996
52840 12449   .07189542800188065 -.0601 .0268 .032799999999999996
79481 12449   .08641975373029709 -.0601 .0268 .032799999999999996
17989 12449  -.08695652335882187 -.0601 .0268 .032799999999999996
11211 12449   .09756097197532654 -.0601 .0268 .032799999999999996
11430 12449   .08620689809322357 -.0601 .0268 .032799999999999996
77611 12449 -.008771929889917374 -.0601 .0268 .032799999999999996
41718 12449   .07616236060857773 -.0601 .0268 .032799999999999996
75645 12449                    0 -.0601 .0268 .032799999999999996
75500 12449  -.09473684430122376 -.0601 .0268 .032799999999999996
69885 12449   -.1666666716337204 -.0601 .0268 .032799999999999996
50092 12449   -.0441988967359066 -.0601 .0268 .032799999999999996
10074 12449   .15909090638160706 -.0601 .0268 .032799999999999996
76384 12449                    1 -.0601 .0268 .032799999999999996
10009 12449  .014285714365541935 -.0601 .0268 .032799999999999996
11043 12449    .3636363744735718 -.0601 .0268 .032799999999999996
79532 12449   .03999999910593033 -.0601 .0268 .032799999999999996
79563 12449   .10144927352666855 -.0601 .0268 .032799999999999996
33814 12449  .037383176386356354 -.0601 .0268 .032799999999999996
76960 12449     .036363635212183 -.0601 .0268 .032799999999999996
60935 12449    .0560000017285347 -.0601 .0268 .032799999999999996
10243 12449  -.05511811003088951 -.0601 .0268 .032799999999999996
11509 12449 -.049504950642585754 -.0601 .0268 .032799999999999996
47677 12449                    0 -.0601 .0268 .032799999999999996
78955 12449    .0760233923792839 -.0601 .0268 .032799999999999996
76513 12449  .016129031777381897 -.0601 .0268 .032799999999999996
10116 12449  -.10000000149011612 -.0601 .0268 .032799999999999996
79321 12449   .15909090638160706 -.0601 .0268 .032799999999999996
47642 12449  .037931036204099655 -.0601 .0268 .032799999999999996
85746 12449    .3636363744735718 -.0601 .0268 .032799999999999996
79174 12449  -.10000000149011612 -.0601 .0268 .032799999999999996
11819 12449    .0476190485060215 -.0601 .0268 .032799999999999996
48514 12449   .06164383515715599 -.0601 .0268 .032799999999999996
79408 12449   .03030303120613098 -.0601 .0268 .032799999999999996
47626 12449   .05000000074505806 -.0601 .0268 .032799999999999996
79168 12449   -.3235294222831726 -.0601 .0268 .032799999999999996
11437 12449    .1764705926179886 -.0601 .0268 .032799999999999996
67459 12449 -.005517241545021534 -.0601 .0268 .032799999999999996
75596 12449   .09243697673082352 -.0601 .0268 .032799999999999996
46164 12449   .08183673769235611 -.0601 .0268 .032799999999999996
77989 12449   .13615024089813232 -.0601 .0268 .032799999999999996
77157 12449  -.06060606241226196 -.0601 .0268 .032799999999999996
10812 12449   .24242424964904785 -.0601 .0268 .032799999999999996
76132 12449   -.0714285746216774 -.0601 .0268 .032799999999999996
90086 12449   .09259258955717087 -.0601 .0268 .032799999999999996
20264 12449   .18518517911434174 -.0601 .0268 .032799999999999996
11590 12449   -.1764705926179886 -.0601 .0268 .032799999999999996
11823 12449   .08909090608358383 -.0601 .0268 .032799999999999996
41064 12449   -.0882352963089943 -.0601 .0268 .032799999999999996
76621 12449   .20000000298023224 -.0601 .0268 .032799999999999996
20336 12449   .11764705926179886 -.0601 .0268 .032799999999999996
55212 12449  -.02957746386528015 -.0601 .0268 .032799999999999996
77847 12449   .21739129722118378 -.0601 .0268 .032799999999999996
77964 12449   .20512820780277252 -.0601 .0268 .032799999999999996
76892 12449   -.0458715595304966 -.0601 .0268 .032799999999999996
17961 12449   .04848484694957733 -.0601 .0268 .032799999999999996
22103 12449  .016597511246800423 -.0601 .0268 .032799999999999996
76329 12449  -.05882352963089943 -.0601 .0268 .032799999999999996
77526 12449   .07482992857694626 -.0601 .0268 .032799999999999996
91273 12449 -.016949152573943138 -.0601 .0268 .032799999999999996
79906 12449    .2361111044883728 -.0601 .0268 .032799999999999996
81700 12449   .16129031777381897 -.0601 .0268 .032799999999999996
76626 12449   .24229075014591217 -.0601 .0268 .032799999999999996
77029 12449 -.025641025975346565 -.0601 .0268 .032799999999999996
20125 12449   .02777777798473835 -.0601 .0268 .032799999999999996
77473 12449   .13333334028720856 -.0601 .0268 .032799999999999996
75237 12449 .0013468016404658556 -.0601 .0268 .032799999999999996
18091 12449  .010416666977107525 -.0601 .0268 .032799999999999996
77331 12449   .23188406229019165 -.0601 .0268 .032799999999999996
75881 12449 -.024390242993831635 -.0601 .0268 .032799999999999996
53189 12449   .09741935133934021 -.0601 .0268 .032799999999999996
76532 12449   .03333333507180214 -.0601 .0268 .032799999999999996
38105 12449  -.05263157933950424 -.0601 .0268 .032799999999999996
32943 12449  .011363636702299118 -.0601 .0268 .032799999999999996
68743 12449  -.13793103396892548 -.0601 .0268 .032799999999999996
11144 12449   .01587301678955555 -.0601 .0268 .032799999999999996
64397 12449    .1341991275548935 -.0601 .0268 .032799999999999996
76458 12449   .21052631735801697 -.0601 .0268 .032799999999999996
77188 12449  .005291005130857229 -.0601 .0268 .032799999999999996
end
format %d date

------------------ copy up to and including the previous line ------------------

Listed 100 out of 1210032 observations
Use the count() option to list more

gen ff3residuals = .
local N = _N
qui forvalues i = 1/`N' {

regress return mrp smb hml
predict temp, residuals
replace ff3residuals = temp
drop temp

}
end

[Wessel de Kroo]

1,2mil observations inherently takes a while ..?

[William Lisowski]

You are accomplishing nothing with your loop. Nothing within your loop depends on the local macro i which is the index of your loop. Thus you are running exactly the same code 1.2 million times.

You run the same regression on all the observations each time through the loop, and then replace all the values of ff3residuals with the predictions from the most recent regression results. The end result will be the same as

gen ff3residuals = .
local N = _N
regress return mrp smb hml
predict temp, residuals
replace ff3residuals = temp
drop temp

With regard to "how long does this take" before running the entire loop (once you correct your code to do whatever you intended to do within the loop) I would run something like

quiet forvalues i=10000(10000)`N' {

to test the loop with 10000, 20000, ... , 1200000 observations - 120 time through the loop, see how long that takes. Then you can estimate what 1200000 times through the loop will take.

[Jorrit Gosens]

A single regression with 1.2 million obs and 4 variables on my machine still takes only 0.16 seconds.
As William notes, the issue is you're running 1.2 million regressions, and unfortunately for no good reason, because it is the same regression 1.2 million times.

You will probably be best helped if you explain what you were trying to do with the loop. Regressions by certain groups?

[William Lisowski]

Which means - since the number of times through the loop is equal to the number of observations - 1.2 million times through the loop will take 192,000 seconds - 53 hours and 20 minutes.