How to make a forval loop more efficient

[Attaullah Shah]

In stock portfolio setting, I am running the following loop

 forval i = 1 / 497 {
               forval s = 1 / 5  {
                   forval h = 1 / 5  {
                             qui sum  ri_rf if sizef == `s' & valuef == `h' & w_ser == `i'              
        
                            qui replace  S`s'H`h' = r(mean) if w_ser == `i'
                    }
                }
        }

where sizef and valuef are variables having values from 1 to 5, and ri_rf has stock returns.
It has been more than an hour and it is still running, can we make it more efficient?

[Nick Cox]

Whether "efficient" means "faster" or "using less storage", then the advice appears to be the same here.

0. Don't loop at all unless you really need to.

1. Don't create separate variables to hold separate results unless you really need them.

2. Don't use summarize when you can use summarize, meanonly. http://www.stata-journal.com/sjpdf.h...iclenum=st0135

3. Avoid repeated if when you can. In this problem using if will imply that most of the selection entails rejecting observations that don't satisfy as only a fraction 1 / (5 * 5 * 497) of the observations qualify each time round the innermost loop. (The fraction will differ if those categories have unequal frequencies.) That's a lot of checking.

This seems to boil down to the calculation of means for cross-combinations:

bysort w_ser sizef valuef : gen mean  = sum(ri_rf) / sum(ri_rf < .)
by w_ser sizef valuef: replace mean = mean[_N]

If you really want those results redistributed in 25 variables, first say why.

[Attaullah Shah]

Thank you Nick. The 25 variables are my portfolios from the intersection of valuef and valuef rankings. So how can I modify your code so that the 25 portfolios are assigned mean returns as in my code.

[Nick Cox]

I asked for why you want them, not what they are (which I could work out from your code).

[Attaullah Shah]

The 25 portfolios are designed to have features progressively influenced by the value and size factors of stock. This way, these portfolios will exhibit return patterns that are not biased solely by value or size of stocks.

[Nick Cox]

Still puzzled at why you need 25 separate variables rather than use one, selecting which observations to use in certain problems.

Have you considered separate? Otherwise you want something like this:

bysort w_ser sizef valuef : gen mean  = sum(ri_rf) / sum(ri_rf < .)
by w_ser sizef valuef: replace mean = mean[_N]

forval s = 1 / 5  {
    forval h = 1 / 5  {
       gen S`s'H`h' = mean if sizef == `s' & valuef == `h' 
       bysort w_ser (S`s'H`h') : replace S`s'H`h' = S`s'H`h'[1] 
    }
}

[Attaullah Shah]

That is a wonderful solution as ever. Thanks

[Attaullah Shah]

Nick you have referred to _^separate, is that a function in Stata.

[Sergiy Radyakin]

http://www.stata.com/manuals13/dseparate.pdf

[Nick Cox]

As Sergiy pointed out, it is a documented command in Stata (and not a function).