Forvalues loop stops running in the middle of the code

[Jana He]

Hello,

I run the following forvalues loop:

local count=_N
            local j = 1
            local u = 1
            local i = 2
            gen crspret= 1+vwretd 

            set trace on
            
            forvalues j = `u'/`count'{
                if (speriod[`u']==bdate[`i']) & (cusip[`u']==cusip[`i']) {
                    replace crspret = crspret[`u']*(1+vwretd[`i']) in `u' 
                    local u = `u'+1
                    local i = `u'+1
                }
                
                else if speriod[`u']!=bdate[`i'] & (cusip[`u']==cusip[`i']) {
                    replace crspret = (crspret[`u'])*(1+vwretd[`i']) in `u' 
                    local i = `i'+1
                }
                
                else if cusip[`u']!=cusip[`i'] {
                    replace crspret =. in `u'
                    local u = `u'+1
                    local i = `u'+1
                }
            }

And this is what my data looks like after running the code:

* Example generated by -dataex-. For more info, type help dataex
clear
input float(cusip bdate speriod vwretd crspret)
1  1  5   .01 1.0192497
1  2  6    .1 1.0293413
1  3  7  -.06  .9264072
1  4  8 -.025 1.0249612
1  5  9  .001  1.166879
1  6 10   .02 1.1610502
1  7 11  -.01 1.1212103
1  8 12   .04  1.200488
1  9 13   .11 1.1427721
1 10 14 -.004 1.0346719
1 11 15 -.015 1.0336591
1 12 16   .06      1.06
1 13 17  -.01       .99
1 14 18  .005     1.005
2  1  5   .01      1.01
2  2  6    .1       1.1
2  3  7  -.06       .94
2  4  8 -.025      .975
2  5  9  .001     1.001
2  6 10   .02      1.02
2  7 11  -.01       .99
2  8 12   .04      1.04
2  9 13   .11      1.11
2 10 14 -.004      .996
2 11 15 -.015      .985
2 12 16   .06      1.06
2 13 17  -.01       .99
2 14 18  .005     1.005
3  1  5   .01      1.01
3  2  6    .1       1.1
3  3  7  -.06       .94
3  4  8 -.025      .975
3  5  9  .001     1.001
3  6 10   .02      1.02
3  7 11  -.01       .99
3  8 12   .04      1.04
3  9 13   .11      1.11
3 10 14 -.004      .996
3 11 15 -.015      .985
3 12 16   .06      1.06
3 13 17  -.01       .99
3 14 18  .005     1.005
end

The code just stops at line 11 and does not run through (the values for the variable "crspret" are just (1+vwretd) which means that they did not change however they should have changed if the loop would be applied). I cannot explain why this is the case. Maybe someone has an idea what the issue is.

[Mike Lacy]

Setting aside the issue of why your -forvalues- loop fails, I'd point out that the way you are using the -if- command is very likely not what you want. If I'm right, this is a more important problem than what is happening with your loop.

Stata's -if command- is used for programming that does not depend on the values of individual observations. (It can be used with such references, but that is almost never what a user wants.) Conditional assignments of values that depend on the values of individual observations, such as what you are doing with the -replace- command, are done in Stata with the -if qualifier-. I'd recommend you take a look at -help ifcmd- as a pointer towards understanding the difference. This feature of Stata -- the "if command" vs. the "if qualifier" is confusing to most new users who have experience programming in other (non-vectorized) languages, so you're not alone in this difficulty.

For example, the Stata-ish version of the first statement of your -forvalues- loop you is probably something like:

replace crspret = crspret[`u']*(1+vwretd[`i']) in `u' ///
                  if (speriod[`u']==bdate[`i']) & (cusip[`u']==cusip[`i'])

However, there are enough other unusual things about your code that I'd suspect that what you need is a completely different and likely easier approach to what you want to happen. I don't know what goal you are trying to accomplish, so I can't suggest what that approach might be. (And, if I did understand, I'm not certain I'd know the solution <grin>, so others are welcome to jump in and help here.) My suggestion would be to step back from asking "fix my loop code" to explaining what you want to do *at a more conceptual level,* so that you can get help in implementing that in Stata.

[Nick Cox]

A major difficulty with this code is that the outside loop is cycling over observations while the code inside is also moving from observation to observation. You can't happily mix that main loop over observations with whatever else you are doing,.

Here set trace is not so helpful as just displaying what the code is looking at.

* Example generated by -dataex-. For more info, type help dataex
clear
input float(cusip bdate speriod vwretd _crspret)
1  1  5   .01 1.0192497
1  2  6 .1 1.0293413
1  3  7  -.06  .9264072
1  4  8 -.025 1.0249612
1  5  9  .001  1.166879
1  6 10   .02 1.1610502
1  7 11  -.01 1.1212103
1  8 12   .04  1.200488
1  9 13   .11 1.1427721
1 10 14 -.004 1.0346719
1 11 15 -.015 1.0336591
1 12 16   .06  1.06
1 13 17  -.01   .99
1 14 18  .005 1.005
2  1  5   .01  1.01
2  2  6 .1   1.1
2  3  7  -.06   .94
2  4  8 -.025  .975
2  5  9  .001 1.001
2  6 10   .02  1.02
2  7 11  -.01   .99
2  8 12   .04  1.04
2  9 13   .11  1.11
2 10 14 -.004  .996
2 11 15 -.015  .985
2 12 16   .06  1.06
2 13 17  -.01   .99
2 14 18  .005 1.005
3  1  5   .01  1.01
3  2  6 .1   1.1
3  3  7  -.06   .94
3  4  8 -.025  .975
3  5  9  .001 1.001
3  6 10   .02  1.02
3  7 11  -.01   .99
3  8 12   .04  1.04
3  9 13   .11  1.11
3 10 14 -.004  .996
3 11 15 -.015  .985
3 12 16   .06  1.06
3 13 17  -.01   .99
3 14 18  .005 1.005
end

local count=_N
 local j = 1
 local u = 1
 local i = 2
 gen crspret= 1+vwretd

 
 
 forvalues j = `u'/`count'{
 if (speriod[`u']==bdate[`i']) & (cusip[`u']==cusip[`i']) {
 quietly replace crspret = crspret[`u']*(1+vwretd[`i']) in `u'
 local u = `u'+1
 di "1: u is `u'"
 local i = `u'+1
 di "1: i is `i'"
 }
 
 else if speriod[`u']!=bdate[`i'] & (cusip[`u']==cusip[`i']) {
 quietly replace crspret = (crspret[`u'])*(1+vwretd[`i']) in `u'
 local i = `i'+1
 di "2: i is `i'"
 }
 
 else if cusip[`u']!=cusip[`i'] {
 quietly replace crspret =. in `u'
 local u = `u'+1
 di "3: u is `u'"
 local i = `u'+1
 di "3: i is `i'"
 }
 }

Here is the output from that code.

2: i is 3
2: i is 4
2: i is 5
1: u is 2
1: i is 3
2: i is 4
2: i is 5
2: i is 6
1: u is 3
1: i is 4
2: i is 5
2: i is 6
2: i is 7
1: u is 4
1: i is 5
2: i is 6
2: i is 7
2: i is 8
1: u is 5
1: i is 6
2: i is 7
2: i is 8
2: i is 9
1: u is 6
1: i is 7
2: i is 8
2: i is 9
2: i is 10
1: u is 7
1: i is 8
2: i is 9
2: i is 10
2: i is 11
1: u is 8
1: i is 9
2: i is 10
2: i is 11
2: i is 12
1: u is 9
1: i is 10
2: i is 11
2: i is 12
2: i is 13
1: u is 10
1: i is 11
2: i is 12
2: i is 13
2: i is 14
1: u is 11
1: i is 12
2: i is 13
2: i is 14

That jumping around is not I guess what you really want. Also, the code stops because you have reached `count' and so there is nothing else to do.


Here (I guess) programming experience with other software is a mixed blessing as it's very rare in Stata that you need to loop over observations.

I would back up and explain the desired calculation in words. A starting point is that working separately by cusip implies a framework

bysort cusip (bdate) :

What you want is probably obvious to someone working with this kind of data, but that is not me. I note that speriod is bdate + 4 in the data example.

Extra hint: Some calculations that involve looking forward in the data are often trivial if you temporarily reverse time (e.g. by negating your date or time variable).

EDIT: @Mike Lacy's reply in #2 was not visible as I was drafting this but he is essentially giving the same broad advice.

[Jana He]

I try to explain what my goal with this loop is. I am looking at three different stocks (stock 1, 2 and 3 (column 1)) and I have 14 time periods (column 2). column 4 gives me the daily return of the stock. Now, I am looking at a 5 day window. Column 3 indicated the end of the 5 day window. In line 1, I look at period 1 to 5 and in line 2, I look at period 2 to 6 and so. I want to sum up all returns lying in this period window by calculating: (1+return[1])*(1+return[2])*(1+return[3])*(1+return[4])+(1+return[5]) and this result is stored in the new variable "crspret"

For example:
The value for variable "crspret" in line 1 should be (1+0,01)*(1+0.1)*(1+-0.06)*(1+-0.025)*(1+0.001)
And if I am not able to get a 5 day window (which is the case at the end of the 14 period), I want that crspret = .

@Nick: Why has the loop reached `count'? It stopped in line 11 and not line 42?

[Nick Cox]

Your forvalues loop stops when the loop counter reaches `count'. The loop itself never knew, and certainly does not remember, that that number is equal to the number of observations, nor that your intent is that it keeps going until the end of the data.. It has gone round the loop 42 times which is exactly what you asked.

This may help:

gen work = 1 + vwretd
bysort cusip (bdate) : gen  double wanted = work * work[_n+1] * work[_n+2] * work[_n+3] * work[_n+4]

I take it that + (1+return[5]) should be * (1+return[5])

[Jana He]

Thank you Nick, that is exactly what I was looking for!
I have an additional question: I am also looking at larger windows like 500 days, is there a way to achieve this without typing 500 variables into the code (from work[_n+1] to work [_n+499]?

[Nick Cox]

That could be a loop problem.

gen work = 1 + vwretd

bysort cusip (bdate) : gen  double wanted = work  

forval j = 1/499 {      
     by cusip : replace wanted = wanted * work[_n + `j']  
}

Consider also something like rangestat from SSC.

gen work = ln(1 + vwretd)  
rangestat (sum) wanted =work, int(bdate 0 499) by(cusip)  
replace wanted = exp(wanted)

No code was tested here.

[Jana He]

The first code works! Thank you very much!