Trouble with "forvalues" loop

[Hanna Thompson-Schumann]

Nick Cox Hello,

I was wondering if you could help me figure out how to get the sum of this formula, perhaps by using a forvalues loop,

the formula is: (age+1-k)^1.5

for example if k = 0,1,2,3 it would look like this:


gen weight3=(3+1-0)^1.5+(3+1-1)^1.5+(3+1-2)^1.5+(3+1-3)^1.5

I need the total as it is the denominator to another formula: (the picture in the centre)


Any help would be greatly appreciated!

Hanna

[Clyde Schechter]

The formula you want to evaluate is clear enough, but you have given no information about your data or some other key aspects of the problem.

1. Please post back with an example data set that includes all of the variables needed to evaluate the formula. Use the -dataex- command to do that. If you are running version 17, 16 or a fully updated version 15.1 or 14.2, -dataex- is already part of your official Stata installation. If not, run -ssc install dataex- to get it. Either way, run -help dataex- to read the simple instructions for using it. -dataex- will save you time; it is easier and quicker than typing out tables. It includes complete information about aspects of the data that are often critical to answering your question but cannot be seen from tabular displays or screenshots. It also makes it possible for those who want to help you to create a faithful representation of your example to try out their code, which in turn makes it more likely that their answer will actually work in your data.

2. Is age a variable in your data set? What about lambda? If either of these is not a variable but some constant to be applied in every situation, where does this constant "live?" That is, how would one access its value.

3. The use of age as the upper limit on the summation in the denominator implies that age is always an integer. Is that true in your data set? If not, how should that exception be handled?

When asking for help with code, always show example data. When showing example data, always use -dataex-.

[Daniel Schaefer]

Hi Hanna,

In general it is wise to follow Clyde's advice to get a good answer that fits your particular problem. However, perhaps this code snippet can help you get started on your own particular solution.

quietly capture program drop calculate_denominator
program calculate_denominator
    args k lambda age
    scalar result = 0
    forv j = 0/`k'{
        scalar result = result + (`age' + 1 - `j')^`lambda'
    }
    di result
end
calculate_denominator 3 1.5 3

Here I assume the answer to Clyde's question 2 is that all of the inputs are constant scalars and the output should be written to the console. Given these inputs, this program will print 17.02458 to the console.

[Hanna Thompson-Schumann]

Hi All,

Thank you for your responses. I am looking to generate a weighting function for each ages 65-80 years old, for each year of their life. Age and lambda are not variables in the data set, and lambda will always be set to 1.5. My variables are January (1094-2002) and the percent change in returns over each year. I am trying to simplify this code:

gen weight80=(81-k[_n])^1.5/23985.199
gen weight79=(80-k[_n])^1.5/23256.199
gen weight78=(79-k[_n])^1.5/22540.658
gen weight77=(78-k[_n])^1.5/21838.49
gen weight76=(77-k[_n])^1.5/21149.613
gen weight75=(76-k[_n])^1.5/20473.941
gen weight74=(75-k[_n])^1.5/19811.389
gen weight73=(74-k[_n])^1.5/19161.869
gen weight72=(73-k[_n])^1.5/18525.297
gen weight71=(72-k[_n])^1.5/17901.58
gen weight70=(71-k[_n])^1.5/17290.645
gen weight69=(70-k[_n])^1.5/16692.389
gen weight68=(69-k[_n])^1.5/16106.726
gen weight67=(68-k[_n])^1.5/15533.568
gen weight66=(67-k[_n])^1.5/14972.826
gen weight65=(66-k[_n])^1.5/14424.408

So far I have manually imputed the denominator for each year using this code:

forvalues i=0/80 {
gen w`i'=(80+1-`i')^1.5
}

where I would change 80 to 79 and so on for each "weightx".


The variable 'k' in my dataset is there for the purpose of generating the numerator in "weightx"

I then need to simplify this code:

gen exp80=(pchange[_n]*weight80[_n])
gen exp79=(pchange[_n]*weight79[_n])
gen exp78=(pchange[_n]*weight78[_n])
gen exp77=(pchange[_n]*weight77[_n])
gen exp76=(pchange[_n]*weight76[_n])
gen exp75=(pchange[_n]*weight75[_n])
gen exp74=(pchange[_n]*weight74[_n])
gen exp73=(pchange[_n]*weight73[_n])
gen exp72=(pchange[_n]*weight72[_n])
gen exp71=(pchange[_n]*weight71[_n])
gen exp70=(pchange[_n]*weight70[_n])
gen exp69=(pchange[_n]*weight69[_n])
gen exp68=(pchange[_n]*weight68[_n])
gen exp67=(pchange[_n]*weight67[_n])
gen exp66=(pchange[_n]*weight66[_n])
gen exp65=(pchange[_n]*weight65[_n])

and then this:

egen avgexpo80=sum(exp80)
egen avgexpo79=sum(exp79)
egen avgexpo78=sum(exp78)
egen avgexpo77=sum(exp77)
egen avgexpo76=sum(exp76)
egen avgexpo75=sum(exp75)
egen avgexpo74=sum(exp74)
egen avgexpo73=sum(exp73)
egen avgexpo72=sum(exp72)
egen avgexpo71=sum(exp71)
egen avgexpo70=sum(exp70)
egen avgexpo69=sum(exp69)
egen avgexpo68=sum(exp68)
egen avgexpo67=sum(exp67)
egen avgexpo66=sum(exp66)
egen avgexpo65=sum(exp65)

gen totalold=expo1+expo2+expo3+expo4+expo5+expo6+expo7 +expo8+expo9+expo10+expo11+expo12+expo13+expo14+ex po15+expo16

* Example generated by -dataex-. For more info, type help dataex
clear
input double(date return) float pchange byte k
1904.01    1248.23   -.13275203 80
1905.01    1611.89    .29134053 79
1906.01    1955.31    .21305424 78
1907.01    1884.29  -.036321606 77
1908.01       1460   -.22517234 76
1909.01    1970.63     .3497466 75
1910.01    2069.47    .05015655 74
1911.01    2143.84   .035936736 73
1912.01    2242.43    .04598757 72
1913.01    2240.09 -.0010435109 71
1914.01    2091.28   -.06643037 70
1915.01    1957.31   -.06406125 69
1916.01    2494.28     .2743408 68
1917.01     2399.7   -.03791876 67
1918.01    1633.69    -.3192107 66
1919.01    1636.57  .0017628804 65
1920.01    1673.77   .022730466 64
1921.01    1462.55   -.12619416 63
1922.01    1810.08     .2376192 62
1923.01    2351.23     .2989647 61
1924.01    2407.88    .02409377 60
1925.01    3060.82      .271168 59
1926.01    3723.59     .2165335 58
1927.01    4249.17    .14114873 57
1928.01    5895.51     .3874498 56
1929.01    8804.81     .4934772 55
1930.01    7974.79   -.09426893 54
1931.01    6627.43    -.1689524 53
1932.01    4107.16    -.3802786 52
1933.01    4272.25    .04019566 51
1934.01    6540.99      .531041 50
1935.01     5840.4   -.10710764 49
1936.01    8918.92     .5271077 48
1937.01   11585.07     .2989319 47
1938.01    7812.89     -.325607 46
1939.01    9293.35    .18948942 45
1940.01    9646.35    .03798415 44
1941.01    8665.18   -.10171412 43
1942.01    7076.25   -.18336953 42
1943.01    7994.13    .12971278 41
1944.01    9598.49    .20069225 40
1945.01   11230.45    .17002258 39
1946.01   15307.15    .36300415 38
1947.01   11398.53   -.25534603 37
1948.01   10612.86   -.06892731 36
1949.01   11482.96    .08198544 35
1950.01    13792.4    .20111887 34
1951.01   17173.45    .24513863 33
1952.01    20063.1    .16826263 32
1953.01   22923.72    .14258115 31
1954.01   23353.99    .01876964 30
1955.01   34546.45     .4792526 29
1956.01   44378.01     .2845896 28
1957.01   46055.26    .03779462 27
1958.01   41866.58     -.090949 26
1959.01   57959.93     .3843961 25
1960.01   61753.63    .06545384 24
1961.01   64691.74    .04757793 23
1962.01   76532.11    .18302754 22
1963.01   73576.49   -.03861935 21
1964.01    87755.2    .19270708 20
1965.01   100818.6    .14886184 19
1966.01  110413.15    .09516647 18
1967.01   99889.01     -.095316 17
1968.01  111911.38    .12035728 16
1969.01  118590.74    .05968437 15
1970.01  102142.93   -.13869388 14
1971.01  104325.21   .021364965 13
1972.01  115163.39     .1038884 12
1973.01  130965.19    .13721201 11
1974.01  100242.17    -.2345892 10
1975.01   70780.24   -.29390755  9
1976.01   92323.27      .304365  8
1977.01   97614.89    .05731621  7
1978.01   83151.16   -.14817135  6
1979.01   88473.88    .06401258  5
1980.01    91006.7   .028627884  4
1981.01  102595.08    .12733546  3
1982.01   87844.65   -.14377327  2
1983.01  110225.04    .25477237  1
1984.01  127357.06    .15542766  0
1985.01  132779.32    .04257526  .
1986.01   161562.9     .2167776  .
1987.01  209259.81    .29522192  .
1988.01  196340.74   -.06173699  .
1989.01  221275.81    .12699896  .
1990.01  258732.92     .1692779  .
1991.01  242865.96   -.06132563  .
1992.01  312353.99     .2861168  .
1993.01  325917.99    .04342509  .
1994.01  355117.56    .08959177  .
1995.01  349446.13  -.015970571  .
1996.01   460345.5     .3173576  .
1997.01  569065.03    .23616943  .
1998.01  716668.37    .25937867  .
1999.01  927039.53    .29354045  .
2000.01 1042897.76    .12497658  .
2001.01  952957.71   -.08624052  .
2002.01  815250.19   -.14450538  .
2003.01  634691.41   -.22147653  .
end

Thank you!

[Hanna Thompson-Schumann]

Please ignore "gen totalold=expo1+expo2+expo3+expo4+expo5+expo6+expo7 +expo8+expo9+expo10+expo11+expo12+expo13+expo14+ex po15+expo16"

[Daniel Schaefer]

Hi Hanna,

I am unsure as to why you need the square brackets and _n in these lines (e.g.):

gen exp80=(pchange[_n]*weight80[_n])

Aren't these redundant, or am I misunderstanding something? Regardless, how about something like this?

forv i=0/80{
    gen weight`i'=(80+1-`i')^1.5
    gen exp`i' = pchange*weight`i'
    egen avgexpo`i' = sum(exp`i')
}

[Hanna Thompson-Schumann]

Hi Daniel,

Thank you for your response! I was able to figure out most of what I originally sent. This is what I came up with:

#weighted experience per year of life for older generation (65-80) and younger generation (1-35)

forvalues i=65/80 {
gen exp`i'=(pchange[_n]*weight`i'[_n])
}

forvalues i=1/35 {
gen exp`i'=(pchange[_n]*weight`i'[_n])
}

#total weighted lifetime experience for older generation (65-80) and younger generation (1-35)

gen totalexpo=0
forvalues i=65/80 {
egen sumexp`i'=sum(exp`i')
replace totalexpo=(totalexpo+sumexp`i')
}

gen totalexpy=0
forvalues i=1/35 {
egen sumexp`i'=sum(exp`i')
replace totalexpy=(totalexpy+sumexp`i')
}

#diference in average weighted lifetime experience between older generation (65-80) and younger generation (1-35)

gen avgo=totalexpo/15
gen avgy=totalexpy/35

gen difference=avgo-avgy


I used [_n] as I need each percent change in the variable list.

I am still having trouble with the weighting function however..

For example,

I need a weight for every year of an 80-year-old's life, and I need this for ages 65-80 years old.


the function is: weight=(age + 1 - k)/ Σ^age_k^'=0(age +1 - k^')

when I use this code:

gen a=0
forvalues i=0/80 {
gen a`i'=(80+1-`i')^1.5
replace a=a+a`i'
}

forvalues i=65/80 {
gen weight82=(80+1-k[_n])^1.5/a
}

I get the correct weights for each year of an 80-year-old's life:

* Example generated by -dataex-. For more info, type help dataex
clear
input double(date return) float pchange byte k float weight80
1904.01    1248.23   -.13275203 80 .00004169238
1905.01    1611.89    .29134053 79 .00011792385
1906.01    1955.31    .21305424 78 .00021663995
1907.01    1884.29  -.036321606 77   .000333539
1908.01       1460   -.22517234 76 .00046613495
1909.01    1970.63     .3497466 75  .0006127503
1910.01    2069.47    .05015655 74  .0007721537
1911.01    2143.84   .035936736 73  .0009433908
1912.01    2242.43    .04598757 72  .0011256943
1913.01    2240.09 -.0010435109 71  .0013184288
1914.01    2091.28   -.06643037 70  .0015210578
1915.01    1957.31   -.06406125 69  .0017331196
1916.01    2494.28     .2743408 68   .001954212
1917.01     2399.7   -.03791876 67  .0021839803
1918.01    1633.69    -.3192107 66   .002422108
1919.01    1636.57  .0017628804 65   .002668312
1920.01    1673.77   .022730466 64  .0029223354
1921.01    1462.55   -.12619416 63   .003183944
1922.01    1810.08     .2376192 62  .0034529245
1923.01    2351.23     .2989647 61  .0037290796
1924.01    2407.88    .02409377 60   .004012228
1925.01    3060.82      .271168 59   .004302201
1926.01    3723.59     .2165335 58  .0045988415
1927.01    4249.17    .14114873 57   .004902002
1928.01    5895.51     .3874498 56   .005211547
1929.01    8804.81     .4934772 55   .005527346
1930.01    7974.79   -.09426893 54   .005849279
1931.01    6627.43    -.1689524 53    .00617723
1932.01    4107.16    -.3802786 52    .00651109
1933.01    4272.25    .04019566 51   .006850757
1934.01    6540.99      .531041 50   .007196133
1935.01     5840.4   -.10710764 49   .007547127
1936.01    8918.92     .5271077 48   .007903648
1937.01   11585.07     .2989319 47   .008265613
1938.01    7812.89     -.325607 46    .00863294
1939.01    9293.35    .18948942 45   .009005554
1940.01    9646.35    .03798415 44    .00938338
1941.01    8665.18   -.10171412 43   .009766345
1942.01    7076.25   -.18336953 42   .010154384
1943.01    7994.13    .12971278 41    .01054743
1944.01    9598.49    .20069225 40    .01094542
1945.01   11230.45    .17002258 39   .011348295
1946.01   15307.15    .36300415 38   .011755994
1947.01   11398.53   -.25534603 37   .012168462
1948.01   10612.86   -.06892731 36   .012585644
1949.01   11482.96    .08198544 35   .013007487
1950.01    13792.4    .20111887 34   .013433942
1951.01   17173.45    .24513863 33   .013864957
1952.01    20063.1    .16826263 32   .014300486
1953.01   22923.72    .14258115 31   .014740482
1954.01   23353.99    .01876964 30     .0151849
1955.01   34546.45     .4792526 29   .015633697
1956.01   44378.01     .2845896 28    .01608683
1957.01   46055.26    .03779462 27   .016544258
1958.01   41866.58     -.090949 26   .017005943
1959.01   57959.93     .3843961 25   .017471842
1960.01   61753.63    .06545384 24   .017941922
1961.01   64691.74    .04757793 23   .018416142
1962.01   76532.11    .18302754 22    .01889447
1963.01   73576.49   -.03861935 21   .019376867
1964.01    87755.2    .19270708 20     .0198633
1965.01   100818.6    .14886184 19    .02035374
1966.01  110413.15    .09516647 18    .02084815
1967.01   99889.01     -.095316 17     .0213465
1968.01  111911.38    .12035728 16   .021848755
1969.01  118590.74    .05968437 15    .02235489
1970.01  102142.93   -.13869388 14    .02286488
1971.01  104325.21   .021364965 13   .023378683
1972.01  115163.39     .1038884 12    .02389628
1973.01  130965.19    .13721201 11    .02441764
1974.01  100242.17    -.2345892 10    .02494274
1975.01   70780.24   -.29390755  9   .025471553
1976.01   92323.27      .304365  8    .02600405
1977.01   97614.89    .05731621  7   .026540203
1978.01   83151.16   -.14817135  6   .027079994
1979.01   88473.88    .06401258  5   .027623396
1980.01    91006.7   .028627884  4   .028170384
1981.01  102595.08    .12733546  3   .028720936
1982.01   87844.65   -.14377327  2    .02927503
1983.01  110225.04    .25477237  1   .029832637
1984.01  127357.06    .15542766  0   .030393744
1985.01  132779.32    .04257526  .            .
1986.01   161562.9     .2167776  .            .
1987.01  209259.81    .29522192  .            .
1988.01  196340.74   -.06173699  .            .
1989.01  221275.81    .12699896  .            .
1990.01  258732.92     .1692779  .            .
1991.01  242865.96   -.06132563  .            .
1992.01  312353.99     .2861168  .            .
1993.01  325917.99    .04342509  .            .
1994.01  355117.56    .08959177  .            .
1995.01  349446.13  -.015970571  .            .
1996.01   460345.5     .3173576  .            .
1997.01  569065.03    .23616943  .            .
1998.01  716668.37    .25937867  .            .
1999.01  927039.53    .29354045  .            .
2000.01 1042897.76    .12497658  .            .
2001.01  952957.71   -.08624052  .            .
2002.01  815250.19   -.14450538  .            .
2003.01  634691.41   -.22147653  .            .
end

so my question is, how can I do this for years 65-80 years old when 'k' must change for every age (i.e. k=0/79 for a 70 year old and so on..) and the denominator is the sum of year of life.

Thank you!

[Daniel Schaefer]

Hi Hanna,

What about something like this?

forvalues age=65/80 {
gen result`age'=(`age'+1-k)^1.5/a
}

I used [_n] as I need each percent change in the variable list.

I don't believe this is true. See below. Note that both lines below produce an identical variable.

. gen result80_with_n = (80+1-k)^1.5/a
(19 missing values generated)

. gen result80_without_n = (80+1-k[_n])^1.5/a
(19 missing values generated)

. list result80_with_n result80_without_n if result80_with_n != result80_without_n

.
end of do-file

[Hanna Thompson-Schumann]

Hi Daniel,

My mistake you are correct with the [_n].
The forvalues code you sent does work, but I'm still trying to find a solution for the denominator as it is different for each age. I was told to do a nested loop but it doesn't seem to be working.


For age 80 this works perfectly:

gen a=0
forvalues i=0/80 {
gen w`i'=(80+1-`i')^1.5
replace a=a+w`i'
}

This gives me the correct denominator ('a') for age 80.

I was told to do something like this:

gen a=0
forvalues j=1/80 {
forvalues i=0/`j' {
gen w`i'=(`j'+1-`i')^1.5
replace a=a+w`i'
}
}

However, it's not working.

Any help would be greatly appreciated.

Hanna

[Daniel Schaefer]

Hi Hanna,

Could you say a little bit more about the way in which this isn't working? Does it produce an error, or does it produce incorrect or nonsensical values?

The first thing that pops out at me about this is that you may need to "reset" `a' on each new iteration of the outer loop.

gen a=0
forvalues j=1/80 {
    replace a = 0
    forvalues i=0/`j' {
        gen w`i'=(`j'+1-`i')^1.5
        replace a=a+w`i'
    }
}

If I'm reading this right, you loop through each age from 1 to 80 in the outer loop, and then the inner loop goes through each age from 0 to the current age. The inner loop calculates the denominator of the function at the top of the thread then?

Is it a problem that the outer loop starts counting at 1, but the inner loop starts counting at 0, or is that by design?

Also, If the inner loop calculates the denominator, why not let the outer loop calculate the numerator and divide by `a' while you are already in the loop? Maybe something like this:

gen a=0
forvalues j=1/80 {
    replace a = 0
    forvalues i=0/`j' {
        gen w`i'=(`j'+1-`i')^1.5
        replace a=a+w`i'
    }
    gen result`j'=(`j'+1-k)^1.5/a
}

[Daniel Schaefer]

I guess you may also need to deal with the fact that -gen- wont let you create a variable that already exists. You could

 capture drop w`i'

, but I think you can neatly avoid generating the new variable in the first place:

gen a=0
forvalues j=1/80 {  
    replace a = 0
    forvalues i=0/`j' {      
        replace a = a + (`j'+1-`i')^1.5  
    }  
    gen result`j'=(`j'+1-k)^1.5/a
}

[Hanna Thompson-Schumann]

Hi Daniel,

Thank you so much! The last code you sent worked perfectly, and the way you explained everything really helped with understanding.

I have one last question that I would really appreciate your help on:

I'm trying to automate this series of code. The years go up in increments of 5 for years 1984-1999 and then increments of 2 for years 1999-2019.

gen pchange84=pchange if inrange(date, 1904, 1985)

gen pchange89=pchange if inrange(date, 1909, 1990)

gen pchange94=pchange if inrange(date, 1914, 1995)

gen pchange99=pchange if inrange(date, 1919, 2000)

gen pchange01=pchange if inrange(date, 1921, 2002)

gen pchange03=pchange if inrange(date, 1923, 2004)

gen pchange05=pchange if inrange(date, 1925, 2006)

gen pchange07=pchange if inrange(date, 1927, 2008)

gen pchange09=pchange if inrange(date, 1929, 2010)

gen pchange11=pchange if inrange(date, 1931, 2012)

gen pchange13=pchange if inrange(date, 1933, 2014)

gen pchange15=pchange if inrange(date, 1935, 2016)

gen pchange17=pchange if inrange(date, 1937, 2018)

gen pchange19=pchange if inrange(date, 1939, 2020)

So far I have:

gen p=0
forvalues i=1984(5)1999{
forvalues j=1904/1919{
forvalues g=1984/`i'{
replace p=0
replace p=pchange if inrange(date, `j', `g')
}
}
gen result`i'=p
}

For the years going up in increments of five, but it's not working fully. The ending years are correct but the start years all begin in 1919.


Hanna

[Clyde Schechter]

That nested loop structure is all wrong for this project. And it will produce some pretty weird results, not just wrong start years. What you want is:

foreach n of numlist 1984(5)1999 2001(2)2019 {
    local nn = mod(`n', 100)
    local nn: display %02.0f `nn'
     gen pchange`nn' = pchange if inrange(date, `=`n'-80', `=`n'+1')
}

[Daniel Schaefer]

Great! I was just typing up something similar, but less elegant. I didn't realize you could combine ranges of values like this: 1984(5)1999 2001(2)2019, but it makes sense given the way the interpreter seems to work.

The nested loop structure is definitely wrong for this particular situation, but right for the problem you posed a few posts back.

[Hanna Thompson-Schumann]

Apologies for the late post. Thank you Clyde and Daniel, your feedback helped me out tremendously!