Adding chunks of code to loops

[Javier Amaya-Nieto]

Hi everyone,

I am having a problem when building a loop that allwos me to add chunks of code to the next loop. Unfortunally I cant share the data because it is protected by national law. But I hope you understand what I am trying to achieve:

Here is the idea. I need to perform some operations (using the command nlcom) after running a regression and this has to be done for many lags of the data. And in each lag I have to add a chunck of code in the nlcom estimation.

In the next code I will present the code without using a loop for the first and second Lags(this code works well and get the estimator I want):

yvar: Outcome variable (continous variable)
Copay: independant variable (0/1)
data: independant variable (0/1/2) indicating where the data comes from as I am stacking data (lag0, lag 1 and lag 2 regressions)



************************************************** ************************************************** **********
*** 6 step: running the integrated regression

regress yvar i.Copay_lag#i.data_index, cluster(PersonaBasicaID)

************************************************** ************************************************** **********
*** 7 step: calculating coheficients needed

** Creating a matrix for collecting the results of TOTs(coheficients I am interestd in)
matrix TOTsI=J(10,3,-99)
matrix colnames TOTsI = TOT_I se Lag
matrix list TOTsI

** TOT lag0
local tot0 _b[1.Copay_lag#1.data_index]

nlcom `tot0'

matrix TOTsI[1,1] = r(b)[1,1]
matrix TOTsI[1,2] = r(V)[1,1]
matrix TOTsI[1,3] = 0
matrix list TOTsI

** TOT lag1
local itt1 _b[1.Copay_lag#2.data_index]-_b[0.Copay_lag#2.data_index]
local pi1 _b[1.Copay_lag#12.data_index]-_b[0.Copay_lag#12.data_index]

nlcom `itt1' - ((`pi1')*(`tot0')) // -------------------------------------- lines that I haven´t been able to change in the loop formulated.

matrix TOTsI[2,1] = r(b)[1,1]
matrix TOTsI[2,2] = r(V)[1,1]
matrix TOTsI[2,3] = 1
matrix list TOTsI

local tot1 (`itt1') - ((`pi1')*(`tot0')) // -------------------------------------- lines that I haven´t been able to change in the loop formulated.

** TOT lag2
local itt2 _b[1.Copay_lag#3.data_index]-_b[0.Copay_lag#3.data_index]
local pi2 _b[1.Copay_lag#13.data_index]-_b[0.Copay_lag#13.data_index]

nlcom `itt2' - ( ((`pi1')*(`tot1')) + ((`pi2')*(`tot0')) ) // -------------------------------------- lines that I haven´t been able to change in the loop formulated.

matrix TOTsI[3,1] = r(b)[1,1]
matrix TOTsI[3,2] = r(V)[1,1]
matrix TOTsI[3,3] = 2
matrix list TOTsI

local tot2 (`itt2') - ( ((`pi1')*(`tot1')) + ((`pi2')*(`tot0')) ) // -------------------------------------- lines that I haven´t been able to change in the loop formulated.




After thinking throug the problem I came with the next code to loop over the lags but I haven´t been able to solve the problem with the lines indicated the previous code and in the next with "// -------------------------------------- lines that I haven´t been able to change in the loop formulated."



************************************************** ************************************************** **********
*** 6 step: running the integrated regression

regress yvar i.Copay_lag#i.data_index, cluster(PersonaBasicaID)



************************************************** ************************************************** **********
*** 7 step: calculating coheficients needed

** Creating a matrix for collecting the results of TOTs(coheficients I am interestd in)
matrix TOTsI=J(10,3,-99)
matrix colnames TOTsI = TOT_I se Lag
matrix list TOTsI

** TOT lag0
local tot0 _b[1.Copay_lag#1.data_index]

nlcom `tot0'

matrix TOTsI[1,1] = r(b)[1,1]
matrix TOTsI[1,2] = r(V)[1,1]
matrix TOTsI[1,3] = 0
matrix list TOTsI

foreach lag of numlist 1 2 {

di "----------------------- Lag " `lag'

** Creating locals requiered
local one `lag'
local two = 1 + `one'
local three = 10 + `two'
local tot_term1_lag = `lag' - 1
local tot_term2_lag = `lag' - 2

** TOT lag
local itt`lag' _b[1.Copay_lag#`two'.data_index]-_b[0.Copay_lag#`two'.data_index]
local pi`lag' _b[1.Copay_lag#`three'.data_index]-_b[0.Copay_lag#`three'.data_index]

nlcom (`itt`lag'') - (((`pi`lag'')*(`tot`tot_term1_lag''))) // -------------------------------------- lines that I haven´t been able to change in the loop formulated because I need to add the terms "+ ((`pi2')*(`tot0')) " for the loop in teh lag 2 as shown in the first code presented.

di "-----------------------------tot_term1_lag " `tot`tot_term1_lag''

matrix TOTsI[(`lag'+1),1] = r(b)[1,1]
matrix TOTsI[(`lag'+1),2] = r(V)[1,1]
matrix TOTsI[(`lag'+1),3] = `lag'
matrix list TOTsI


local tot`lag' (`itt1') - ((`pi1')*(`tot0')) // -------------------------------------- lines that I haven´t been able to change in the loop formulated because I need to add the terms "+ ((`pi2')*(`tot0')) " for the loop in teh lag 2 as shown in the first code presented.
macro list
}




THANK YOU VERY MUCH IF SOME ONE TAKE SOME TIME TO HELP ME WITH THIS PROBLEM.

[Daniel Schaefer]

Hi Javier,

The code you have provided is very difficult to parse. Here is my attempt at simplification. It looks like you want something in a form like this:

local pil0 = 0
local tot0 = _b[1.Copay_lag#1.data_index]
forv i = 1/10 {
    local `j' = i - 1
    local tot`i' = `itt`i'' - ((`pil`j'' * `tot`j'') + (`pi`i'' * `tot0'))
    nlcom `tot`i''
}

Here, instead of trying to dynamically update the code I just write a single equation that will update your local to the (I believe) correct value on each iteration of the loop. Notice that on the first iteration, pil0 = 0 means that the content of the first set of parenthesis before the addition sign evaluates to zero. Obviously I can't test this code locally without example data. Even made-up data would be helpful for this. If the above doesn't get you on the right track, it would be very helpful if you could help us to simplify the problem. Can you express what you need as an equation? Ideally as a summation of some kind to capture the loop logic?

[Javier Amaya-Nieto]

Hi Daniel,

Thank you very much for your help. I tryed the code you wrote in the post and it worked for the first two lags(As I mentioned, this code is from an analysis that uses lags over time). The problem is that my final analysis must have 107 lags and Stata 17 stops at the lag 7 because the terms are too long to be used by the nlcom command. However, as you asked, I am posting a slide whit the original formula and the formulas I re-wrote for the first 10 lags.

I would appreciate very much if you could give me any suggestion. I have been looking for different solutions but without any up to this moment.

[Daniel Schaefer]

I tryed the code you wrote in the post and it worked for the first two lags

Does Stata throw an error or does the loop start to produce incorrect results after the second iteration?

Stata 17 stops at the lag 7 because the terms are too long to be used by the nlcom command

Okay, but what is the precise error given? Is it a macro substitution error or something else?

I am posting a slide whit the original formula and the formulas I re-wrote for the first 10 lags.

Great, thanks. Please correct me if I am wrong, but it still sounds to me like you are trying to create Stata code that dynamically expands the equation as you have done in the 10 lines of the slide posted above. I would not at all be surprised if the interpreter gave you an error after attempting this. The correct strategy is to write a loop equivalent to the general case given in equation 11 in the upper right hand corner of the slide. You should use your loop to simulate the summation, not the expansion of the equation.

[Javier Amaya-Nieto]

Thank you Daniel,

I run your code and it worked for the first lags but there is a point in which the exprerssion gets too long for stata and it dosn´t work. At the end I had to run parts of the code in Stata and some others in R to solve the problem. I am still working on it as the database is getting too big when using 107 lags. But I already formulated the question in other post.

[Daniel Schaefer]

Thank you for following up.

there is a point in which the exprerssion gets too long for stata and it dosn´t work

That is surprising. So long as you aren't doing any crazy macro/string building, the whole expression should be evaluated on that line in constant time. The call stack should unwind (so to speak) on each iteration of the loop. The complexity of the expression should not grow as the number of iterations of the loop grows.

the exprerssion gets too long for stata

I must have misunderstood what you meant by this. It certainly seemed like you were describing some sort of recursive problem. I apologize if I miss-understood. I'm glad to know you've found a solution that works for you.