[Function evaluator program] generate terms in a loop in order to do a non linear regression with hundreds of variables

[Justin Debu]

Hi everyone,

This is my first post here, i read the FAQ and advice but please let me know if I should change something to make this post more "palatable" to regular users.

I need to do a non linear regression using about 1300 variables that interact in pairs, so overall this amounts to about 650 parameters, therefore I am trying to write a function evaluator program.

My question is two fold. I will outline it before giving more details :

1. Is it possible to generate the required terms for my -nl- regression in a loop inside the function evaluator program
2. If the above answer is yes, could anyone help me figure out how to write this program properly (cf below a tentative minimal program with the features that I need for my regression) ?

First, I use Stata 12. To make sure everything is properly configured on my end I ran the example code provided on page 15 of the help file for the -nl- command with success.

I found function evaluator programs with about 50 variables and parameters which the users had painfully defined one by one. Here as I have to define about 650 terms, I would really like to be able to compute those terms in a loop. An added bonus would be if I could batch-initialize my parameters to the same value, but that is not crucial at all.

I wrote what I think is a minimal version of the code that I need to use, for which I gel the following error "verify that nlwind is a function evaluator program".

Any help is very much appreciated. I first write below a pseudo code in order to try and make what I am trying to do as clear as possible, then I paste my tentative function evaluator program whith comments

---

PSEUDO CODE - to generate non linear terms from specific pairs of variables

program with between 1 and 1500 potential variables as input
define parameters that are the same across pairs of variables
retrieve their values

loop on the pairs{
extract an identifier from the name of the variables belonging to the pair to generate a pair-specific parameter name
retrieve its value
generate the non-linear term combining the two variables of the pair, the pair-specific parameter and the general parameters
}

do the regression of the dependent variable on all 650 non linear terms generated in the loop
end

---

program nlwind
        version 12
        syntax varlist(min=1 max=1500) if, at(name)
        *dependent variable is the first one in my variable list
        local depend : word 1 of `varlist'
        *as I have to work on my variable in pairs I count how many pairs I have in addition to my dependent variable
        local count : word count `varlist'
        replace count=(count-1)/2
        
        // Retrieve parameters out of at matrix
        *those are two parameters that are the same across pairs of variables
        tempname sigma0 sigma1  
        scalar `sigma0' = `at'[1,1]
        scalar `sigma1' = `at'[1,2]
        
        //loop to create all the terms
        *I need to generate terms using pairs of variables (650pairs) so I use a loop
        forvalues i=1/`count'  {
            // local variables
            *extract the code of the pair from their names, the name of "even" variables is of the type pinstallCODE where CODE is a 4 or 5 digit code
            local pinst : word 2*`i' of `varlist'
            local ins= substr("`pinst'", 9, .)
            *create the index that will let me access both variables of pair `i'
            local j=2*`i'
            local k=2*`i'-1
            *locals of those variables
            local pinst`ins' : word `j' of `varlist'
            local wprev`ins' : word `k' of `varlist'
            
            // Retrieve parameters from the loop out of at matrix
            *define a pair specific parameter, coded with the 4- or 5- digits extracted above and stored in local `ins'
            tempname multi`ins'
            scalar `multi`ins'' = `at'[1,2+i]
            
            // Some temporary variables
            *generate pair-specific non linear terms, which have to be summed for the overall regression, so I sum them directly in another variable to avoid writing the 650 of them in the regression below
            tempvar prodvent`ins' sumterms
            generate double `prodvent`ins''=pinst`ins'*(14/(14 + exp(-1*`sigma0' *(`multi`ins''*wprev`ins'- `sigma1'))))*(`multi`ins''*wprev`ins'<25) `if'
            capture gen double `sumterms'=`prodvent`ins'' `if'
            capture replace `sumterms'=`sumterms'+`prodvent`ins'' `if'
        }

        //Fill dependant variable
        *the regression takes place on the sum of the terms generated in the loop, the sum is stored in `sumterms'
        replace `depend'=`sumterms' `if'
end

---

Thank you in advance for any advice on how to tackle this.

Justin Debu