nested optimization: moptimize(), ml, and constraints

[David Chan]

Hi everyone,

I am trying to figure out the best way to nest an optimization routine within another optimization routine. To be specific, I am wanting to perform non-linear least squares on panel data inside of a conditional logit model. Consider a panel data setup with units i and "time" t within the units. The overview of what I want to do is something along these lines:

1. Evaluate parameter u_it, which is a linear function of X's for each observation
2. Given u, fit nonlinear least squares to determine parameters W and EW, in turn linear functions of X's for each observation. Although W and EW are linear functions of X's, the moment condition that determines them is nonlinear. In particular, I aim to minimize the norm of W_i - log(\sum_i exp(EW_it))
3. Evaluate the log likelihood of the data given u_it and EW_it, where for y_it \in {0,1}, \sum_i \sum_t 1(y_it==1)*log(Pr(y_it==1)), where Pr(y_it) = exp(u_it+EW_it)/(sum_t' exp(u_it'+EW_it'))
4. Repeat steps #1-3 until log likelihood is maximized

In order to implement step #2, I need to code my own optimization function using moptimize() because there is no off-the-shelf non-linear least squares panel data command that I know of. While I code the function in Mata, I run the maximization in Stata using the -ml- commands, because I have 50 constraints that seem easier to define as a loop using variable names, rather than constructing a matrix to be used by moptimize_init_constraints().

I nest the -ml- command for #2 within an overall Stata program for the conditional logit log likelihood. However, the problem I run into when doing this is that after I run -ml- within the program, the outer maximization routine seems to forget its macros.

So I have a few questions that I would be very grateful to have addressed:

1. Is the best (or only) way to do this to nest moptimize() within moptimize()? The manual (see p. 31 in the link) appears to say that this is possible, since each optimization function has its own handle.
2. If so, what would be the best way to handle a large number of constraints directly within Mata?

I have included some code at the bottom with some comments for where I am encountering problems.

Many thanks in advance.

Best,
Dave

mata:
function nlls_panel(transmorphic M, real scalar todo, real rowvector b, obj, g, H)
{
 ... // Lines omitted for brevity
}                    
end

capture program drop myclogit
program myclogit
args todo b lnf g H
{
    tempvar denom p EW_xb y1
    capture drop u_xb
        // This is not a temporary variable, because I'm not sure how to pass
        // temporary variables to the Mata optimization routine
    mleval u_xb = `b', eq(1)
    gen double `y1'=$ML_y1
        // $ML_y1 seems to be dropped after -ml maximize- below. Using a temporary
        // variable instead of the global macro is a way around this.
    
    // Non-linear least squares (see Mata program)
    ml model d1 nlls_panel() (W: W_*, nocons) (EW: EW_*, nocons), constraints(1/50)
    ml maximize
    predict double `EW_xb', xb equation(EW)
    estimates store EW
    
    // Calculating log-likelihood
    egen double `denom' = sum(exp(ubar+`EW_xb')), by(groupid)
    gen double `p' = exp(ubar+`EW_xb')/`denom'
    mlsum `lnf' = ln(`p') if `y1'==1
        // However, it appears that `lnf' is not reachable. The error message I
        // get is "ml model not found" r(304)
    if (`todo'==0 | `lnf'>=.) exit
}
end

ml model d0 myclogit (choice = u_*, nocons)
ml maximize