Store number of iterations from eg. -mixed-

[Simon Turner]

Dear Stata experts,

I have managed to find solutions to all my other problems so far by browsing the forums, but I'm stuck on this one.
I found a similar question dating back about 10 years and wondered if things might be different now (or if my search-fu was just too weak).

I am running a simulation study (using Stata 15) and am using -mixed-.

Obviously there can be convergence issues from time to time and I know that I can set the maximum number of iterations to try using iter and whether or not it converged from e(converged).

My question is whether, when it does converge, the number of iterations is stored somewhere that I can access?

If not, is there any sneaky way of finding out (I don't really want to go through hundreds of thousands of logs visually inspecting them)?

Cheers,
Simon Turner.

[Phil Bromiley]

You didn't get a quick response. You'll increase your chances of a useful answer by following the FAQ on asking questions - provide Stata code in code delimiters, readable Stata output, and sample data using dataex.

I'm afraid this is a technical issue on a estimator I don't use so I can't help you on the substance. If you'd provided an e.g. I might have run it to see what I could find.

mixed is an ado file (probably in your stata15/ado/base/m subdirectory). So, if nothing else works, you can modify the ado to save the iterations.

[Weiwen Ng]

This is interesting. Most models estimated via maximum likelihood will put the number of iterations in a temporary scalar called e(ic), even though this isn't necessarily documented in each command's help file (although the help for -maximize- does say that e(ic) is usually returned if the model was fit under ML). So, you could type

regress y x
display e(ic)

The problem is that -mixed- doesn't seem to return e(ic), even though it is estimated by maximum likelihood by default.

[Simon Turner]

My apologies, I'll attach some code...

clear
set obs 100
gen time = _n
gen y = rnormal()
mixed y time,  res(ar 1, t(time)) var reml

Most of these will take between 1 and 3 iterations, so won't take long to run (obviously "real" ones take longer). But I'd like to know if there's a quick way to get the number of iterations.
Thanks Phil for the idea to look at the ado file, I may do that, was just hoping for a quicker solve =)
Cheers!
Simon.

[Andrew Musau]

Obviously there can be convergence issues from time to time and I know that I can set the maximum number of iterations to try using iter and whether or not it converged from e(converged).

At the expense of computing power and time, you can exploit the fact that mixed allows you to specify a maximum number of iterations and outputs the binary scalar -e(converged)- to determine the total number of iterations required. Successively, you can save these across regressions. If your programming skills are good enough, incorporate a code that produces the scalar e(ic) into the mixed ado-file.

webuse nlswork  
forval i= 0(1)1000{
qui mixed ln_w grade age c.age#c.age ttl_exp tenure c.tenure#c.tenure || id: tenure, iter(`i')
if e(converged)==1{
di as text "iterations= `i', converged"
continue, break
}
}

Resulting in

. forval i= 0(1)1000{
  2.
. qui mixed ln_w grade age c.age#c.age ttl_exp tenure c.tenure#c.tenure || id: tenure, iter(`i')
  3.
. if e(converged)==1{
  4.
. di as text "iterations= `i', converged"
  5.
. continue, break
  6.
. }
  7.
. }
iterations= 3, converged

[Rafal Raciborski (StataCorp)]

We will add e(ic) to the stored results for mixed in a future update.

[daniel klein]

If not, is there any sneaky way of finding out (I don't really want to go through hundreds of thousands of logs visually inspecting them)?

Let Stata go through the logs for you. Here is a sketch that might get you started (or might even be what you want).

capture program drop get_iter
program get_iter , rclass
    version 14
    
    tempfile tmp
    tempname tmplog fh
    
    quietly log using "`tmp'" , text name(`tmplog')
    `0'
    quietly log close `tmplog'
    
    file open `fh' using "`tmp'" , read
    file read `fh' line
    while (!r(eof)) {
        gettoken Iteration line : line
        if (`"`Iteration'"'=="Iteration") gettoken n_iter : line , parse(":")
        file read `fh' line
    }
    file close `fh'
    display _newline as txt "Last iteration: " as result `n_iter'
    return scalar ic = `n_iter'
end

clear
set obs 100
gen time = _n
gen y = rnormal()
get_iter mixed y time,  res(ar 1, t(time)) var reml

Best
Daniel

[Simon Turner]

You didn't get a quick response. You'll increase your chances of a useful answer by following the FAQ on asking questions - provide Stata code in code delimiters, readable Stata output, and sample data using dataex.

I'm afraid this is a technical issue on a estimator I don't use so I can't help you on the substance. If you'd provided an e.g. I might have run it to see what I could find.

mixed is an ado file (probably in your stata15/ado/base/m subdirectory). So, if nothing else works, you can modify the ado to save the iterations.

Had a try at that, could find the starting values by EM loop, but couldn't find the Iterations listed in the gradient-based optimization loop. Thanks for the idea though!

[Simon Turner]

Let Stata go through the logs for you. Here is a sketch that might get you started (or might even be what you want).

capture program drop get_iter
program get_iter , rclass
version 14

tempfile tmp
tempname tmplog fh

quietly log using "`tmp'" , text name(`tmplog')
`0'
quietly log close `tmplog'

file open `fh' using "`tmp'" , read
file read `fh' line
while (!r(eof)) {
gettoken Iteration line : line
if (`"`Iteration'"'=="Iteration") gettoken n_iter : line , parse(":")
file read `fh' line
}
file close `fh'
display _newline as txt "Last iteration: " as result `n_iter'
return scalar ic = `n_iter'
end

clear
set obs 100
gen time = _n
gen y = rnormal()
get_iter mixed y time, res(ar 1, t(time)) var reml

Best
Daniel

Thanks Daniel, that works really well!
Will obviously slow things down a little, but a really great solution for me now!
Cheers and thanks very much!
Simon.

[daniel klein]

Will obviously slow things down a little,

Actually, you should not even notice this. The log-file only logs the output created by one mixed command. Compared to the running time of the latter to fit the model, the time for parsing a couple of lines in the log-file is expected to be immaterial.

Glad I could help.

Best
Daniel

[Rafal Raciborski (StataCorp)]

The fix is included in today's update.

[Simon Turner]

The fix is included in today's update.

Thanks! And thanks Daniel too, the scrape worked really well!