problems with the capture command

[Glenn Harrison]

Does anyone have experience with the the capture command failing to capture some error? I am using it, I believe consistently with the documentation, in the following way:

capture: ml model ETC, iterate(100)

It is doing this command in the middle of a long series of loops over sub-samples of my data (corresponding to individuals). Then I immediately try to only process solved cases with

if _rc ~= 0 | e(ic) == 100 {

ETC. The problem is that every now and then, and enough times to be time-consuming to "capture by hand" (i.e., skipping that instance by hardwiring the code), I get an error message after the capture command and before the block that processes solved cases.

[Nick Cox]

I think you need to give us a reproducible example.

 
. capture di 42 ! 42

. di _rc
198

shows the principle. The output is suppressed but you can pick a non-zero error code.

[William Lisowski]

Welcome to Statalist! OK, here comes a shot in the dark, perhaps an explanation of the source this problem, although not a solution.

The problem is that every now and then, and enough times to be time-consuming to "capture by hand" (i.e., skipping that instance by hardwiring the code), I get an error message after the capture command and before the block that processes solved cases.

By that, do you mean that your program functions as you expect, but the error message appears in your results window and you wish it suppressed?

If that is the case, consider this. A look at viewsource ml.ado shows that it contains a capture noisily command within the section that processes ml model, and this is overriding the (implicit) quietly that normally accompanies the use of capture. Run the following three lines from the do-file editor

capture {
capture noisily di 42 ! 42
}

and you will see the following in your results window.

. capture {
42!42 invalid name

.
end of do-file

Now, unfortunately I have no solution, because adding an explicit quietly to the initial capture command does not seem to override the noisily within the captured commands. Perhaps a more experienced Stata programmer than I can find a way of tricking Stata. Perhaps you may find that the maximize option to ml will cause the error message to be suppressed down in the bowels of the ml code.

Noninteractive mode is invoked by specifying the maximize option. Use maximize when ml will be used as a subroutine of another ado-file or program and you want to carry forth the problem, from definition to posting of results, in one command.

[Glenn Harrison]

I appreciate the point that Nick makes -- when I have students send me code to debug, I ask for all of their data, and DO file that takes me right to the error. Unfortunately this problem does not lend itself to a simple, toy example, or even a stripped down version. It is an interaction between a specific subample of data and the lengthy ML program involved, and it is not appropriate to use a post like this for material of that kind. But I did use the suggestions from William to perhaps better document the issue. Here is the code I run:

ml clear
ml model lf ML_rdu_neo (r: choice $Rdata = ) (gamma_: ) (delta_: ) (mu: ) if sid==`s', technique(bfgs)
ml init `rNEO' `gamma_' `delta_' `mu', copy
capture noisily {
ml maximize, iterate(50)
}
di "... the return code is " _rc " and the # of iterations is reported to be " e(ic)

Assume the locals are indeed initialized. Here is an illustrative display when this bombs and is not "captured" as I expected:

Iteration 18: log likelihood = -27.850199
Iteration 19: log likelihood = -27.850199 (backed up)

Number of obs = 50
Wald chi2(0) = .
Log likelihood = -27.850199 Prob > chi2 = .

------------------------------------------------------------------------------
| Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
r |
_cons | .7352536 .1373595 5.35 0.000 .466034 1.004473
-------------+----------------------------------------------------------------
gamma_ |
_cons | -13.63789 901.8493 -0.02 0.988 -1781.23 1753.954
-------------+----------------------------------------------------------------
delta_ |
_cons | 1.65909 .622019 2.67 0.008 .4399555 2.878225
-------------+----------------------------------------------------------------
mu |
_cons | .0597694 .0299802 1.99 0.046 .0010094 .1185295
------------------------------------------------------------------------------
... the return code is 0 and the # of iterations is reported to be 19
Maximum number of iterations exceeded.
r(498);

So it seems to think the return code _rc is 0, which should signify no problem, and it reports 19 iterations, but claims that the number of iterations has exceeded the limit I set of 50. This is not the only type of non-captured error I see, but "suggests" that it is something to do with the maximization pointers.

Again, I appreciate that this is forensics without a concrete example or "body," but would welcome any suggestion from someone that might have encountered a similar issue. If I track down an understanding of the problem I'll post back.

Just for context, this is ML code that is debugged of syntax and other errors, and what I am doing is running it at the level of individuals, picked out by the id variable. I have checked, by the way, that this particular id has valid data.

Glenn

[William Lisowski]

It seems to me that you believe that the r(498) and its associated error message pertain to the ml maximize command that completed 19 iterations and yielded a 0 return code. That is not the case. The error message and r(498) come from a subsequent command, perhaps the next ml maximize on the next value of the id variable, or perhaps from some other part of your code that you have not shown.

You might consider temporarily adding a line at the top of your code to make it clear where you are in the process when the subsequent error message appears.

di "... beginning ml process for sid `s'"
ml clear
ml model lf ML_rdu_neo (r: choice $Rdata = ) (gamma_: ) (delta_: ) (mu: ) if sid==`s', technique(bfgs)
ml init `rNEO' `gamma_' `delta_' `mu', copy
capture noisily {
ml maximize, iterate(50)
}
di "... the return code is " _rc " and the # of iterations is reported to be " e(ic)a

A more general way of accomplishing this is with the trace options.

set tracedepth 1
set trace on

Some experimentation with tracedepth may be needed depending on how deeply your do-files are nested. See help trace for details.

Please note the guidance in the Statalist FAQ linked to from the top of the page, as well as from the Advice on Posting link on the page you used to create your initial post. See especially sections 9-12 on how to best pose your question, noting that it's particularly helpful to copy commands and output from your Stata Results window and paste them into your Statalist post using CODE delimiters, as described in section 12 of the FAQ, as Nick and I have done.

[Glenn Harrison]

William:

Thanks for the suggestion. In fact, I did have such a display option prior to the "ml" commands, and there is no other "ml maximize" statements in the loop in question. The block of code I displayed was not edited "within" that block.

I will certainly follow your suggestion on the trace options, and post-formatting style.

Glenn

[William Lisowski]

After review I have another question. In your initial post you wrote

Then I immediately try to only process solved cases with

if _rc ~= 0 | e(ic) == 100 {

Did you perhaps mean to write unsolved cases? If that is a correct copy of your code, it seems to me you are processing only cases for which _rc is not zero or for which your iteration limit was reached.

My point being that by revealing just small pieces of your code, chosen to support your assumption that the capture command is not working as documented, you have made it difficult for others to explore alternative diagnoses for your problem. I do think your best hope at this point is to work with trace to ensure that you understand exactly what commands are being executed in what order by your program. Let us know what you find!

[Glenn Harrison]

That was indeed just a typo in my comment statement. Do understand that I am not claiming there is a bug in the -capture- command. Nor did I "reveal" bits of code to support an assumption, at least not deliberately. I am expecting that there is indeed some issue with how I am processing something that I do not understand. But no criticism of Stata or this command is implied. I'm working on the -trace- option per your valuable suggestion, and will report back anything I find in case others make the same sort of mistake I must be making.

[Glenn Harrison]

Problem solved. Indeed, the -trace- command was the perfect thing to use here, as William initially suggested. I quickly saw that the error came downstream from the -ml maximize- when I was doing a -testnl- command on the estimated model. My mistake was forgetting something I should not have forgotten -- when I saw the error statements about "too many iterations" I assumed incorrectly that it had to be from the maximization I mentioned in my initial post and was focused on, since I had no other maximizations in that loop, but of course there are some iterations needed in any -testnl- command or indeed in any -predictnl- command. So it was one of those that was running into problems, due no doubt to ill-conditioned estimates from the -ml maximize-. Once trace identified the program generating the error, I could add some "noi di" statements flagging what command I was up to, and I found it at once.

Many thanks for the suggestion and patience.