loop skipping help

[Reese Crispen]

Hello,

I'm using Version 14 and am looping over a set of datasets defined by the following locals. The code works, but seems to skip over about a dozen of the datasets by no pattern I can identify--and with no error return code to give me a hint. I'm really puzzled by this. Are there any diagnostic checks any of you would recommend that I check for to figure out why the skipped over datasets were skipped?

local    t1    T21993 
local    t2    T21022 
local    t3    T56803 
local    t4    T60389 
local    t5    T64030 
local    t6    T55336 
local    t7    T35438 
local    t8    T14499 
local    t9    T13816 
local    t10    T43334 
local    t11    T1447 
local    t12    T10178 
local    t13    T49379 
local    t14    T24265 
local    t15    T29591 
local    t16    T62567 
local    t17    T61807 
local    t18    T42889 
local    t19    T32784 
local    t20    T6773 
local    t21    T60593 
local    t22    T60045 
local    t23    T982 
local    t24    T19037 
local    t25    T57733 
local    t26    T15506 
local    t27    T213 
local    t28    T34407
local    t29    T36045
local    t30    T57179 
local    t31    T25565
local    t32    T54219 
local    t33    T51368 
local    t34    T63414 
local    t35    T51494 
local    t36    T15478
local    t37    T52401 
local    t38    T56070

local path "I:\EPI\Informatics\CUSTOMERS\Neurology\Neurology Datasets/"

forv i=1/38{
use "`path'`t`i''L", clear
    foreach var of varlist _all{
    cap if !mi(real(`var')) {
    destring `var', replace
}
}
save "`path'`t`i''L", replace
}

Thanks so much Statalist for the endless generosity

Reese

[Carole J. Wilson]

I would suggest using either -display- or -set trace on- to identify the problem:

forv i=1/38{
    display "`path'`t`i''L"
    }
*OR (choose a small number for tracedepth 1, 2 or 3 should be enough to see the error)
set tracedepth 3
set trace on

forv i=1/38{
    use "`path'`t`i''L", clear
    foreach var of varlist _all{
        cap if !mi(real(`var')) {
            destring `var', replace
            }
        }
    save "`path'`t`i''L", replace
    }
set trace off

[Mike Lacy]

Some thoughts, which might or might help with your immediate problem:
1) cap if !mi(real(`var'))
will test for missing on only the *first* observation's value of `var'. The if command, as opposed to the if qualifier, is executed once for the whole data set, not for each observation. See -help ifcmd-. The -if- command quietly handles impossible references like this by using the first observation only. The code you have here is likely not what you intend, but I don't see how it would produce the problem you describe.
2) Using -capture- could be hiding something strange from you.

[Nick Cox]

I agree with Mike in #5 and think that your code is catching something problematic, perhaps metadata in observation 1!

On #5 1) I would go further.

destring, replace

automatically loops over all variables and changes to numeric if and only if variables are string and can all be converted to numeric unproblematically, so it's hard to imagine how your own loop could improve on that. You will get informative messages otherwise.

[Sven-Kristjan Bormann]

My guess is that you meant instead of

 cap if !mi(real(`var'))

, you meant something like

confirm numeric `var'

.
What you currently do, is to convert the first observation of each variable to a number and test if this conversion does not evaluate to missing.
What you probably want to do is to convert each variable from string to numeric. Another alternative is a code along the lines the below:

program define string_to_numeric
quietly ds, has(type string)
local varlist `r(varlist)'
foreach var of local varlist{
    capture destring `var', replace
    if _rc==0{
        disp "Variable `var' changed from string to numeric"
    }
    if _rc!=0{
     disp "Variable `var' NOT changed"
    }
}
compress
end

It does probably what you intended to do, but only in more verbose and longer (codewise) way.
If your code skips some datasets, then look at the datasets and see if there are any characters in the variables, which might cause problems while converting to numeric.

[Reese Crispen]

Wow, thanks so much everyone. I think you all correctly identified that the problem concerned my attempt to destring only numeric variables with

 cap if !mi(real(`var')) {

I removed that line and settled with only the simple destring line (as Nick suggested), but with -capture- preceding, to bypass errors when the variables weren't numeric.

    foreach var of varlist _all{     destring `var', replace }

Thank you all so very much. Reese p.s. I don't know why the wrap code feature is formatting my code as one line (and this part, too. Also meant to be separate lines)

[Nick Cox]

As said, you don't need a loop there at all.

foreach var of varlist _all {          
    destring `var', replace  
}

is just what

destring, replace

does any way. It was written to do that....

[Reese Crispen]

Nick Cox Oh, sorry I misunderstood you the first time. Thanks for reiterating. It's a great little detail