Loop over stata files in a folder no error but not working

[Craig Yang]

Hi All,

I'm using the following code to loop over files in a folder to sort two variables, Destination and Country:

local files: dir "${table}/Destination_same country" files "*.dta"

foreach file of local files {
    use  `"`file'"' , clear
    sort Destination Country
    save `"`file'"', replace
    }

I checked the macro, the local looks like:

. macro list
_files:         "Destination_same country_1.dta" "Destination_same country_3.dta" "Destination_same country_2.dta" "Destination_same
                country_4.dta"

The above code ran smoothly, but when I check the datasets, the two variables are still not sorted, it seems the loop code did not work. I can't figure out why. I'm wondering where I did wrong.

Any help is appreciated!

Many thanks,
Craig

[Nick Cox]

Puzzling, but there is at least one good reason for this.

The local macro has to be visible to the loop. Stata will happily loop over a local macro it can't see and do nothing. I just did this

foreach f of local frog {
    di 42
}

and nothing happened. The local macro frog doesn't exist but it's not illegal as such to have non-existent macros.

A command may be illegal for other reasons if it mentions a macro that doesn't exist, but that's a different issue.

In your case you defined the local macro somewhere but is it visible? Try displaying the macro just before the loop in the same place as the loop. What people sometimes do is define a local macro in one place, say the main session, and also have loop code in a do-file; and that won't work.

[Craig Yang]

Hi Nick,

Thank you very much for your help and the clear explanations! I think you are right, it might because the local is not visible. I defined the local in command window and ran the loop code in a do-file. I did this because the local defining code above does not work in my do-file, this is another problem that has been bothering me for a while...Below are my code in do-file:

local files: dir "${table}/Destination_same_country" files "*.dta"

display "`files'"

foreach file of local files {
    use  `"`file'"' , clear
    sort Destination Country
    save `"`file'"', replace
    }

Below are Stata result:

. do "/var/folders/jt/mvxh0c8d3x9ghhgfwk45zmp00000gn/T//SD43094.000000"

. local files: dir "${table}/Destination_same_country" files "*.dta"

. 
. display "`files'"
Destination_same not found
r(111);

I checked macro and the local "files" is not defined:

. macro list
S_level:        95
S_ADO:          BASE;SITE;.;PERSONAL;PLUS;OLDPLACE
S_FLAVOR:       Intercooled
S_OS:           MacOSX
S_OSDTL:        10.15.3
S_MACH:         Macintosh (Intel 64-bit)

But if I copy and paste the exact same local defining code into command window:

local files: dir "${table}/Destination_same_country" files "*.dta"

The local will be successfully defined:

. macro list
S_level:        95
S_ADO:          BASE;SITE;.;PERSONAL;PLUS;OLDPLACE
S_FLAVOR:       Intercooled
S_OS:           MacOSX
S_OSDTL:        10.15.3
S_MACH:         Macintosh (Intel 64-bit)
_files:         "Destination_same country_1.dta" "Destination_same country_3.dta" "Destination_same country_2.dta" "Destination_same
                country_4.dta" "Destination_same country.dta"

I can't figure out why...Did I do anything wrong?

Thank you,
Craig

[Nick Cox]

In your second block of code

 
 display "`files'"

needs to be

 
 display `"`files'"'

As the macro contains double quotes, the outermost " " will be stripped on display, so to stop that you need the compound double quotes.

Otherwise you seem to be confirming my guess in #2. Whenever the local macro is defined in a different place, it won't be visible to your code. EITER define and use it in your do file OR define and use it in your main interactive session. Mixing styles won't work.

Local macros have local scope. That's the principle.

[Craig Yang]

Hi Nick, thank you very much for your help, I appreciate it! I see where I did wrong.