foreach, forvalues and capture confirm to import a series of .csv files

[Steen Harsted]

Hi Stata users

I have some data organized in 14 folders, each folder contains a number of .csv files that have similar names that only varies by a number.
I have written the below code to import the .csv files. Because the varying number in the filenames are different for each folder (but always between 100 and 130) I use the "capture confirm file" command.
The code almost works as intendend, but has the following problem - Whenever Stata finds a .csv filename in a folder - it runs the code in the if_rc==0 bracket as intendend, but then it jumps to next step in the "foreach" loop instead of the next step in the "forvalues" loop. The result being that it only imports the the .csv file with the lowest session number from each folder.

Can you see where my code fails?

clear all
set more off

foreach ID in 01 02 03 04 05 06 07 08 09 10 11 12 13 14 {
cd "Q:\PhD\Validation\Raw Angles\KGC `ID'"

forvalues Session = 100(1)130 {
capture confirm file "Session`Session'.csv"
if _rc==0 {
import delimited "Session`Session'.csv", delimiter(tab) varnames(nonames) rowrange(6)
capture log close
log using "Q:\PhD\Validation\Cleansed Angles\Logs\KGC_`ID'_Session_`Session'log.txt", text replace

* Here I do alot of dropping, renaming and generation of variables - this part works fine I think

cd "Q:\PhD\Validation\Cleansed Angles"
save "KGC_`ID'_Session_`Session'.dta", replace
clear
log close
}
}
}

[Clyde Schechter]

First, it would be a lot easier to follow your code if you used indenting to set off nested levels of control structures. Like this:

clear all
set more off

foreach ID in 01 02 03 04 05 06 07 08 09 10 11 12 13 14 {
    cd "Q:\PhD\Validation\Raw Angles\KGC `ID'"

    forvalues Session = 100(1)130 {
        capture confirm file "Session`Session'.csv"
        if _rc==0 {
            import delimited "Session`Session'.csv", delimiter(tab) varnames(nonames) rowrange(6)
            capture log close
            log using "Q:\PhD\Validation\Cleansed Angles\Logs\KGC_`ID'_Session_`Session'log.txt", text replace

            * Here I do alot of dropping, renaming and generation of variables - this part works fine I think

            cd "Q:\PhD\Validation\Cleansed Angles"
            save "KGC_`ID'_Session_`Session'.dta", replace
            clear
            log close
        }
    }
}

Your problem is coming from the command that I have put in bold face. That command pulls you out of the directory where your .csv files are, so when it goes back to the next value of Session it doesn't find the file--because you are now in the wrong directory. I'm not susre why that command even appears in the code at all. But at the very least it belongs after all of the loops. Anywhere inside, it will disrupt the loops by throwing you into the wrong directory at some point.

[Steen Harsted]

I´m sorry for not using indenting and for the inconvenience it has caused. I will use it in the future.

You are absolutely correct with your analysis - I have now changed the lines:

 cd "Q:\PhD\Validation\Cleansed Angles" 
save "KGC_`ID'_Session_`Session'.dta", replace

to

save "Q:\PhD\Validation\Cleansed Angles\KGC_`ID'_Session_`Session'.dta", replace

And now the code works as intendened. Thanks alot!

I will be more careful with using the "cd" command in the future :-)

[William Lisowski]

Steen -

Let me add a hint that may help you avoid problems in the future when you include Windows filesystem paths in Stata commands. The backslash character

\

has meaning to Stata: it will prevent the interpretation of characters that have special meaning to Stata, in particular

`

will not be interpreted as indicating a reference to a macro. But all is not lost: Stata will allow you to use the forward slash character in path names on any operating system, and on Windows will take care of doing what must be done to appease Windows.

. local prefix batch42

. display "Q:\HOME\`prefix'_filename"
Q:\HOME`prefix'_filename

. display "Q:/HOME/`prefix'_filename"
Q:/HOME/batch42_filename

[Steen Harsted]

Thanks a lot William - you have without a doubt saved me from some future problems :-)