Exporting .dta files from different folders and appending using foreach and for values loop

[Pabitra Chowdhury]

I have three datasets located in this below file structure,

1. data_2017_2018/merged_2017.dta
2. data_2018_2019/merged_2018.dta
3. data_2019_2020/merged_2019.dta

I want to append all three .dta datasets with some cleaning without moving three datasets in one folder. The way I am doing is,

foreach i in data_2017_2018 data_2018_2019 data_2019_2020 {

            forvalues year = 2017 (1) 2019 {

            di in red "codes running on year `year'"

            use "$globalpath/`i'/merged_`year'" , clear

           *cleaning codes

                           if `year' == 2017 {

                                *some cleaning code

                                 tempfile datacomb
                                 save `datacomb'
                           }

                          else {
                                    append using `datacomb' , force

                                    *some cleaning codes

                                     save `datacomb' , replace
                                   }
               }
}

In the above code, the first iteration is running but followed that is not working. Can anyone please help on how to fix this issue?

[Clyde Schechter]

The problem is you are using nested loops when you should have `year' and `i' running in parallel. With the nested loops, you are telling Stata to do things like find merged_2018.dta in folder data_2017_2018. As there is no such file there, our code breaks. So the structure should be like this:

forvalues i = 2017/2019 {
    local j = `i' + 1
    display "codes running on year `i'"
    use $globalpath/data_`i'_`j'/merged_`i', clear
        // etc.
}

Note that there is only a single loop involved. I chose to iterate over the year and then compute the folder name. You could also do it the other way: loop over the folder names and extract the year from it--but that's a bit more complicated to do. But, either way, there is only a single loop.