Advice on creating loops in Stata 14

[Monzur Alam]

Hello Statalisters,

I downloaded the following code made publicly available by Olken, Onishi & Wong, (2014) from https://economics.mit.edu/faculty/bolken/data, and am using it in Stata 14. I am not too familiar with loops in Stata, although it seems to me that it is possible to use a loop which would allow me to run the same commands for Wave II and Wave III data and save them in two separate tables. It would be great if anyone could suggest a way to put this code into a loop.

Thanks,

Monzur

Reference: Olken, B. A., Onishi, J., & Wong, S. (2014). Should Aid Reward Performance? Evidence from a field experiment on health and education in Indonesia. American Economic Journal: Applied Economics, 6(4), 1-34.

    cd ../
    cd "Wave II"
    
    use table_1_2_vars_w2_with_w1controls, clear
    
        foreach q in gross_SD_enroll_kec gross_SMP_enroll_kec{
        bys Location_ID: egen `q'_w1_temp = max(`q'_w1) if obs_type == 10
        replace `q'_w1 = `q'_w1_temp if obs_type == 10
        drop `q'_w1_temp
    }
    
    save table_1_2_w2, replace
    
    
    cd ../
    cd "Wave III"
    
    use table_1_2_vars_w3_with_w1controls, clear
    
    
        foreach q in gross_SD_enroll_kec gross_SMP_enroll_kec{
        bys Location_ID: egen `q'_w1_temp = max(`q'_w1) if obs_type == 10
        replace `q'_w1 = `q'_w1_temp if obs_type == 10
        drop `q'_w1_temp
    }
    
    save table_1_2_w3, replace

[Daniel Bela]

Try this as a start:

cd ../

forvalues num=2/3 {

    if (`num'==2) local romannum "II"
    if (`num'==3) local romannum "III"

    cd "Wave `romannum'"
    
    use table_1_2_vars_w`num'_with_w1controls, clear
    
    foreach q in gross_SD_enroll_kec gross_SMP_enroll_kec{
        bys Location_ID: egen `q'_w1_temp = max(`q'_w1) if obs_type == 10
        replace `q'_w1 = `q'_w1_temp if obs_type == 10
        drop `q'_w1_temp
    }
    save table_1_2_w`num', replace    
    cd ../
}

This is untested code, as you did not share a data example (as recommended in the FAQ) to try it.

You can choose to rename your folders to contain arabic instead of roman numbers ("Wave 2" instead of "Wave II"), hard-code the conversion between the two (as in my example code) or use available user-written programs (see search roman) to handle the conversion.

Regards
Bela

[Marcos Almeida]

Just wish to add two helpful resources to hone skills with regards to loops: the Stata Manual on programming and Christopher Baum's book on programming.

[Monzur Alam]

Try this as a start:

cd ../

forvalues num=2/3 {

if (`num'==2) local romannum "II"
if (`num'==3) local romannum "III"

cd "Wave `romannum'"

use table_1_2_vars_w`num'_with_w1controls, clear

foreach q in gross_SD_enroll_kec gross_SMP_enroll_kec{
bys Location_ID: egen `q'_w1_temp = max(`q'_w1) if obs_type == 10
replace `q'_w1 = `q'_w1_temp if obs_type == 10
drop `q'_w1_temp
}
save table_1_2_w`num', replace
cd ../
}

This is untested code, as you did not share a data example (as recommended in the FAQ) to try it.

You can choose to rename your folders to contain arabic instead of roman numbers ("Wave 2" instead of "Wave II"), hard-code the conversion between the two (as in my example code) or use available user-written programs (see search roman) to handle the conversion.

Regards
Bela

Thank you very much Bela, for the code, it works! I didn't include the dataset because the do file is very messy and makes use of a large number of tempfiles created from a number of datasets. In order to run this code, I would need to upload a number of datasets and 200 more lines of code. But I will keep the guidelines in mind for next time.

I had one question about the loops though. For the following code (which is a continuation of the above code), it seems that there is a loop within a loop (because foreach x is looking at the variables from each round of data). How do I modify the code to account for this?

    foreach x of var pre_natal_visits_w`num' good_assisted_delivery_w`num' post_natal_visits_w`num' iron_pills_categ_w`num' imm_uptak_pct_23mons_w`num' times_weighed_last3months_w`num' vitA_total_6mons_2years_w`num' mal_weightforage_w`num' enroll_age7to12_w`num' enroll_age13to15_w`num' age7to12_pct_twoweeks_w`num' age13to15_pct_twoweeks_w`num'{ 
        **Grabbing the kabupaten average for 3202270
        bys kab_id:        egen ka`x' = mean(ka12`x')    
        replace k12_`x' = ka`x' if Location_ID == "3202270"
        drop ka`x' ka12`x'
    }

[Daniel Bela]

[...]
I had one question about the loops though. For the following code (which is a continuation of the above code), it seems that there is a loop within a loop (because foreach x is looking at the variables from each round of data). How do I modify the code to account for this?
[...]

I don't understand the question; you can nest loops without a problem in Stata, like this:

foreach num in 1 2 3 {
    foreach let in A B C {
        display `"`let'`num'"'
    }
}

In fact, even the code we already know of contains a foreach-loop inside a forvalues-loop. Stata is totally fine with it.

Thus said, all you have to do to adjust your code in order to work inside of foreach is replace every instance of a wave number (2 or 3) with "`num'". That's more or less it, I guess.

Regards
Bela

[Monzur Alam]

I don't understand the question; you can nest loops without a problem in Stata, like this:

foreach num in 1 2 3 {
foreach let in A B C {
display `"`let'`num'"'
}
}

In fact, even the code we already know of contains a foreach-loop inside a forvalues-loop. Stata is totally fine with it.

Thus said, all you have to do to adjust your code in order to work inside of foreach is replace every instance of a wave number (2 or 3) with "`num'". That's more or less it, I guess.

Regards
Bela

Sorry, my question was a little unclear. I tried replacing every instance of a wave number with "`num'" so that it looks like this. The first part, which is your code runs fine, but then I get an error message " ka12pre_natal_visits_w2 not found", and was wondering if there is a problem with my adjustments.

cd ../

forvalues num=2/3 {

if (`num'==2) local romannum "II"
if (`num'==3) local romannum "III"

cd "Wave `romannum'"

use table_1_2_vars_w`num'_with_w1controls, clear

foreach q in gross_SD_enroll_kec gross_SMP_enroll_kec{
bys Location_ID: egen `q'_w1_temp = max(`q'_w1) if obs_type == 10
replace `q'_w1 = `q'_w1_temp if obs_type == 10
drop `q'_w1_temp


    foreach x of var pre_natal_visits_w`num' good_assisted_delivery_w`num' post_natal_visits_w`num' iron_pills_categ_w`num' imm_uptak_pct_23mons_w`num' times_weighed_last3months_w`num' vitA_total_6mons_2years_w`num' mal_weightforage_w`num' enroll_age7to12_w`num' enroll_age13to15_w`num' age7to12_pct_twoweeks_w`num' age13to15_pct_twoweeks_w`num'{ 
    
        bys kab_id:        egen ka`x' = mean(ka12`x')    
        replace k12_`x' = ka`x' if Location_ID == "3202270"
        drop ka`x' ka12`x'
    }

}
save table_1_2_w`num', replace
cd ../
}

[Daniel Bela]

Of course I can't test your code without a data example, but my guess is you're not closing the loops in the correct order. It seems that you run (1) my forvalues loop, then inside of it a (2) a foreach loop and (3) another foreach loop inside of (2), which maybe inappropriate.

How about this?

cd ../
forvalues num=2/3 {
    if (`num'==2) local romannum "II"
    if (`num'==3) local romannum "III"
    cd "Wave `romannum'"
    use table_1_2_vars_w`num'_with_w1controls, clear
    foreach q in gross_SD_enroll_kec gross_SMP_enroll_kec{
        bys Location_ID: egen `q'_w1_temp = max(`q'_w1) if obs_type == 10
        replace `q'_w1 = `q'_w1_temp if obs_type == 10
        drop `q'_w1_temp
    } /* end of "foreach q in ..." */
    foreach x of var pre_natal_visits_w`num' good_assisted_delivery_w`num' post_natal_visits_w`num' iron_pills_categ_w`num' imm_uptak_pct_23mons_w`num' times_weighed_last3months_w`num' vitA_total_6mons_2years_w`num' mal_weightforage_w`num' enroll_age7to12_w`num' enroll_age13to15_w`num' age7to12_pct_twoweeks_w`num' age13to15_pct_twoweeks_w`num'{
        bys kab_id: egen ka`x' = mean(ka12`x')
        replace k12_`x' = ka`x' if Location_ID == "3202270"
        drop ka`x' ka12`x'
    } /* end of "foreach x of var ..." */
    save table_1_2_w`num', replace
    cd ../
} /* end of "forvalues num=... */

Regards
Bela

[Monzur Alam]

Of course I can't test your code without a data example, but my guess is you're not closing the loops in the correct order. It seems that you run (1) my forvalues loop, then inside of it a (2) a foreach loop and (3) another foreach loop inside of (2), which maybe inappropriate.

How about this?

cd ../
forvalues num=2/3 {
if (`num'==2) local romannum "II"
if (`num'==3) local romannum "III"
cd "Wave `romannum'"
use table_1_2_vars_w`num'_with_w1controls, clear
foreach q in gross_SD_enroll_kec gross_SMP_enroll_kec{
bys Location_ID: egen `q'_w1_temp = max(`q'_w1) if obs_type == 10
replace `q'_w1 = `q'_w1_temp if obs_type == 10
drop `q'_w1_temp
} /* end of "foreach q in ..." */
foreach x of var pre_natal_visits_w`num' good_assisted_delivery_w`num' post_natal_visits_w`num' iron_pills_categ_w`num' imm_uptak_pct_23mons_w`num' times_weighed_last3months_w`num' vitA_total_6mons_2years_w`num' mal_weightforage_w`num' enroll_age7to12_w`num' enroll_age13to15_w`num' age7to12_pct_twoweeks_w`num' age13to15_pct_twoweeks_w`num'{
bys kab_id: egen ka`x' = mean(ka12`x')
replace k12_`x' = ka`x' if Location_ID == "3202270"
drop ka`x' ka12`x'
} /* end of "foreach x of var ..." */
save table_1_2_w`num', replace
cd ../
} /* end of "forvalues num=... */

Regards
Bela

I have attached the link to the dataset (sorry about the size!) and part of the code ( pasted below and also in google drive) I have been trying to adjust. The loop starts to work fine until about line 217 (in Stata 14) where it shows the error that I mentioned previously. Thank you very much for your help so far!

https://drive.google.com/drive/u/3/f...zdTT2JTbWl0Ulk

 

local rundir "/Desktop/vars2/"


cd/
cd `rundir'

    
local do_table3 = 1
 

*****


clear
set more off
set mem 400m
set matsize 6000


*********
* Generate w1 kec means of the 12 main indicators to use as controls in all specifications
*********
    use "kecs_info_clean", clear
    gen kab_id = llk02_cd
    gen prov = llk01_cd
    keep Location_ID kab_id drop
    sort Location_ID
    tempfile temp
    save `temp', replace
    
    
    ***B/C we need to substitute in kabupaten avg for kecamatan in some cases where missing (for school variables, b/c not have kids in some kec), want to make sure we have only proper non-dropping kecs going into average
    use "Wave II/table_1_2_vars_w2", clear
    keep Location_ID kab_id
    duplicates drop

    cap drop _merge
    sort Location_ID
    merge Location_ID using `temp', uniqusing
    tab _m
    cap codebook Location_ID if _m == 2
    drop _m
    drop if drop == 1
    keep Location_ID kab_id
    sort Location_ID
    
    tempfile kecs_match_for_avg
    save `kecs_match_for_avg', replace
    
    use "Wave I/table_1_2_vars_generasi_w1", clear    
    cap drop _merge
    sort Location_ID
    merge Location_ID using `kecs_match_for_avg', uniqusing
    tab _merge
    keep if _merge == 3
    drop _merge
    
    //Creating kecamatan level Wave 1 averages of the 12 indicator variables
    foreach x of var  pre_natal_visits_w1 good_assisted_delivery_w1 post_natal_visits_w1 iron_pills_categ_w1 imm_uptak_pct_23mons_w1 times_weighed_last3months_w1 vitA_total_6mons_2years_w1 mal_weightforage_w1 enroll_age7to12_w1 enroll_age13to15_w1 age7to12_pct_twoweeks_w1 age13to15_pct_twoweeks_w1 { 
        bys Location_ID: egen k12_`x' = mean(`x')
        bys kab_id:        egen ka12`x' = mean(`x')    
        replace k12_`x' = ka12`x' if k12_`x' == .    
    }
    
    keep Location_ID k12_*_w1 ka12*_w1
    duplicates drop 
    
    sort Location_ID
    tempfile k12_avg_w1
    save `k12_avg_w1', replace
    
    
    **Identify kab_id and Location_ID info based on ea_code (for those .dta files for which not already have merged in at this point)
    use "Wave I/table_1_2_vars_w1", clear
    keep ea_code kab_id Location_ID
    drop if ea_code == ""
    duplicates drop
    sort ea_code
    tempfile kab_info
    save `kab_info', replace

    

*******************************************************
//Carrying out regressions for Table 3 indicators
******************************************************

if `do_table3' == 1 {

set more off    
    cd/
    cd `rundir'
    cd "Wave I"
    
    use table_1_2_vars_generasi_w1, clear
    
    //Keeping only non-drop kecamatan
    cap drop _merge
    sort Location_ID
    merge Location_ID using `kecs_match_for_avg', uniqusing
    tab _merge
    keep if _merge == 3
    drop _merge
    
    //Creating kecamatan level Wave 1 averages of the 12 indicator variables (+ 4 other alt specs) [not want to do the kab average fill in for the gross enrollment kec variables, since doing regression at kec level]
    foreach x of var  pre_natal_visits_w1 good_assisted_delivery_w1 post_natal_visits_w1 iron_pills_categ_w1 imm_uptak_pct_23mons_w1 times_weighed_last3months_w1 vitA_total_6mons_2years_w1 mal_weightforage_w1 enroll_age7to12_w1 enroll_age13to15_w1 age7to12_pct_twoweeks_w1 age13to15_pct_twoweeks_w1 cond_age13to15_pct_2wks_w1 enroll_age13to15_SMP_w1 enroll_age7to12_SD_w1 enroll_age13to15_other_w1{ 
        bys Location_ID: egen kec_`x' = mean(`x')
        bys kab_id:        egen kab_`x' = mean(`x')    
        replace kec_`x' = kab_`x' if kec_`x' == .
    }
    
    drop _merge
    keep Location_ID kec_*_w1 kab_*_w1 
    duplicates drop 
    
    sort Location_ID
    tempfile kec_avg_w1
    save `kec_avg_w1'
    
    ****
    
    use Data/R, clear
    
    gen panel_hh_id_back_w2 = substr(rid,1,3) + substr(rid,5,5) + "00"
    gen panel_hh_id_back_w3 = substr(rid,1,3) + substr(rid,5,5) + "00" + "00"
    
    //Renaming variable for use to fill in information for panel households in wave 2
    rename rlk12  HH_type_fill
    
    keep panel_hh_id_back* HH_type_fill
    duplicates drop
    tempfile tempback
    save `tempback'

    drop panel_hh_id_back_w3
    sort panel_hh_id_back_w2
    tempfile hhtype_w2
    save `hhtype_w2'
    
    use `tempback'
    drop panel_hh_id_back_w2
    sort panel_hh_id_back_w3
    tempfile hhtype_w3
    save `hhtype_w3'
    
    ********************************************
    *****           Wave II                ***** 
    ********************************************
    
    cd ../
    cd "Wave III"
    
    use table_1_2_vars_generasi_w3_with_w1controls, clear
    saveold table_1_2_vars_w3_with_w1controls, replace

    
    ****Bela's Code****
    
    cd ../

forvalues num=2/3 {

    if (`num'==2) local romannum "II"
    if (`num'==3) local romannum "III"

    cd "Wave `romannum'"
    
    use table_1_2_vars_w`num'_with_w1controls, clear

 
    
    drop _merge

    
    //Merging in treatment/control information
    sort Location_ID
    merge Location_ID using "../kecs_info_clean", uniqusing
    tab _merge
    keep if _merge == 3
        
    
    //Dropping observations from the drop kec list
    drop if drop == 1
    cap drop _m*
    
    //Merging in kec level wave 1 values to the wave 2 data
    sort Location_ID
    merge Location_ID using `kec_avg_w1', uniqusing
    tab _merge
    
    drop if _merge == 2
    drop _merge
    
    sort panel_hh_id_back_w`num'
    merge panel_hh_id_back_w`num' using `hhtype_w`num'', uniqusing _merge(_m2)
    tab _m2 panel_hh, m
    
    drop if _m2 == 2
    drop _m2
    
    ** Merge in 12 main indicator kec avg levels from baseline
    cap drop _merge
    sort Location_ID
    merge Location_ID using `k12_avg_w1', uniqusing
    tab _merge
    drop if _merge == 2
    drop _m*

    
    ****
    **Incorporating kab level averages for kec 3202270 since not have kec level w1 values for it

    
    
    foreach x of var pre_natal_visits_w`num' good_assisted_delivery_w`num' post_natal_visits_w`num' iron_pills_categ_w`num' imm_uptak_pct_23mons_w`num' times_weighed_last3months_w`num' vitA_total_6mons_2years_w`num' mal_weightforage_w`num' enroll_age7to12_w`num' enroll_age13to15_w`num' age7to12_pct_twoweeks_w`num' age13to15_pct_twoweeks_w`num'{ 
        **Grabbing the kabupaten average for 3202270
        bys kab_id:        egen ka`x' = mean(ka12`x')    
        replace k12_`x' = ka`x' if Location_ID == "3202270"
        drop ka`x' ka12`x'
    }


    foreach x of var pre_natal_visits_w`num' good_assisted_delivery_w`num' post_natal_visits_w`num' iron_pills_categ_w`num' imm_uptak_pct_23mons_w`num' times_weighed_last3months_w`num' vitA_total_6mons_2years_w`num' mal_weightforage_w`num' enroll_age7to12_w`num' enroll_age13to15_w`num' age7to12_pct_twoweeks_w`num' age13to15_pct_twoweeks_w`num' cond_age13to15_pct_2wks_w`num' enroll_age13to15_SMP_w`num' enroll_age7to12_SD_w`num' enroll_age13to15_other_w`num'{ 
        **just using this for kec 3202270, so mean will just be the kab value we already have for the other kecs in the kab
        bys kab_id:        egen ka`x' = mean(kab_`x')
        replace kec_`x' = ka`x' if Location_ID == "3202270"
        drop kab_`x' ka`x'
    }
 
    ****

    ** New household - Wave II info     -> hh_type
    ** Panel household - Wave I info    -> hh_type_fill
    ** Split household - Wave I info    -> hh_type_fill

    tab HH_type HH_type_fill, m
    
    ****    
    **Panel and split are panel HH's 
    cap drop panel_hh
    gen panel_hh = (rar00x == 1 | rar00x == 2)

    replace HH_type = HH_type_fill if panel_hh == 1
    tab HH_type rar00x, m

    **Include missing option so that those with HH_type info still missing and those with panel_hh info n/a are put into own groups
    egen HH_type_and_panel = group(HH_type panel_hh), missing
    
    ****
        
    gen categ = ""
    gen sub_categ = ""
    
    foreach x of numlist 1/18{
        gen indic_`x'_gen_1plusA = .
        gen indic_`x'_gen_2plusA = .
    }
    
    replace categ = "Generasi Year 1, Versi A" in 1
    replace sub_categ = "Coeff" in 2
    replace sub_categ = "SE" in 3
    replace sub_categ = "P-val" in 4
    
    replace categ = "Generasi Year 1, Year 2, Versi A" in 5
    replace sub_categ = "Coeff" in 6
    replace sub_categ = "SE" in 7
    replace sub_categ = "P-val" in 8
    
    
    //Creating age value that will not overlap with any true age values to create a dummy group for those children with missing ages
    //(this okay since doing dummies for age use the i.age_years method below)
    **Are actually no children presently with missing age_years, but have this in case need in future
    replace age_years = 8888 if age_years == . & obs_type != 6
    
    //Generating variable for the interaction of province and P
    egen prov_P = group(prov_id P)
    
    
    ********
    * create differenced versions
    ********

**Only have gross enrollment wave I values for those in original wave, so need to apply values to all obs
    foreach q in gross_SD_enroll_kec gross_SMP_enroll_kec{
        bys Location_ID: egen `q'_w1_temp = max(`q'_w1) if obs_type == 10
        replace `q'_w1 = `q'_w1_temp if obs_type == 10
        drop `q'_w1_temp
    }
    

    
    foreach q in pre_natal_visits good_assisted_delivery post_natal_visits iron_pills_categ imm_uptak_pct_23mons times_weighed_last3months vitA_total_6mons_2years mal_weightforage enroll_age7to12 enroll_age13to15 age7to12_pct_twoweeks age13to15_pct_twoweeks cond_age13to15_pct_2wks enroll_age13to15_SMP enroll_age7to12_SD enroll_age13to15_other{
        g dk_`q'_w`num' = `q'_ - kec_`q'_w1
     
        g di_`q'_w`num' = `q'_w`num' - kec_`q'_w1
        replace di_`q'_w`num' = `q'_w`num' - `q'_w1 if `q'_w1 != .
            
    }
    
    g dk_gross_SD_enroll_kec_w`num' = gross_SD_enroll_kec_w`num' - gross_SD_enroll_kec_w1
    g dk_gross_SMP_enroll_kec_w`num' = gross_SMP_enroll_kec_w`num' - gross_SMP_enroll_kec_w1
    
    g dk_gross_SD_enroll_kec_crsec_w`num' = gross_SD_enroll_kec_crsec_w`num' - gross_SD_enroll_kec_w1
    g dk_gross_SMP_enroll_kec_crsec_w`num' = gross_SMP_enroll_kec_crsec_w`num' - gross_SMP_enroll_kec_w1
    ***
    
    **generating dummy for whether the wave 1 individual level control variable is missing
    foreach x in pre_natal_visits good_assisted_delivery post_natal_visits iron_pills_categ imm_uptak_pct_23mons times_weighed_last3months vitA_total_6mons_2years mal_weightforage enroll_age7to12 enroll_age13to15 age7to12_pct_twoweeks age13to15_pct_twoweeks cond_age13to15_pct_2wks enroll_age13to15_SMP enroll_age7to12_SD enroll_age13to15_other{
        gen ind_`x'_m1 = (`x'_w1 == .)
        gen ind_`x'_w1 = `x'_w1
        replace ind_`x'_w1 = 0 if `x'_w1 == .
    }
    

    
    gen include_x = 0
    
    gsort Location_ID obs_type -gross_SMP_enroll_kec_w`num' -gross_SD_enroll_kec_w`num' -gross_SMP_enroll_kec_w1 -gross_SD_enroll_kec_w1
    by Location_ID obs_type: replace include_x = 1 if _n == 1 & obs_type == 10
        
    //gen gen_pooled = (gen_year1 == 1)
    //gen gen_pooled_versi_A = (gen_year1_versi_A == 1)
    
    gen wave = 2

    **Generating dummies equivalent to FE methods used previously (can't use areg, xi, etc. b/c of suest being particular)
                tab prov_P, gen(prov_P_)
                tab HH_type_and_panel, gen(HH_type_and_panel_)
                tab age_years, gen(age_years_)
                                
                rename k12_times_weighed_last3months_w1 k12_times_weighed_last3m_w1 /* the name is too long, max name character is 32 */
                rename kec_times_weighed_last3months_w1 kec_times_weighed_last3m_w1
                rename ind_times_weighed_last3months_w1 ind_times_weighed_last3m_w1
                rename ind_times_weighed_last3months_m1 ind_times_weighed_last3m_m1
                rename times_weighed_last3months_w1 times_weighed_last3m_w1
                rename times_weighed_last3months_w`num' times_weighed_last3m_w`num'
                rename dk_times_weighed_last3months_w`num' dk_times_weighed_last3m_w`num'
                rename di_times_weighed_last3months_w`num' di_times_weighed_last3m_w`num'

    **Saving list of kecs not dropped for use in constructing comparable baseline mean
    preserve
    keep Location_ID
    duplicates drop
    sort Location_ID
    tempfile nodrop_kecs
    save `nodrop_kecs'
    restore
    
 
    
    save table_1_2_w`num', replace    
    cd ../
} /* end of forvalues num=2/3*/
}    /* end of if do_table*/

[Monzur Alam]

Just wish to add two helpful resources to hone skills with regards to loops: the Stata Manual on programming and Christopher Baum's book on programming.

Thank you!