Error looping over folders (file not found)

[Greg Saldutte]

Hello,

I am attempting to run the same code over two different subfolders. My thinking was that I would set the working directory, then set a global macro for a subdirectory, then a local macro for each folder over which I needed to loop the same code. Here is what I have:

clear 

global root = "/Users/gsaldutte/Downloads/opm-federal-employment-data/opm-federal-employment-data/data/"

cd "$root"

global dataDir1="../1973-09-to-2014-06"
global dataDir2="../2014-09-to-2016-09"
global dataDir3="../2016-12-to-2017-03"

local subfolders `""$dataDir1/dod/dynamic" "$dataDir1/non-dod/dynamic""'

foreach f of local subfolders{

forval year = 1982/2013{

    foreach mth in DEC JUN MAR SEP {

        infix float psuedo_id 1-9 str employee_name 10-32 str agency 33-36 ///
        str accession_indicator 37-38 float effective_date 39-46 str age ///
        47-52 str pay_plan 53-54 str grade 55-56 str los_level ///
        57-62 str duty_station 63-71 str occupation 72-75 str occ_cat 76 ///
        str adj_basic_pay 77-82 str type_appointment 83-84 str work_sched 85 ///
        using "`f'/`mth'`year'.DOD.FO05M4.TXT", clear
         
        gen date2=date(string(effective_date,"%8.0f"),"YMD")
        format date2 %td
        
        replace effective_date = date2
        
        drop date2
        
        save "Stata/`mth'`year'.DOD.FO05M4.dta", replace
        }
    }
}

Stata returns the error file ../1973-09-to-2014-06/dod/dynamic/DEC1982.DOD.FO05M4.TXT not found. To me this suggests that the problem are the two periods preceding years in the dataDir globals, but I have had success using this format before.

Thanks.

[Nick Cox]

Is that correct? That there is no such file? Your recipe

 
 `f'/`mth'`year'.DOD.FO05M4.TXT

insists on two periods around DOD, regardless.

[William Lisowski]

The wording of your question suggests that you may not fully understand what "../" means at the start of a file path. It means to go to the directory containing your current working directory.

Since you start with

global root = "/Users/gsaldutte/Downloads/opm-federal-employment-data/opm-federal-employment-data/data/"
cd "$root"

then that is your current working directory.

The full path from there to the file you are trying to read

../1973-09-to-2014-06/dod/dynamic/DEC1982.DOD.FO05M4.TXT

would thus be

/Users/gsaldutte/Downloads/opm-federal-employment-data/opm-federal-employment-data/1973-09-to-2014-06/dod/dynamic/DEC1982.DOD.FO05M4.TXT

As Nick asks, does the file exist in that location? Or is it in

/Users/gsaldutte/Downloads/opm-federal-employment-data/opm-federal-employment-data/data/1973-09-to-2014-06/dod/dynamic/DEC1982.DOD.FO05M4.TXT

in which case you should remove the "../" from each of your dataDir globals.

[Nick Cox]

I think William Lisowski may have nailed it.

[Greg Saldutte]

Thank you both for your responses. I did not correclty understand what "../" meant. I've removed it and my code now works.

[Greg Saldutte]

If I may ask a related question, I've realized that the file `f'/`mth'`year'.DOD.FO05M4.TXT only exists in one of the subfolders, with the name `f'/`mth'`year'.DOD.FO05M3.TXT in another subfolder.

Would it be possible to set a loop so that Stata reads in DOD.FO05M4.TXT from the dod folder, while reading in DOD.FO05M3.TXT from the non-dod folder?

[William Lisowski]

If I understand you correctly, the name of the TXT file in the non-dod subfolder ends in M3 while the name of the corresponding TXT file in the non-dod subfolder ends in M4.You will want this difference preserved in the name of the output dataset, otherwise, the second of each pair written would overwrite the first, as it currently would if they both ended in M4 as the code in post #1 assumes.

If so, I think something like this, based on your code in post #1, would do the do the trick.

clear 

global root = "/Users/gsaldutte/Downloads/opm-federal-employment-data/opm-federal-employment-data/data/"

cd "$root"

global dataDir1="../1973-09-to-2014-06"
global dataDir2="../2014-09-to-2016-09"
global dataDir3="../2016-12-to-2017-03"

local subfolder4 "$dataDir1/dod/dynamic" 
local subfolder3 "$dataDir1/non-dod/dynamic"

foreach s of numlist 4 3 {

  local f `subfolder`s''

  forval year = 1982/2013{

    foreach mth in DEC JUN MAR SEP {

        infix float psuedo_id 1-9 str employee_name 10-32 str agency 33-36 ///
        str accession_indicator 37-38 float effective_date 39-46 str age ///
        47-52 str pay_plan 53-54 str grade 55-56 str los_level ///
        57-62 str duty_station 63-71 str occupation 72-75 str occ_cat 76 ///
        str adj_basic_pay 77-82 str type_appointment 83-84 str work_sched 85 ///
        using "`f'/`mth'`year'.DOD.FO05M`s'.TXT", clear
         
        gen date2=date(string(effective_date,"%8.0f"),"YMD")
        format date2 %td
        
        replace effective_date = date2
        
        drop date2
        
        save "Stata/`mth'`year'.DOD.FO05M`s'.dta", replace
        }
    }
}

[Greg Saldutte]

Thank you for this. It appears to solve the problem of the file names. However, I neglected that in the non-dod/dynamic folder, "NON" precedes "DOD" in the file name, e.g., 1982DEC.NONDOD.FO05M3.txt.

[William Lisowski]

Solving the substitution of DOD or NONDOD in the file name can be handled similarly to the subfolder substitution

I'm sympathetic to you as a new user of Stata - it's a lot to absorb. And even worse if perhaps you are under pressure to produce some output quickly. Nevertheless, I'd like to encourage you to take a step back from your immediate tasks.

When I began using Stata in a serious way, I started, as have others here, by reading my way through the Getting Started with Stata manual relevant to my setup. Chapter 18 then gives suggested further reading, much of which is in the Stata User's Guide, and I worked my way through much of that reading as well. There are a lot of examples to copy and paste into Stata's do-file editor to run yourself, and better yet, to experiment with changing the options to see how the results change.

All of these manuals are included as PDFs in the Stata installation (since version 11) and are accessible from within Stata - for example, through the PDF Documentation section of Stata's Help menu. The objective in doing the reading was not so much to master Stata as to be sure I'd become familiar with a wide variety of important basic techniques, so that when the time came that I needed them, I might recall their existence, if not the full syntax, and know how to find out more about them in the help files and PDF manuals.

Stata supplies exceptionally good documentation that amply repays the time spent studying it - there's just a lot of it. The path I followed surfaces the things you need to know to get started in a hurry and to work effectively.