Tokenize and looping through tokens

[Federico Nutarelli]

Hi Statlist,

I have the following problem. I have defined 3 local variables defining the path to some folders containing 3 different .dta files.
What I would like to do actually is open each file separately and work on it in a single loop. The pseudo code should read something like this:

local folder_product "/Users/federiconutarelli/Desktop/PhD/Lavoro_Riccaboni/DATi_USA/Pharma_project_2019 /prodmol_aggregation"
local folder_firm "/Users/federiconutarelli/Desktop/PhD/Lavoro_Riccaboni/DATi_USA/Pharma_project_2019 /firm_aggregation"
local folder_atc2 "/Users/federiconutarelli/Desktop/PhD/Lavoro_Riccaboni/DATi_USA/Pharma_project_2019 /ATC_aggregation"

local aggr "product firm atc2"
tokenize `aggr' // syntax (cfr help tokenize)

foreach y in `aggr'{
local filelist: dir "`folder_"+`1' files "DB_" + `1'  + "_aggregation.dta" // where `1' should be "product" in the token right?
foreach file of local filelist {
  use "`file'", clear
[DO SOME STUFFS]
}
local filelist: dir "`folder_"+`2' files "DB_" + `2'  + "_aggregation.dta" // where `2' should be "firm"
foreach file of local filelist {
  use "`file'", clear
[DO SOME STUFFS]
}
local filelist: dir "`folder_"+`3' files "DB_" + `3'  + "_aggregation.dta" // where `3' should be "atc2" in the token right?
foreach file of local filelist {
  use "`file'", clear
[DO SOME STUFFS]
}
}

Of course it does not work but I cannot figure out why. Can someone please help me?

Thanks,

Federico

[Clyde Schechter]

The syntax in your -local filelist- statements is mangled. For example, the first one should read:

local filelist: dir "`folder_`1'" files "DB_`1'_aggregation.dta"

By the way, if the stuff you do with each group of files is actually the same, consider wrapping the whole thing in a -forvalues- loop to shorten the code and make the parallelism obvious.

Added: By the way, the code I show above is syntactically correct and I believe it represents what you intended with the code you wrote. But it makes no sense. The pattern shown after -files- will be matched by only one folder, if any. So either you need to expand the pattern in some way to include multiple files in that folder, or your -foreach file of local filelist- loops are unnecessary: you can just put the actual filename into the commands inside the loops.

[Bjarte Aagnes]

"`folder_"+`1' files "DB_" + `1' + "_aggregation.dta"

try to use string concatination which is not needed.

Macros are placeholders for text an will be expanded to the content:

local a A
local b B
local c C

di "`a'`b'`c'"

ABC

concatination is not neccessary and add complexity (quoting and evaluation)

di "`= "`a'" + "`b'" + "`c'" '"

So, just include the macros in the string

folder_`1' files DB_`1'_aggregation.dta

[William Lisowski]

I cannot figure out why it doesn't work, either, but that's because I cannot understand what you are trying to do.

You have 3 directories and 3 different data files. Is that 3 data files in each of the 3 folders? Nine files total?

Do the files have the same names in each of the folders?

You told us the names of your 3 folders. Pick one of the folders and tell us the name(s) of the file(s) within that that you wish to use. Then tell us how the filenames in the other two folders differ.

Are the three sets of "[DO SOME STUFFS]" the same code, or are they different?

[Federico Nutarelli]

Sorry for the lack of details. So with these I hope to be clearer.

Is that 3 data files in each of the 3 folders? Nine files total? Do the files have the same names in each of the folders?

The files are 3 and are called DB_product_aggregation.dta, DB_firm_aggregation.dta and DB_atc2_aggregation.dta. They are located in 3 different folders. Specifically:

 local folder_product "/Users/federiconutarelli/Desktop/PhD/Lavoro_Riccaboni/DATi_USA/Pharma_project_2019 /prodmol_aggregation" // HERE IS LOCATED DB_product_aggregation.dta

local folder_firm "/Users/federiconutarelli/Desktop/PhD/Lavoro_Riccaboni/DATi_USA/Pharma_project_2019 /firm_aggregation" // HERE IS LOCATED DB_firm_aggregation.dta

local folder_atc2 "/Users/federiconutarelli/Desktop/PhD/Lavoro_Riccaboni/DATi_USA/Pharma_project_2019 /ATC_aggregation" // HERE IS LOCATED DB_atc2_aggregation.dta

Are the three sets of "[DO SOME STUFFS]" the same code, or are they different?

What I am trying to do actually is to first access to DB_product_aggregation.dta and do some analysis, then to DB_firm_aggregation.dta and do some other analysis. Finally to DB_atc2_aggregation.dta and do some other analysis. The "[DO SOME STUFFS]" part is basically construct a graphic with the three databases but working on different variables specific to each of the databases. Actually, I can generalize by asking if there is a way to access to multiple .dta files "sequentially" in a loop.

Thanks a lot!

Federico

[William Lisowski]

Thank you for the clarifications.

It seems to me the following would do what you describe: read 3 datasets each from a different directory, with no need for a loop.

local file_product "/Users/federiconutarelli/Desktop/PhD/Lavoro_Riccaboni/DATi_USA/Pharma_project_2019 /prodmol_aggregation/DB_product_aggregation.dta"
local file_firm    "/Users/federiconutarelli/Desktop/PhD/Lavoro_Riccaboni/DATi_USA/Pharma_project_2019 /firm_aggregation/DB_firm_aggregation.dta"
local file_atc2    "/Users/federiconutarelli/Desktop/PhD/Lavoro_Riccaboni/DATi_USA/Pharma_project_2019 /ATC_aggregation"/DB_atc2_aggregation.dta"

use "`file_product'", clear
[DO SOME STUFFS]
use "`fiile_firm'", clear
[DO SOME STUFFS]
use "`file_atc2'", clear
[DO SOME STUFFS]

[Federico Nutarelli]

Thank you for the clarifications

Thank you for the advice!

Federico