Using filelist to import and append several tsv files

[Edo Marchesi]

I would like to import and then append a multitude of tsv files which are located within different subfolders of my main folder. I am trying to replicate the example at the end of filelist's help file, with a minor twist (instead of importing a csv I am importing a tsv and I want 3 columns to be read as strings).

Unfortunately, I get the following error:

in range not allowed
r(101);

Here is my code

filelist, dir("/Users/emarchesi/Desktop/TEST") pat("*.tsv") save("tsv_datasets.dta")

use "tsv_datasets.dta", clear
         local obs = _N
         forvalues i=1/`obs' {
           import delimited "tsv_datasets.dta" in `i', stringcols(1 2 3) clear
           local f = dirname + "/" + filename
           insheet using "`f'", clear
           gen source = "`f'"
           tempfile save`i'
           save "`save`i''"
         }
        
 use "`save1'", clear
 forvalues i=2/`obs' {
     append using "`save`i''"
 }

[Robert Picard]

You messed up the filelist example as you adapted it to your needs. It should go something like this:

filelist, dir(".") pat("*.tsv") save("tsv_datasets.dta") replace

use "tsv_datasets.dta", clear
local obs = _N
forvalues i=1/`obs' {
    use "tsv_datasets.dta" in `i', clear
    local f = dirname + "/" + filename
    
    import delimited "`f'", stringcols(1 2 3) clear

    gen source = "`f'"
    tempfile save`i'
    save "`save`i''"
}

use "`save1'", clear
forvalues i=2/`obs' {
    append using "`save`i''"
}

Personally, I now prefer to do this using runby (also from SSC):

clear all
filelist, dir(".") pat("*.tsv")

program do1case
    local f = dirname + "/" + filename
    import delimited "`f'", stringcols(1 2 3) clear
    gen source = "`f'"
end
runby do1case, by(dirname filename) verbose

You can see another example of the technique here.