Import a folder with .csv files

[Francisco Nobre]

Dear all,

I am trying to import a folder with .csv files to stata. Each file corresponds to a month of data, and they cover the period January 2013 to December 2020.
I am using the following code:

clear all
cd "${folder}"
local satafiles: dir . files "*.csv"
foreach file of local satafiles {
preserve
import delimited using "`file'", clear
save temp,replace
restore
append using temp
}
rm temp.dta
save "${folder}dataset", replace
clear all

However, I get the error "temp.dta not found".
Is there any typo/mistake in the code?
If someone could help, that would be great! Thanks!

[William Lisowski]

Perhaps one of the csv files has no data, or has some other problem, so that after the import delimited, there were no observations and no data, and thus the subsequent save command did not actually create a new temp dataset, although it did delete the previous version. And then the subsequent append command failed.

That's what I'm going to assume, because you presented none of the output from the Results window for us to inspect for clues as to what went wrong. You told us the error, but you showed us nothing that led up to it. In future posts, please help us help you. The Statalist FAQ provides advice on effectively posing your questions, posting data, and sharing Stata output.

You should investigate your data further and see if that file needs to be corrected or moved out of the directory. Looking at your log, following the import delimited command you should see "(0 vars, 0 obs)" as a clue that the file was empty. Of course, you can't tell what file Stata was attempting to import, so you need to add

display "importing `file'"

before the import delimited command, so when things go wrong you have some idea of what Stata was attempting at the time.

You can however just ignore the problem in your data by adding the emptyok option to the save command so that it will create an empty temp dataset if the import failed to find any data, and the append command will be able to run and append nothing to the existing data.

[Mike Lacy]

Adding to William Lisowski 's useful comments and advice about diagnosis and clarity, I'd like to show a slightly different approach to what you are doing. This approach avoids any need for reserve/restore or temp files, and is generally a popular technique among experienced users.

clear all
cd "${folder}"
save "${folder}dataset", emptyok   // Target file will be empty to start with.
local satafiles: dir . files "*.csv"
foreach file of local satafiles {
   import delimited using "`file'", clear
   append using "${folder}dataset"
   save "${folder}dataset", replace
}

[Francisco Nobre]

Thank you both for the very useful comments!

[kabir ahmed]

"How to erase all datafiles with empty records?" I have generated many files with empty records, now I like to erase all the files in a go, please let me know how to do it.

[William Lisowski]

kabir ahmed

The advice you were given earlier at

https://www.statalist.org/forums/for...46#post1602146

still pertains. Please start a New Topic. Near the top of this page, click on the Forums tab, then click on the General Forum, then click on the "+ New Topic" button.

Your question is very unclear, and in your new topic you should address the following.

• You apparently have a directory containing many files, are these files Stata datasets?
• Some files have "empty records". What does that mean? Do you mean they have 0 observations, or do you mean there are observations that have something wrong with them?